Golden Privet (Ligustrum x vicaryi)

I planted a row of these last spring (actually some of them were planted 2 springs ago and most of them did not make the first winter so were replaces last spring). They were really slow to leaf out (did not leaf out until may 1st) So many of my other bushes were leafed out much earlier. I planted these because I wanted something that would grow fast between my neighbor's property and mine (his place is kind of junky) and would not need a lot of water after establishing. I also loved the yellow-lime-green color. I planted them along with Red Prince wigelias. The problem I am having is that the foliage is looking brown, washed out in places and the leaves are much smaller in size than they were when I bought them. I am just very disappointed in their looks. So my question is, do they just need more time? I thought after they developed a better root structure they may look healthier. But we have had alot of rain this spring, I don't know how that factors in. They do not appear to have grown hardly at all since I planted them. My soil is kind of heavy. I can't decide if I should keep them or not