The only problem I have with comfrey is getting rid of it.
I use quite a bit of it to feed the veggie garden and compost. I used to feed it to the goats and chickens too, when we had them. Here in Pa. it can be slightly invasive, and to remove it can take a couple of years if it was well established. Getting all the tap roots can be difficult and persistence in digging any sprouts is the only way to get it all.
For the most part I find it's benefits outweigh it's drawbacks.