When I have nests in my trees... I avoid doing work on those trees (or near).
Valuable garden help...
Someone bent on destruction would rain down the wrath of god at night while the critters are asleep...
Simply severing the nest from whatever it's attached to is enough...
once the nest hits the ground... it's prey to other predatory species... in my area... fire ants would get the larvae and nothing the adults can do to prevent.
At your house? maybe possums or skunks or armadillo.