You know what else colonizes?
robinia... Far more than paw paw...
Callery pear... This actually should be cut down...
Kentucky coffee tree...
Sassafras...
Hmmm
How about sumac?
Plenty of plants colonize... and there are easy methods of control... I hate it that you had a fruit that you liked, and cut it down!