Not necessarily however that depends on what your plants like to grow in. I have not seen issue with vegetables. I would like organic matters to last longer in ground since I have perennials that love to grow in well drain organic soil, however these voracious worms converts organics matters to fine compost in a couple years. I had to replenish the soil with organics more often.
I also grew hundreds of Japanese maples from seeds in small plastic pots, worms got in pots where it touches the ground and clogged the holes with their compost which I noticed little late. I have lost some of Japanese maples seedlings after heavy rain last year.