I always do a fall and winter clean up here in GA. I've already started with the peonies that have gone dormant.
We also get mildew here, and we don't stay very cold all the time. The leaves on some of my other plants tend to get slimy😖, and they're no fun to clean up. Further, my roses get blackspot, so leaving the fallen debris below them was not an option.
