Don't know about you, Kabby but we are having summer weather still - a very long growing season. It would seem like some pollinator has found your gingers that doesn't usually show up until after they're finished blooming?
I've been out at night with a flashlight, and seen huge moths at my Hedychium flowers. They have eyes that glow, it's really cool. Might have been these:
http://www.fs.fed.us/wildflowe...
In any case, if there are still any flowers, go out with a flashlight before bed some night and see what you can see. I put my fingers around the light to dim it a bit, just so I don't blind all the night-flying critters.
