While most sources recommend removing the blooms, I just happened to realize I see gorgeous commercial displays of Caladiums that I'm sure no one worries with snapping off the blooms. So I bet its not a big deal if they remain. One year I was growing a vivid hot pink Thai variety that had large white blooms that stood above the foliage. It was such a pretty contrast against the bright foliage, I left them till the blooms started to fade. Typically the blooms are mostly green and aren't showy at all.
