That's your call, but you're right in thinking that removing the blooms will free up energy the plant can allocate to energy sinks not as strong as the plants reproductive parts.
Plant parts have a pecking order based on how plants are genetically programmed to allocate resources. Energy is first allocated to respiratory function, i.e. to maintenance of living tissues, then, to production of fine roots, followed by flower and seed/fruit production (reproductive parts), then primary growth (extension of both roots and shoots), then secondary growth (thickening), and finally, the synthesis of defensive chemicals. Eliminating blooms directs energy to the sinks lower on the ladder than reproductive parts.
Al