So I took some of the flowers down to the nursery. She is very knowledgeable. She said it's not powdery mildew, because it's not powdery. *sigh.
She recommended I talk to the Farm Advisor. He was out so I have the number for UC Master Gardeners.
Still on the hunt for the culprit that is destroying my Jackmanii