They are aggravating. I think they encourage spring sickness, but that's just a personal idea of mine which may not have any truth to it all.
I get plenty of spring sickness but have not had aphids on the daylilies, so ss can occur without aphids. I suppose it is possible that aphids could contribute by damaging the plant and introducing a fungus, but spring sickness certainly can occur heavily without them.