It does sound scary!
I think the social bees/wasps that build nests together can become aggressive if they feel their territory is threatened. But otherwise my experience is the same as J's--the solitary bees/wasps just go about their business gathering pollen and pretty much ignore me. Maybe it was one of the social bees?
