Agree, castor bean is grown by the millions as ornamental every summer. Just like opium poppies, foxglove, monkshood, oleander, and so many more.
The power of some of these plants has been harnessed as medication. Foxglove (Digitalis) is used to make heart medication.
Every gardener has that moment when they realize some of their plants are "poisonous." Most of them go on to learn that unless they eat those plants, they are harmless. Toxicity can take various forms, such as from mere contact, contact with the sap, and ingestion. More rarely, from inhaling the smoke from burning.
An extremely small number of plants cause a significant adverse effects from just touching them, like poison ivy, hogweed, machineel tree, various nettles.
The most likely adverse effect experience from plants is from coming in contact with the sap - for those with skin that is sensitive to the particular sap. More than half of my houseplants will cause a rash to my ridiculously sap-sensitive skin if I get the sap on me and don't wash it off right away.
A discussion about that:
The thread "Mystery poison ivy, contact dermatitis rash" in
Houseplants forum
