I've never actually seen a Vitex anywhere around here...or in a garden centre. The only reason I have one is because a Southern friend gifted me with some seeds so I could experiment.
Maybe next Spring I'll give it a boost with a little greenhouse hat..but you could very well be right, just too cold up here no matter what and we just can't fool it.