I have not had a lot of success with sustaining a year round worm population in my raised beds either. Having said that, it is only fair to mention that outside of introducing a few containers of fishing worms to the beds I have not given much effort towards that goal.
It's quite possible that birds and lizards got most of them when they surface after a heavy watering or rain.
Long ago I had a number of worm bins on my property in N Cal. They were 4'x8' and about 15" deep. I would water them lightly about once a week and feed them corn meal and coffee grounds as well as kitchen scrap. When I started there were several thousand worms in each bed. After 2 years when I sold a couple the estimated population was around 20/25 thousand per bed.
That project didn't pan out finanically so I dumped the remaining worm bins (10) out and created a 40' by 80' vegetable garden on that spot. For the next 25 years I had an incredible vegetable garden!
