I have considered fencing the entire area or an electric fence.
I have had that entire area with in ground plantings of daylilies and the Deer didn't bother it much, but last year they did wonder in. I already have an electric fence in a bed in front of the house, just because those 4 legged things came in and ate an entire bed of seedlings a couple years ago.
I have friends who uses green plastic coated wire 4' from TSC, because it blends in with the grass and bushes. They use 4x4's on the corners and depending on how big you make it support it every so many feet, with metal stakes.