How long in the same pot and soil?
Soil can break down eventually, reducing aeration and drainage.
I have a distrust of big pots with one small hole. Really, almost don't like any pot with just one hole- they don't seem to drain/aerate very well for me.
I've had a lemon for a few years, it got scale and had a couple winters of losing about every leaf, then rebounding amazingly when back out for summer (I'm also in MD). So I don't think yours will die, but I'd be careful with water, and plan a rehab for early spring. My 2 cents.