Dave, I don't fertilize either. I do amend as I plant, usually with some kind of natural compost. Do a simple soil test. Fill a glass jar about 1/3 to 1/2 full of dirt. Fill with water. Leave a little bit of room for good shaking action. Shake. Then let everything settle- it'll take a few hours. The dirt will fall out largest to smallest particles. So sand and gravel at the bottom, then clay and silt at the top of the dirt. Organic matter floats.
Then a soil nutrient test, for N-P-K and pH. Get a kit from any bog box hardware store. Your soil might be better than you think.
I'd start with adding organic matter, as most gardens need more. It contains a lot of nutrients and it helps with water holding capacity and drainage both.
