I'm surprised nobody answered you about struggles with pests in the garden. That's what being a gardener is all about: fighting the endless war against nature, while still maintaining a good attitude and sense of humor.
Here are some garden enemies for your game. Squirrels, chipmunks, gophers, moles, mice, cutworms, slugs, chickens, rabbits and skunks love to dig and ruin young plants. Deer, squirrels, birds, and raccoons love to eat the mature plants, berries, and vegetables. Bad neighbors can steal your produce, but good ones can water the garden for you if you're gone. Pets can be friends or enemies of the garden. Dogs dig up plants and generally wreck things, but they also chase away deer and other critters. My cat loves to roll in the dirt and squish little plants, but he also eats so many rabbits and squirrels that his destructive tendencies are forgiven.
I don't have a specific answer about the nut trees, but I would speculate that it takes ten years or so to get some nuts, and even more time to have a large harvest. I would think that nut trees might take a little bit longer than fruit trees to bear a good crop.
