If your yard is kept hygienic, it's not likely they will bother you. They don't like people, just their trash, as Keith mentioned.
Yesterday I saw a man in line at Home Depot with 5 liters of Bug-Be-Gone spray. Then, a woman squawking about "Why don't you carry a weed killer/fertilizer application all in one!?" Try pulling the weeds, lady. Yes, with your hands.
So depressing, the ignorance in this country.