My yellow squash has had a few failed fruits and I was not surprised- assumed lack of pollination on mine, too. . In my case, I could see those were in weird positions and likely the flower couldn't even open very well-- plus, the plant is loaded with females and not many males.
I don't think you can do much in case of too much nitrogen except, don't add any more.
