Downy mildew is more recognizable by seeing the top of the leaves, so I don't know if that's what's afflicting the leaves. However, when I enlarge the image, it looks like that there's some damage from chewing insects.
If you were to have downy mildew, I wouldn't rely on neem extract, even if the label claims that it treats it. In my garden it can be such a destructive fungus (more closely related to algae, actually) that I don't dare take chances with it. So I use Aliette fungicide, which used to be sold in small quantities. However, now it's sold in a very expensive, large 5 lb bag. It's suitable for very large gardens, but would be far too expensive for smaller gardens. Others may weigh in with what they use.