This can be caused by many things, including stress from temperature fluctuations, over or under watering, not enough sun, or what I expect, the flowers are not being pollinated. Peppers are self pollinating but need a visit from a pollinator to loosen the pollen from the male parts to the female. Wind can also do the job. If you have an old electric toothbrush you can gently put it on the flower to stimulate a pollinator.
Also the plant just might not have been ready. Any update on this. I had this problem last year with my peppers. Its better this year.
