It might help to know where this tree is located. It looks like leaf miner damage, but I've never heard of them attacking the fruit. It could just be mechanical damage from the fruit rubbing against a thorn or twig, I suppose. If you have an extension Dept. , that would be the place to ask. Citrus have lots of bug damage and a few diseases in my zone.
