The pot sounds big enough. Is the soil to the top? My biggest citrus pot is 10 gallons but i use all available space - the soil level is within an inch of the rim.
My suggestion is to leave it in the pot but repot. Use citrus appropriate soil and invest in some citrus specific fertilizer. The soil should be moisture retentive but fast draining. I use regular potting soil but add about 1/3 volume course perlite.
The right fertilizer is important as citrus have very specific nutrient needs. If you repot this fall, don't fertilize until you see spring growth. Then use the fertilizer at half strength every other month until active growth ceases next fall.
Your tree looks really good so it may be its just missing some key nutrients to set and hold fruit. That hole is worrisome. Cut it open and see what's in there.
