Hello brikuz20, and welcome to the Garden Forum!
Yes, a picture of the plant and pot would be helpful. The plastic nursery pots with lots of drainage holes work quite well for me, although they aren't the prettiest to look at. I'd wait awhile to repot.
Replacing soil won't get rid of the gnats, they'll be back!
Letting the soil dry more will discourage them a bit.
Yellow sticky traps will help catch the adults.
Products containing Bt-i are a least-toxic, more natural treatment that will kill off the larvae in the soil. Gnatrol is formulated for houseplants, but Mosquito Bits and Mosquito Dunks can also work. You'll need to repeat the treatment every few months, and of course correct the conditions that encourage the gnats.
If the Bt-i doesn't do it, (it usually does), and you want to nuke them with the most toxic, environmentally unfriendly insecticide, look for a granular product with Imidacloprid. "Bonide Systemic House Plant Insect Control Granules" is one brand, there are many others.
With any insecticide, please carefully read and follow the label directions.
