Prune back your Ficus as much as you can stand. Remove all of the dry, leafless, brittle stems as they are dead.
After pruning, move it back inside to a location in front of a sunny uncovered window. It does best with lots of direct indoor sun.
The pot size is fine, but it should be watered thoroughly as soon as the surface of the soil feels almost dry to the touch. It is hard to overwater this plant so aim to keep the soil damp at all times.
With more sun and water I think you will see some good improvement.
