If you need to constantly prune it back because it's too big, then it IS too big for the space. Looking at the top of the branches it looks like it's the case.
Rootwise it could do better also. It's root expanding capabilities have been reduced to practically 25% (the other 75% is taken up by the patio and house), making it susceptible for uprooting.
Concerning damage: are we talking about damage to the tree or the hardscape around it?
