Should I be concerned about these insects in my peace lily soil?

If indoors I'd say probably yes. What's the reason almost all the leaves have been cut off?

It is indoors. I cut them to clean it up, was this not a good idea? Can you tell what the insects are from the picture?

The picture is not too clear, but I suspect they are symphylans. They look like little centipedes but they are white. They eat plant roots, so they are not good to have. I don't know how to get rid of them. Submerging the whole pot in soapy water, and rinsing them off the leaves might do it. It might take more than once.