...... Depends on the soap, but if dishsoap - YES, it will deter or kill the bugs that were in contact with the solution; however, dishsoap solutions will also dissolve cuticular wax in the leaf skin (which accelerates water loss through the leaf cuticle), make it easier for disease pathogens to enter the plant, make it easier for the next marauding band of ugly bugs to do their dirty work, and diminish the leaves' resistance to harmful UV rays.
Soaps made from vegetable products, like olive oil, are made from long chain fatty acids which have proven to be much less harmful than most household cleaning products. For instance, Dawn Dishsoap has at least 11 phytotoxic (poisonous to plants) ingredients.
If the solution makes its way into the soil, it can dissolve the lipid membranes in root cells, making water and nutrient uptake more difficult until the plant is able to regenerate enough roots to fully support top growth.
Al