In electricity an electrical path to ground is provided by code at all electrical outlets for safety. In the event of a failure or circumstance that might normally cause the operator (or anyone coming in to electrical contact with connected equipment) to be the return path for the electrical current the ground can shunt this current away safely and return it to earth, which is where it is trying to go anyway. The alternative of the operator being the return path can cause electrocution.
In audio, ground usually refers to either the electrical ground mentioned above, or to an audio shield. An audio shield is not always a ground and should never be used as a safety ground. That they are often at ground potential is a function of how they may be connected to other equipment. Many audio devices have the ability to disconnect their signal paths entirely from electrical ground as a way to prevent hum or ground loop problems.
Using ground lift switches is considered to be far safer than the old "3-to-2 prong AC adapter" solution (which we heartily DON'T recommend, see TTOTD from February 24, 1997 for more on this topic). However, occasionally ground lift switching will not be as effective in resolving problems as the brute force AC adapter might be.