The word "ampersand" comes from the amalgamation of the phrase "& per se &", where "&" is pronounced as "and". The phrase was common in the 1800s, as it was standard practice to pronounce letter names (as opposed to single letter words like "a") in the form <letter> per se <letter>, and ampersand was considered the 27th letter of the alphabet. "Per se" was used as it means "by itself" in Latin.
An alternate form "epershand" originates from Scotland.
More on ampersand.