From the beginning of history men wore skirts (long like robes and short like Scottish kilts) and then in the late 14th to early 16th centuries breeches were popular. Breeches were knee length tight pants worn with hose.
Knickerbockers were loose breeches gathered or pleated into a buckled band at the knee. They were the fashion for boys around 1863 and worn with a short collarless jacket. Men started wearing knickerbockers or plus fours (because they came 4 inches below the knee) for golf and sportswear in the 1890’s. Older boys wore a vest (waistcoat) and after 1890's a Norfolk jacket with their knickerbockers. Knickers, actually refer to ladies underpants, but is a common short word for knickerbockers.
Knee pants were short light pants extending to the knees and worn by boys 8 to 16 years old from the late 19th century until after WWI. The word "shorts" for short pants came into being in 1920’s when younger boys ages 2 to 8 begin to wear a wider leg style.
During the 1920’s men wore one-piece swimsuits, which covered their chest. By 1932 the European craze of for swimming without shirts at private beaches gained momentum. New elastic fabrics started to be used in swimwear in the 1950’s, which promised a better fit and held its shape even after the fabric was wet.
In the 1930’s shorts were popular with adults for camping.
Senorita de Alvares played tennis in a below the knee divided skirt in 1931. This set the stage for Alice Marble to wear shorts in 1933 for a professional tennis match and helped the acceptance of shorts in sporting events. But the first sight of tennis shorts in public was on a man, Bunny Austin, who wore shorts at the 1932 U.S. Men’s Championships at Forest Hills, New York.
Bermuda shorts were created at the beginning of the 20th century by British military forces, not in Bermuda, in London! The British military wanted their personnel appropriately, but lightly attired for duty in tropical parts of the Empire!
Before the "standard issue", troops were already cutting off their trousers at the knees to get some heat relief. There were British Regiments in Bermuda where civilians started copying the military shorts.
Men had accepted Bermuda shorts for leisurewear by 1950, and often see paired with sport coats in men’s fashion magazines, even neckties and, of course, long socks.