Lee occupies land that was originally territory of Mahican Indians. The first non-native settlement in the area was known as Dodgetown as early as 1760. Dodgetown was named after its founding settler, Asahel Dodge, who immigrated to the area from Cape Cod. Lee was incorporated in 1777 from parts of Great Barrington and Washington. It is named after Revolutionary War General Charles Lee. Lee is a former mill town.
In the autumn of 1786 during Shays' Rebellion, about 250 followers of Daniel Shays encountered state troops commanded by General John Paterson near East Lee. The Shaysites paraded a fake cannon crafted from a yarn beam, and the troops fled.
Early industries included agriculture, lumbering, and lime making. Abundant streams and rivers provided water power for mills that produced textiles and wire. Papermaking became the principal industry in 1806 with the construction of the Willow Mill by Samuel Church in South Lee. The Columbia Mill in central Lee was established in 1827, and eventually became the first to supply 100% groundwood newsprint to The New York Times. By 1857, there were 25 paper mills in Lee. The Smith Paper Company discovered how to manufacture paper solely from wood pulp in 1867, and through the 1870s was the country's largest producer of paper. The mills previously owned by Smith Paper Company were closed in 2008. Today, Lee has only a single papermaking facility.
The town's marble is famous for its quality. The first quarry was established in 1852. In 1867, almost 500,000 cubic feet (14,000 m3) of marble was excavated and shipped on the Housatonic Railroad. Buildings constructed of Lee marble include a wing of the Capitol in Washington, 250 sculptures adorning Philadelphia City Hall, as well as the General Grant National Memorial, and St. Patrick's Cathedral (both in New York City).
The town's 19th-century prosperity is still evident in its architecture, including its town hall, library, several churches and private homes. South Lee includes a historic district listed on the National Register.