Ashfield was first settled in 1743 and was officially incorporated in 1765. The town was originally called "Huntstown" for Captain Ephraim Hunt, who died in King William's War, and who had inherited the land as payment for his services. The first permanent settlement was in 1745, by Richard Ellis, an Irish immigrant from the town of Easton. The town was renamed upon reincorporation, although there is debate over its namesake; it is either for the ash trees in the area, or because Governor Bernard had friends in Ashfield, England. The town had a small peppermint industry in the nineteenth century, but for the most part the town has had a mostly agrarian economy, with some tourism around Ashfield Lake.
Sam had planned on speaking in Ashfield August 25, 1881 but canceled. He had asked Howells about the place, who replied:
"Your Ashfield audience will be the farmer-folks of the region, quiet and dull on top, but full of grit and fun; they’re fond of speaking, and rather cultivated, but not spoiled. They know you, like a book, and you can trust all your points to them. Their life is one of deadly solitude and suffocating frugality; but they are smart. They will stand lots of human nature from you." [MTHL 1: 365].