Norfolk is perhaps best known as the site of the Yale Summer School of Music—Norfolk Chamber Music Festival, which hosts an annual chamber music concert series in "the Music Shed", a performance hall located on the Ellen Battell Stoeckel estate to the west of the village green. Norfolk has important examples of regional architecture, notably the Village Hall (now Infinity Hall, a shingled 1880s Arts-and-Crafts confection, with an opera house upstairs and storefronts at street level); the Norfolk Library (a shingle-style structure, designed by George Keller, c. 1888/1889); and over thirty buildings, in a wide variety of styles, designed by Alfredo S. G. Taylor (of the New York firm Taylor & Levi) in the four decades before the Second World War.

Clara Clemens spent the summer of 1905 in a cottage, convalescing, in Norfolk.