The Building and Site. The boxy, mixed-use building stood just north of the Market House, the surviving structure known today as Old City Hall. It was built at an unknown date by Oswego businessman and developer Sylvester Doolittle (1800–1881), who erected a number of commercial structures in the Lake Ontario port city. The building was a nondescript wharfside affair. The auditorium, originally named Doolittle Hall, occupied the upper floor. The ground floor housed a produce wholesaler, a ship's chandler, and other businesses.
In 1875, the building was extensively renovated and reopened as the Academy of Music. The auditorium, renamed Academy of Music Hall, operated for eighteen years. It closed when the building was condemned on December 24, 1892. Demolition followed.