The Clemens family moved to Hannibal in November of 1839,  

Marshall Clemens sold his holdings in Monroe County and purchased a city block in Hannibal with a hotel on it, the Virginia House.  Located at the corner of Main Street and Hill, Marshall opened a store on the premises.

Located in the hollow between Holliday’s Hill to the north and Lover's Leap to the south, founded by refugees from the New Madrid earthquakes of 1811-12 and named for the defeated Carthaginian general, Hannibal was little more than a one-horse hamlet in 1840, with a cigar factory, a tobacco warehouse, and a weekly newspaper. W. D. Howells once described it as a “loafing, out-at-elbows, down-at-the-heels, slaveholding Mississippi River town,” but within a couple of years it boasted four general stores, three sawmills, two planing mills, three blacksmith forges, two hotels, three saloons, two churches, a hemp factory, and a distillery. Its pork-packing plants, where some ten thousand hogs were slaughtered every fall and winter, were located on the south side of Bear Creek, which flowed eastward through the center of town before emptying into the Mississippi. Marshall Clemens moved his family into the Virginia House, the hotel he had acquired at the corner of Hill and Main Streets, and opened a store on the premises stocked with inventory he was allowed to buy on credit in St. Louis, some fourteen hours downriver by steamboat. Fifteen-year-old Orion worked in the store and seven-year-old Ben was enrolled in a boys’ academy. Twelve-year-old Pamela and four-year-old Sam attended a dame school on Main Street run by Elizabeth Horr, wife of the town cooper and a New England lady of middle age, who charged twenty-five cents per week per student for instruction. Whereas Pamela was commended by the teacher in November 1840 for her “amiable deportment and faithful application to her various studies,” Sam claimed later that he was switched for misbehavior on his first day of class. He remembered, too, that Lizzie Horr always began the day “with prayer and a chapter from the New Testament.”   The Life of Mark Twain: The Early Years, 1835-1871 (page 13-4)