Picture a map of the United States and what do you see? From the two coasts that frame the mental snapshot up to the friendly mitten of Michigan and down to the staggered edge of Texas, you might be thinking of the contiguous states.

That’s the “logo map” of the country, writes Northwestern historian Daniel Immerwahr, and it’s not quite right. In fact, he pointed out in a recent interview, it’s “only been a correct map of the country for three years of its history.”

It’s not just because the map is missing Alaska and the peppering of the Hawaiian islands. It also excludes the places that are still territories of the United States—Puerto Rico, Guam, the Commonwealth of the Northern Marianas, American Samoa and the U.S. Virgin Islands. It was also common for earlier iterations of U.S. maps to ignore former territories like the Panama Canal Zone, which the U.S. held from 1904 to 1976, and the Philippines, which the U.S. controlled from 1898 to 1946, minus when it was occupied by Japanese forces during World War II.

In his upcoming book How to Hide an Empire, Immerwahr sets out to tell the history of the Greater United States, what lies beyond the mainland. He traces the legacy of empire to the U.S.’s founding, explores why the nation avoids this particular part of its past and fills the book with fascinating stories from past and present territories. Immerwahr spoke with Smithsonian about these missing chapters of American history and what the U.S.’s empire looks like today.