In the summer of 1855, William Walker, a ruthless, ambitious, famously short Tennessean, invaded Nicaragua with a private militia, declared himself president, and reintroduced slavery. For his brief reign, he was, as one of his biographers recently put it, a “five-foot-five colossus astride the isthmus.” But of all the nicknames he ever earned, Walker particularly hated being called a “filibuster”—because it marked him as a brigand and a thief.

Today, whenever the Senate filibuster is in the news, people note the strange word’s probable origins in a seventeenth-century Dutch word for “freebooter,” or pirate. Few go much further than to imply that the politicians obstructing Senate business bear some resemblance to the buccaneers who interrupted colonial maritime trade. Unflattering as this analogy may be, it isn’t the word’s most relevant history in the United States.

In the two decades before the Civil War, private militias that recruited heavily in the slaveholding South, but enjoyed support elsewhere in the country, invaded nations in Latin America. Denounced as filibusteros by their would-be subjects, and alternately praised and prosecuted as filibusters back home, they styled themselves as statesmen and freelancers serving the sacred cause of Manifest Destiny. Some of the invaders set up nominally independent republics. Others, like Narciso López, a Venezuelan who twice invaded Cuba with support from Mississippi’s governor, sought to annex new slave territories to the United States. None had much lasting success.

The word filibuster, though, lived on as a synonym for any furtive attempt to defend the slave interest and, after that, white supremacy by subverting legitimate political procedure. This, and not the Jolly Roger or Blackbeard’s gold, is the legacy of Walker and of another Southerner with a passion for minority rule, Mitch McConnell.