For about as long as the country has existed, the public and the press have imposed few consequences on Presidents for what they do behind closed doors, even when those actions become public, as long as those actions don’t affect the rest of the government. What has changed is the perception of when that line gets crossed.

In the nation’s earliest years, newspapers were associated with political parties, so accusations of infidelity were often brought up to slam political opponents but dismissed by loyalists. “The golden age of America’s founding was also the gutter age of American reporting,” as pundit Eric Burns put it in his book Infamous Scribblers: The Founding Fathers and the Rowdy Beginnings of American Journalism.

The most notorious scandalmonger of that period was James Callender, a Federalist newspaper editor, who, for example, spread stories of Thomas Jefferson’s fathering children with Sally Hemings, a woman enslaved at his estate, and also making a move on the wife of his good friend from college. Of the latter accusation, Jefferson wrote in a July 1, 1805, letter to his Secretary of the Navy Robert Smith, “I plead guilty to one of their charges, that when young & single I offered love to a handsome lady. I acknolege it’s incorrectness; it is the only one, founded in truth among all their allegations against me.” (However, Monticello, the museum at the site of his former home, now acknowledges that the charge about Hemings is true too.)

But such claims about Jefferson didn’t seriously damage his career. The times when personal stories like those did make a difference was when there was concern over whether public figures’ personal lives affected their jobs.