Background

Born in Latonia, Kentucky in 1900, Robert Allen came out fighting. He lied about his age so he could serve in the cavalry during the Mexican border campaign of 1916-17 and in France during World War I. He began his journalistic career at the University of Wisconsin and showed an early instinct for undercover work—at one point joining the Ku Klux Klan to write an exposé. While attending the University of Munich on a scholarship, he covered Hitler’s November 1923 Beer Hall Putsch and the subsequent trial of the Nazi plotters for several American newspapers. Upon his return to the United States, his credentials were such that Allen became Washington bureau chief for The Christian Science Monitor. Liberal in outlook, he took a dim view of the controversial US intervention in Nicaragua that left the Somozas in power.

In 1930, frustrated by the Monitor’s unwillingness to publish stories critical of high-level Washington officials, Allen recruited Drew Pearson, a correspondent for the Baltimore Sun, and began to collect a series of scandalous reports on the government’s operations in the midst of the Depression. The two journalists assembled the material into a book, Washington Merry-Go-Round, which they published anonymously in the summer of 1931.

The book, which violated virtually every journalistic convention of the time, was a hit. According to Richard L. Strout, a Monitor reporter who worked under Allen, “They had made a sensation and struck a gusher: between July and August, 1931 . . . there were seven printings. Nobody in Washington talked about anything else. It carried no preface, no references, no signature.” (Think of Washington Merry-Go-Round as a precursor to Primary Colors, the best-selling 1996 “novel” by Joe Klein that was also published anonymously at first).  Allen and Pearson’s irreverent, gossip-ridden, and sensational exposé brutally ridiculed key figures in the administration of Herbert Hoover, the engineer-turned-president once venerated for his ability to make government work. “The book is flippant and venomous by turns,” Strout wrote. “Drew supplied the social and diplomatic background, Bob the congressional survey and the angry protest against publishers.”

When their identities were discovered, both men were fired from their respective newspapers. Undeterred, Allen and Pearson produced a sequel in the fall of 1932, More Merry-Go-Round. Altogether, the books sold 180,000 copies, and that encouraged them to continue producing their then-unique combination of reporting and commentary. In December 1932, the United Feature Syndicate started to distribute the “Washington Merry-Go-Round” to newspapers. It was one of the first to emphasize reporting at a time when most columnists were paid (and content) to ponder and pontificate. It quickly became one of the most powerful voices reporting from Washington as the role of the federal government in ordinary life was burgeoning. Only seven papers bought the column at first, but by May 1934, “The Washington Merry-Go-Round” was being syndicated to 270 newspapers. In many ways, Pearson and Allen couldn’t have been more different. The son of Governor Paul Pearson of the Virgin Islands, Drew Pearson was calm, urbane, and elegant. “Redheaded” Allen, on the other hand, was described as “short-tempered,” “brusque,” “carelessly dressed” and “pugnacious.” A 1937 Literary Digest profile reported that he “shouts, roars, blusters at government officials, is profane and disrespectful until some one lets out something exclusive.” In his own words, Allen was always ready to “cut somebody’s heart out.”

Allen and Pearson got their information from cabinet members, senators, congressmen, diplomats, hired investigators, clerks, and other journalists who couldn’t use scandalous discoveries in their own articles. According to historian Donald Ritchie, the pair “cultivated, cajoled, conned, and occasionally blackmailed sources for exclusives.” They broke several of the biggest scoops of the era, including FDR’s 1937 court-packing scheme, the 1939 Louisiana Scandals, and the 1940 Destroyers-for-Bases Agreement with Great Britain. Peppering the hard news was insider gossip like Roosevelt’s taste in suits and the pet names First Lady Lou Hoover called President Hoover in private. Though the column had no explicit affiliation, Allen and Pearson were clearly sympathetic to the Roosevelt administration and other liberals, while openly critical and occasionally outright malicious towards Republicans, as well as fellow journalists whom they felt were mouthpieces for conservative interests. The one time the FBI evinced any real interest in Allen, according to his recently declassified Bureau file, was in the mid 1930s, when it made a modest effort to discern the columnists’ sources.

Allen’s Merry-Go-Round Ends

When the United States entered World War II, Allen, despite his age, decided to re-enlist. He negotiated a leave of absence from his partnership with Pearson in 1942, and was assigned to what became General George Patton’s Third Army, serving as head of intelligence when the unit went to England in 1943; eventually he rose to the rank of colonel. On April 7, 1945, less than a month before V-E Day, Allen was riding in a jeep in central Germany when he was ambushed from the rear. Severely wounded and taken prisoner, Allen had his right arm amputated in a German field hospital. Three days later, after the Third Army took Erfurt, he was liberated.

Allen spent the next year at Walter Reed Army Hospital undergoing treatment on his amputated arm. He was often depressed and morose about the prospect of resuming his career as a journalist. In his absence, Drew Pearson had built the “Washington Merry-Go-Round” into an even bigger success, reflecting the war-driven thirst for more news from Washington. He was not inclined to bring Allen back in, and according to Allen, it was six months before Pearson even visited his erstwhile partner in the hospital.

In a 1946 column, Pearson paid tribute to Allen. “Bob’s battles were always for the underdog—Nicaragua, the little countries, little business, the unemployed, the public which sometimes gets it in the neck,” Pearson wrote. “No cause was too hopeless . . . Bob was always putting burrs under sleepy senators. He would walk into their offices, pound on their desks and demand that they get out on the Senate floor and fight.” But privately, Allen was already well into a heated feud with Pearson that destroyed their relationship  forever. Before Allen left for active duty in World War II, Pearson had agreed to pay Ruth Finney, Allen’s wife (who was also a journalist), 10 percent of the gross receipts from the column. But Pearson reneged on the agreement, and for two and a half years while Allen was fighting in Europe, Ruth received no payments.

The dispute over money, naturally, infuriated Allen. In 1947, Pearson finally agreed to pay $45,000 over six years in overdue allowances to Allen, in exchange for a waiver of his rights to the column. But Pearson was often late with his payments, or did not make them at all, which further embittered Allen. He also felt personally betrayed by the man he had mentored for a decade, and these bad feelings would linger for the rest of his life.

Slowly, Allen trained himself to type with his left hand, and in 1947 published a book, Lucky Forward, which glowingly recounted Patton’s dash across France and into the heart of Germany. Not long after, he started his own syndicated Washington column, eventually partnering with newsman Paul Scott to produce what became known as “The Allen and Scott Report.”

The new column followed the same formula as the “Merry-Go-Round,” minus some of the publicity gimmicks. But far from replicating his earlier success, Allen went into a long and slow eclipse as a Washington columnist. In all likelihood, his ideological journey played some role. Initially, the new column was liberal-leaning in tone and outlook, but as the Cold War became a geo-political fixture, Allen’s views began to harden, shifting to the center and then the right. One of the biggest surprises for Allen’s readers undoubtedly occurred on August 12, 1955, when he turned over his space to J. Edgar Hoover, the arch-conservative and much-maligned director of the FBI, for a guest column. Yet “The Allen-Scott Report” did not readily identify with the nascent and growing conservative movement, as defined by William F. Buckley. In fact, after John F. Kennedy won the 1960 election, Allen sent in his resume, hoping to win the job of press secretary at the Pentagon.

By the 1960s, Allen was a certifiable conservative on national security issues, often criticizing various government agencies (though never the FBI) for what he considered their lackadaisical approach towards the “Communist takeover threat.” With the possible exception of columnist David Lawrence, who was also the editor of US News & World Report, no one ran more columns questioning the performance and claims of the Kennedy administration during the missile crisis. Allen’s suspicions undoubtedly derived from his own career as an intelligence officer, and what were probably excellent sources in the Defense Department and intelligence agencies. His criticisms and knowledge so rankled the administration in 1963 that Attorney General Robert F. Kennedy and Defense Secretary Robert McNamara ordered the CIA to wiretap the phones of Allen and Scott, on the grounds that sensitive “national security information” had appeared in their column (six years before the Nixon administration did the same to other reporters). The wiretap was deemed “very productive,” picking up conversations between Allen/Scott and, among others, 12 senators and six congressmen.

None of the column’s scoops and insights increased readership, however, or gained wider syndication for “The Allen-Scott Report,” which in 1968 became “Inside Washington” when Allen teamed up John A. Goldsmith, a veteran UPI reporter. As influential metropolitan newspapers, such as the Los Angeles Times, generally became more liberal and re-imagined their editorial pages in the 1960s, Allen’s column was often one of the first casualties. Late in the decade, “Inside Washington” was still distributed to about 100 newspapers, but often very minor ones; in Washington itself the column appeared only in a suburban paper, the Northern Virginia Sun.

A telling anecdote about Allen’s decline occurred in 1971, after Daniel Ellsberg leaked the Pentagon Papers, a secret study about U.S. intervention in Vietnam. White House adviser Patrick Buchanan, a sharp-tongued conservative, was making a case for ignoring Ellsberg rather than going all-out to discredit him. “The Allen-Scott Report,” of course, had been defunct for almost three years already, but part of Buchanan’s memo to President Nixon read,

An issue that has been decided on the front pages of the nation’s papers, and on the lead on the nation’s [news] networks, is not going to be turned around in the public mind by a few well-placed leaks to back-page obscurantists like Allen and Scott.

Allen continued to work until 1980, but slowly faded into obscurity. On February 23, 1981, a year after his wife’s death and months after being forced into retirement due to the progressively debilitating symptoms of cancer, Allen died in his Georgetown home from a self-inflicted gunshot wound. He was 80 years old.

Agent Allen

Until recently, that was most of what was known about Robert Allen. In 2009, however, release of the Alexander Vassiliev notebooks produced an astonishing revelation: Allen had worked, albeit apparently briefly, as a Soviet intelligence agent in 1933, just as he was beginning his new venture with Drew Pearson.[1]

According to the notebooks, a message dated January 27, 1933 from the New York rezidentura to Moscow stated,

We call your attention to materials being sent with this mail that came from the newly recruited source Sh/147, who in our opinion is of great interest.

Robert Allen (cover name George Parker) is a journalist by trade. In 1931 he wrote the book Washington Merry-Go-Round in which he described official Washington. The characters he depicts . . . are a reflection of the pettiness and emptiness of many of Washington’s current Republican congressmen and Cabinet members. When this essentially malicious lampoon of unscrupulous Washington politicians was published, [President] Hoover insisted that he be fired from the [publication] where he was working. He personally knows most of the lawmakers and Cabinet members, and also has extensive contacts in all of the departments.

He personally knows Prof[essor] [Raymond] Moley, Roosevelt’s chief adviser, and also knows Roosevelt himself, as well as the Democratic majority leader in the Congress.

Sh/147 is a valuable contact, especially bearing in mind Roosevelt’s future administration. (For now the payment is 100 Am[erican] d[ollars] a m[on]th.)

Two stories on a matter of supreme interest—Roosevelt’s plans for official recognition of the Soviet Union—might have precipitated Allen’s recruitment. In December 1932, a month before Allen started working as an agent, one of the first Allen/Pearson columns reported that the “Roosevelt Administration is expected to look favorably on Russian recognition.” In another column published three days later, the duo explained that Roosevelt was receiving advice from Senator William E. Borah (D-Idaho), a leading proponent of Soviet recognition.

Allen was not approached to be a classic spy, i.e., someone who would convey classified information or documents, or steal technical secrets. Rather, the Soviet service was interested in his access to advance or inside intelligence from high-level sources, including privileged information available to him as an elite journalist, some of which might never appear in any published stories.

The Vassiliev notes show that Allen did provide such inside information. One dispatch to Moscow on January 20, 1933 stated,

[Senator] Borah reports privately that in a talk with Senator [Robert] Bulkley, Democrat of Ohio, following the latter’s visit to [President-elect] Roosevelt [at] his home in Hyde Park, that Bulkley remarked: “You are going to win out on Russian recognition when Roosevelt takes office. He told me he was going to act promptly on that as soon as he takes over.”

On this same date, Allen also provided information about four American naval officers, disguised as native fishermen, who infiltrated the Marianas, Caroline, and Marshall Islands against the wishes of the Japanese, who controlled those areas. While there, the naval officers found “extensive harbor developments,” “huge warehouses filled with naval and military stores,” and “a number of heavy concrete big gun foundations, covered over and hidden by under-growth and huts.” From Moscow’s perspective, of course, Japan’s militarization was second only in importance to what was going on in Germany.

A week later, the Vassiliev notebooks cite two more dispatches based on information provided by Allen. In the first, he advised that Norman Davis, President Wilson’s undersecretary of state, would not be appointed secretary of state by Roosevelt because of his banking connections and because of “congressional antipathy” towards him. In the second, Allen mentioned that both Roosevelt and several Navy admirals and strategists hoped for an uprising in Japan. According to the dispatch,

Naval intelligence officers say they are advised that there is widespread unrest in Japan and that “radical” sentiment is gaining powerful headway.

Their attitude is that this is “all to the good”, and it is to be hoped that it will quickly come to a head in serious outbreaks. They are aware that the movement is of a communist origin and character, but manifest no hostility because of this fact. What their attitude would be if a Communist regime were to be set up in Japan they do not say. But to start with they would view with “friendly” interest internal turmoil in Japan.

Finally, on February 19, 1933, two weeks before Roosevelt’s inauguration, three more dispatches arrived conveying information obtained from agent Allen. In the first, Allen reported that Roosevelt’s close friend Sumner Welles would be named undersecretary of state (though he actually became an assistant secretary at first, not rising to undersecretary until 1937). Allen added that Welles was friendly to Russian recognition and also that Roosevelt would be staying in the “Thompson home” on Massachusetts Avenue with Welles prior to moving to the White House. The latter detail might seem innocuous, but it was a good indicator of the close relationship between the two men. (Indeed, Welles proved to be such a key member of Roosevelt’s inner foreign policy circle that Secretary of State Cordell Hull came to believe that it undermined his ability to be FDR’s chief diplomat, and after years of chafing about it, forced Wells out).

Other information conveyed by Allen concerned Senator Claude Swanson (D-Virginia), and the issue of greatest interest to Moscow: diplomatic recognition. Swanson was likely to become chairman of the Foreign Relations Committee but might also be tapped to join Roosevelt’s Cabinet as secretary of the navy (as indeed he was). According to the notebooks, Allen reported that Swanson now supported Russian recognition, which he formerly opposed, and that

Roosevelt had told him, “I intend putting an end to the Russian recognition question.” This is the only direct information as to Roosevelt’s stand. Numerous reports have come from intimates and advisers that he is friendly to recognition, but Swanson quotation is the only specific statement to this effect.

In the last message, Allen reported receiving a letter from Roosevelt brain-truster Raymond Moley, asking him to investigate Henry A. Wallace—most likely to vet him as a candidate for a Cabinet position. Wallace, of course, did become Roosevelt’s first secretary of agriculture.

In total, the Vassiliev notebooks record seven communications from Allen to his case officer, although there may have been others. The intelligence records examined by Vassiliev were in a chaotic state and never organized. He noted down every recruitment that seemed interesting to him, but could not systematically track a given relationship down to its conclusion. There is no mention of Allen in the Vassiliev notebooks after February 19, 1933, and it is likely his association with Soviet intelligence did not last much longer than a few months. Nor is it at all clear which side ended the relationship.

The Reasons Why

Piecing together Robert Allen’s motives for being a Soviet agent is a difficult task. In his published writings and currently accessible personal correspondence, there is nothing to indicate that he ever was a secret Communist, “fellow traveler,” radical, or even Socialist-leaning. He was never particularly receptive to the USSR, even in his early years, and in his later years he was outright hostile.

There is some evidence that Allen may simply have done it for the money. The income he received may not seem very substantial ($100 per month, or $1,675 in 2010 dollars), but in January 1933 that must have seemed a sizable amount and he may well have needed it. Though he and Pearson had already published their best-selling book 18 months earlier, Allen was not in a comfortable financial position. He had been fired from his job at the Monitor in the summer of 1931, and, except for a very brief tenure at Hearst’s International News Service, was unemployed (and perhaps unemployable) for nearly a year and a half until he found work at the liberal, pro-Roosevelt Philadelphia Record (whose chief editorial writer was I. F. Stone). According to Pearson biographer Oliver Pilat, Allen could not even afford the train fare to visit New York when United Feature Syndicate first queried him and Pearson about starting a column.

Even when the “Washington Merry-Go-Round” started in December 1932, roughly a month before Allen began to work as an agent, success was uncertain. The column initially attracted only seven subscribers and generated just $25 a week. Recalling his early days working with Pearson on the column, Allen once acknowledged that “To earn a living until this experiment proved itself, we did other newspaper work.” In fact, though the column began to generate interest almost immediately, it wasn’t until January 1934, a year after Allen’s relationship with Soviet intelligence began, that he finally earned enough from syndication to quit his second job at the Record. Allen may have worked as a Soviet agent simply to earn more income, and ended the arrangement as soon as the column began to take off.

In all likelihood, however, his reasoning was probably more complex. Though Allen never publicly displayed any ideological fervor, there is some information that suggests that, privately, he may have had sympathy for left-wing causes. A memo in his FBI file dated December 27, 1951 from the Bureau to the Secret Service noted that

No investigation has been conducted by the Bureau concerning Robert Sharon Allen. However, the files reflect that Robert S. Allen . . . was associated with the Southern Conference for Human Welfare and the Consumers Union. The Southern Conference for Human Welfare and the Consumers Union have been cited by the House Committee on Un-American Activities [HUAC] as being Communist front organizations.

The memorandum also noted that Allen had been a member of the Social Workers Committee to Aid Spanish Democracy and the Washington Friends of Spanish Democracy during the mid 1930s, both of were cited by HUAC as being Communist fronts. Exactly how much of a role Allen played in these organizations is unclear, though it was probably not a large one. In any event, the Spanish civil war was the political litmus test for one’s ideological sympathies in the period leading up to World War II.

There are several other indicators, too, that Allen may have perceived fascism as the much greater menace. An eyewitness to the birth of the Nazi movement, he often claimed the distinction of having been the “first American journalist to despise Adolf Hitler and Nazism,” according to a 1999 history of newspaper columnists by Sam Riley. Allen almost certainly understood Hitler’s appeal to ordinary Germans, particularly veterans embittered by their sacrifices and the harsh peace imposed by the victors. Lacking an equal exposure to totalitarianism Soviet-style, he may have considered Moscow a bulwark against German militarism and revanchism, and an inevitable ally. Among the items in his recently declassified FBI file is a series of letters to the Bureau, urging them to investigate pieces of what he considered to be Nazi propaganda appearing in American newspapers.

Allen also may have rationalized his actions on the grounds that the Soviets were effectively paying him for an advance peek at what he was going to be publishing anyway. More often than not, the same information encoded in a cable to Moscow appeared later in his columns. For example, a little more than a week after Allen told Soviet intelligence that the Japanese were fortifying the Marshall Islands, he repeated the same information in a column. Sometimes the information he conveyed to his case officer had already been published. He publicly revealed that Norman Davis would not be secretary of state well before giving the information to Soviet intelligence; similarly, he circulated an editor’s note to “Merry-Go-Round” subscribers that broke his (erroneous) scoop about Sumner Welles before it went through Soviet reporting channels. Indeed, if Allen’s case officer came to realize that the columnist was mostly recycling or previewing information, that might have been sufficient to bring an abrupt end to the relationship—before Moscow Center found out it was, in effect, just another subscriber. On the other hand, it might have been the temperamental Allen who cut the relationship off when his case officer pressed for more.

Allen was not doing anything illegal—just highly unethical. It wasn’t until 1938 that the Foreign Agents Registration Act criminalized covert relationships with agencies of a foreign power for all citizens, not just government officials. Since Allen never provided Soviet intelligence with any secrets, he was not breaking the law. Nonetheless, his relationship with a foreign intelligence service was a serious breach of journalistic ethics. He acted covertly, sharing privileged information in exchange for money, and compromised the tacit understanding he was supposed to have with his readers and subscribers.

In the highly engrossing and controversial history of the American journalists who acted as agents for Soviet intelligence in the 1930s, Allen’s case remains a unique one. Unlike nearly all the others, Allen was not a lifelong liberal who, at one time or another, chose to flirt with, accept, or renounce Communism. This does not necessarily make him a mercenary. Instead, his story is a testament to the inconsistencies, adaptations and accommodations we all endure over the course of a lifetime.

Part of our endless fascination with spy stories is the way they cast light on the secrecies and hypocrisies of daily existence. Most men lead double lives, but for these men, those second, clandestine lives were palpable and tangible. We can investigate them, probing their hopes and ideals, and also their weaknesses, inadequacies and dilemmas. Although his time as an agent in the service of Soviet intelligence was brief, Robert Allen exemplified this strange human predicament as much as anyone.