I have just posted a pdf (17 MB file) of my article that appeared in MHQ last year about America's war strategy in the naval War of 1812.
It's a brief introduction to the themes and some of the key personalities (especially Secretary of the Navy William Jones, a man truly ahead of his time) that feature in my soon-to-appear history of the war Perilous Fight . . .
You can read the Publishers Weekly review of the book here.
Tuesday, November 23, 2010
Monday, November 15, 2010
Playing politics in war
A pdf of the talk I gave earlier this year at the Society for Military History conference: it's about the deft political strategy that allowed America to turn a military stalemate into an enduring victory in the War of 1812 — and, by contrast, how the British managed the precise opposite. (It helped that we were the underdog; in more recent wars, the United States has far more resembled the muscle-bound superpower of Great Britain in 1812.)
Thursday, November 4, 2010
Intelligent intelligence history
‘What’s the Use of Cryptologic History?’
STEPHEN BUDIANSKY
Continued secrecy surrounding signals intelligence during the Cold War has profoundly distorted the historical record, making a full and accurate military, diplomatic, and presidential history of the last half century impossible for now. Not only history, but the ongoing intelligence enterprise itself, would be better served by significantly greater openness. This article is adapted from the Schorreck Memorial Lecture in Cryptologic History, presented by the author at the National Cryptologic Museum, Ft. Meade, Maryland, 24 May 2010, and the National Security Agency, 26 May 2010
The work of intelligence, like the work of war and diplomacy it supports, is a complex and risky endeavor, in which failure and blame are always lurking just around the corner. At times the story of modern intelligence seems but a trail of woe, an endless narrative of mistakes.
It is an endeavor burdened all the more by the secrecy that is part and parcel to it. It is no slight upon the professionalism, dedication, or morality of practitioners of intelligence to point out, as did Lord Acton a century and a half ago, that ‘everything secret degenerates’ (1). Or, as Daniel Patrick Moynihan elaborated on this same point somewhat more recently, secrecy insulates us ‘from our own miscalculations’—and insures that our mistakes are ‘not open to correction’ (2).
That is another way of saying that history matters—and, in a democratic society, open access to historical knowledge matters especially. In fairness to the US intelligence agencies, while there are myriad examples one can point to of mistakes that were covered up and lessons that went unheeded, there are also a surprising number of instances in which the agencies forthrightly confronted their errors and endeavored to learn from them. Historical studies conducted by NSA and CIA reveal a willingness—at least at times—to engage in unsparing and self-critical assessments of past blunders and misconduct (3). Credit for this willingness to confront the past should be largely given to the courage and objectivity of the professional historians on their staffs. As CIA historian Nicholas Dujmovic noted, he is not infrequently ‘accused of dwelling too much . . . on the failures’ of his agency; but he has found that ‘most of us in the profession take these cases very much to heart, endeavoring to learn as much from them as we can’ (4).
There is also a caveat that must preface any argument for the importance of intelligence history: namely, that there is only so much that history can inform us about when it comes to the day-to-day operations of such a highly technical enterprise as modern intelligence gathering has become, in which technology, targets, and methods change at a dizzying and accelerating pace.
But there is one aspect of intelligence history that is always relevant, no matter how much technology and the world may change. Intelligence is one of those peculiar human endeavors in which merely getting the right answer is an insufficient condition for success, and much that happens beyond the walls of NSA and CIA for example determines the success or failure of the entire American intelligence enterprise. This aspect of intelligence history has less to do with how to conduct operations than with the institutions, ideas, and leadership that determine whether the right answer gets to the right people, and whether it is used by them (5). It involves questions such as:
• how the intelligence enterprise is organized so that what is collected is recognized and analyzed and relayed in time to the people who need it;
• whether those who need it—military, political, and diplomatic officials—are equipped to understand what intelligence can and cannot do, understand both its importance and its limitations;
• and, inescapably in a democracy, whether the public has realistic expectations for what intelligence can and cannot do, and whether it remains willing to extend the considerable trust to its government to exercise extraordinary powers in extraordinary secrecy.
And the number of times the same stories occur over and over in this part of intelligence history gives one more than a little pause. I would not suggest anything so trivial as ‘those who ignore history are condemned to repeat it’, nor even that history is reducible to a bullet-point list of ‘lessons learned’. And I am mindful of the historian John Lewis Gaddis’s cautionary advice that a person who looks only to history in choosing his course is like someone trying to drive a car with his eyes glued to the rearview mirror: It’s good to know where you’ve been, but it’s also a good way to end up in the ditch (6).
Still, it is striking how many times a phrase along the lines, ‘We had to relearn all of the lessons of . . .’ appears in the memoirs of military, diplomatic, and political leaders who have grappled with a crisis in intelligence organization, particularly that of signals intelligence—Sigint (7). The same mistakes of obstinate disregard of Sigint (by military commanders in particular) have often been replayed; likewise the same mistakes of overenthusiastic and contextless embrace of Sigint (by political leaders in particular). The same scandals have arisen; the same complaints have been uttered by the users and producers of intelligence about one another’s obtuseness; the same unrealistic expectations of success have led to the same hasty overreactions in the aftermath of notorious failures; the same fights to unify collection and analysis and to break down institutional barriers to the sharing of information have been fought.
All of these mistakes were, at least in part, a result of historical blindness. All could arguably have been averted had there been a more robust and open effort made to study, analyze, and disseminate the history of Sigint operations and the history of how Sigint has been incorporated into military, diplomatic, and presidential decision making, for good and for ill.
SIGINT HISTORY IN THE LARGER HISTORICAL CONSCIOUSNESS
Shortly after the end of World War II the British diplomat Sir Alexander Cadogan remarked that intelligence was the ‘missing dimension’ in international history (8). Historians knew that there was an elephant in the room, but it was a largely invisible elephant. With David Kahn’s book The Codebreakers and then the first trickles of revelations about Ultra beginning in the late 1970s came the emergence of cryptologic history, or more accurately Sigint history, as a whole new sub-field of historical inquiry. The release by NSA in 1996 of 1.3 million pages of Sigint material from World War II turned the trickle into a flood. It was what allowed me to write my history of Allied cryptography in World War II, Battle of Wits, and which launched my continuing interest in the subject. And for a while we saw a boom (or at least a ‘boomlet’) in Sigint studies, with conferences, new scholarly organizations, journals, academic and popular books. If probably 90 percent of the scholarship and writing was focused on World War II, still it seemed to promise bigger things to come.
Yet with only a few exceptions, Sigint history—by which I principally mean, à la Cadogan, the study of the role that signals intelligence has played in shaping major, international historical events—never really succeeded in penetrating the larger historical consciousness. The ‘missing dimension’ of signals intelligence is still not reflected in the way most professional historians think and write about history, certainly not to a degree that even remotely matches the evidently immense importance that Sigint has had in shaping the events of our age. It is not reflected in the military, diplomatic, and presidential histories that are read by the general public, by military leaders and future military leaders, and even by presidents and future presidents, and which form widely held conceptions and understanding of history in our society.
And it is no exaggeration to say that we are at a crisis point in the study of intelligence history in general, and signals intelligence history in particular: because there is a very real question of whether any serious historians and writers outside of the intelligence community are going to continue trying to research and understand and write about this subject at all. The cause of this crisis is simple: The practice and craft of history is a document-driven business, and the documents are not available. With the exception of a few highly selective releases (notably on limited aspects of Venona, the Korean War, and the Cuban missile crisis), the entire history of post–World War II signals intelligence in the US and Britain remains a firmly closed book (9). My fellow intelligence historian David Alvarez recently remarked to me that David Kahn may have the unique distinction of having created an entire new field of study, watched it blossom, and lived to see its demise. A search of the Historical Abstracts database and several other databases of published articles in the humanities and social sciences turned up fewer than two dozen articles in the last five years that even mentioned in passing cryptanalysis, Sigint, Comint, signals intelligence, or other related terms. Even the Journal of Military History, over the last decade, has published only four articles even mentioning these subjects; there were two in Foreign Affairs; there were literally zero in any of the premier journals of American or diplomatic history.
Military historians have long complained of being marginalized in the professional study of history; but they have nonetheless enjoyed a vibrant and flourishing field thanks not just to widespread public interest but also to no shortage of primary materials of all kinds that support new research, fresh interpretations, and different perspectives. Military history may get no respect in some parts of the academic world, but surely no one would call war the ‘missing dimension’ in international history.
But signals intelligence history is once again retreating into the shadows. The critical mass of scholars willing to invest the considerable energy required to master the technicalities of a complex and often difficult-to-understand subject is dwindling in the face of the impossibility of making a career in a field where the primary sources—notably nearly all documents relating to the post–World War II period—are locked away and no longer forthcoming.
THE ENDURING HISTORICAL DISTORTIONS OF SECRECY
World War II provides us with a vivid case study of what happens to the writing of history in the absence of access to the signals intelligence record. With hindsight we can see how the era before the Ultra revelations beginning in the 1970s were literally dark ages of World War II history.
The absence of the Sigint record did not merely leave a hole in this history: it left us with a downright fraudulent history, replete with deliberately fabricated cover stories that had been created at the time—and maintained for decades afterwards—to conceal the very existence of codebreaking and signals intelligence. We are still digging our way out from under this. Literally thousands upon thousands of contemporaneous documents (as well as nearly every single one of the memoirs of important military commanders and statesmen of this era) were written in ways deliberately intended to mislead. Hundreds and probably thousands of both popular and academic books and monographs written in the decades after the war repeated these deliberate distortions. Probably the greatest distortion that secrecy caused was to make a number of allied commanders look like intuitive military geniuses for their uncanny ability to anticipate the enemy’s plans—when in fact it was Sigint that really deserved the credit.
The dark ages that beset World War II history before the late 1970s still hold post–World War II history firmly in their grasp. The occasional glimmers that have peeked through the cracks have only emphasized the depth and vastness of the darkness that still enshrouds everything else. One glimmer came in 2008, when George Washington University’s National Security Archive posted on the Internet a highly redacted but nonetheless still fascinating internal NSA history that their researchers had obtained through a declassification request: Thomas R. Johnson’s American Cryptology During the Cold War, 1945–1989. It is especially fascinating for the hints it offers of what an accurate version of military, diplomatic, and presidential history of the post–World War II era might look like if historians actually had access to these crucial intelligence records. It also confirms how seriously the lack of access to these materials has indeed distorted our understanding of the history of the last half century.
For example: the battle for air superiority during the Korean War has been a subject of intense interest to air power historians. Korea was the first air superiority battle of the jet age, and in my book Air Power, I drew on several seemingly authoritative US Air Force studies which stated that the key turning point in the battle for air superiority over Korea came when fifty of the new, more-capable F-86 fighter jets were rushed to Korea to supplement the F-80s already in theater. Subsequently, US fighter pilots attained an extraordinary 13 to 1 kill ratio against MiG-15 jets, a result attributed to the superior performance and capabilities of the F-86 (10).
American Cryptology During the Cold War adds the missing dimension of Sigint. And it is not only an amazing story of how Sigint was employed in real-time on an ongoing operational basis, but it thoroughly rewrites the history of air power as well. In spring 1951 units of the Air Force Security Service began intercepting Soviet ground-control communications. These were communications from ground controllers to pilots telling them where to go and what to do, and they provided US commanders with a windfall of accurate, up-to-the-minute information on the whereabouts and numbers of enemy aircraft. Other signals were also being intercepted that helped supplement this information, including Chinese and Korean voice and Morse code messages from air defense units.
It was not until September 1951, however, that all of these intercept operations were brought together to a single location so the information could be rapidly correlated and the information communicated in real time to US tactical air controllers, who in turn were able to provide real-time warnings and targeting information to US fighter pilots in the air. This greatly extended the distance at which enemy aircraft could be located, well beyond the range provided by US radar, and it was a stunningly powerful force multiplier. Johnson quotes one official involved in the operation who stated at the time: ‘The present day top-heavy success of the F-86s against the MiG-15s dates almost from the day of the inception’ of this new integrated intercept and intelligence service (11).
Another example from the Korean War: It has been argued for some time by historians that General MacArthur and his G-2, Charles Willoughby, manipulated intelligence to conform to MacArthur’s desires and beliefs, and in particular MacArthur’s insistence that the Chinese would not intervene in Korea. David Halberstam’s 2007 book The Coldest Winter: America and the Korean War indeed contains a thoroughly damning indictment of MacArthur and Willoughby on this score. But it has certainly been a hotly debated point.
Again, the Johnson NSA history, drawing on the Sigint documentation, leaves little doubt where the truth lies. The evidence shows that beginning in June 1951 there was solid Sigint intelligence pointing to a massive movement of Chinese army forces to Manchuria. By July the Armed Forces Security Agency had identified six Chinese field armies on the Korean border. Other Sigint made clear the Chinese were preparing to enter the war (12).
So, regarding one of the most important historical questions about the course of the Korean War, regarding MacArthur’s competence as a commander and his arguably lifelong propensity for insubordination, regarding one of the most important lessons of modern history about the dangers of separating collection from analysis, and the dangers of manipulated intelligence—all this lies in documents that historians still do not have the full access to they need to produce a full and accurate account.
A third example from American Cryptology During the Cold War which points to the need for a sweeping revision of historical understanding involves the Vietnam War and Lyndon Johnson’s presidency: specifically LBJ’s fascination, fixation, obsession even, with raw Sigint and the confidence he placed in this source. It is clear from the NSA study that LBJ’s insistence on having a direct supply of Sigint to the White House was central to the daily decisions and the control that he exercised over the conduct of the war. (‘Lyndon Johnson was the most avid consumer of intelligence ever to occupy the White House,’ the study notes, and a ‘flood’ of raw Sigint was fed by NSA directly to the White House Situation Room (13).) From the Gulf of Tonkin through the Tet offensive, Sigint directed White House decision making, for better or worse. And in those two events that bookended the Johnson Administration’s tragic saga of Vietnam—the Gulf of Tonkin and Tet—it was clearly for the worse: the utter faith in Sigint led to grave mistakes in misreading enemy actions and intentions. In the Tonkin Gulf incident, LBJ, relying on an initial raw Sigint report, overruled the commander on the scene. The commander was dubious that North Vietnamese gunboats had in fact attacked his two destroyers during the previous night at all; he believed it may well have been a panicky false alarm on the part of the ships’ officers and crew. But an intercepted North Vietnamese message seemed to confirm that an attack had occurred, and within hours LBJ had ordered a retaliatory airstrike—and begun America’s fatal escalation in Vietnam. A few days later, however, it became clear that this intercepted message had been misinterpreted, that it referred to a different and much more minor incident of several days earlier. This is a crucial revelation that any history of the Vietnam War has to incorporate (14).
We now also know from this study that in the Tet offensive, Sigint in retrospect had only part of the picture; overreliance on this source by the White House and by General William Westmoreland led US forces to believe that the major blow would fall to the north. Too little effort was made to fill out the intelligence picture using other sources that might have given a truer prediction of what was to come (15).
As Thomas Johnson observes, ‘Sigint became the victim of its own success’ (16). That too is a recurring theme in the history of Sigint; the almost magical ability to read the actual words of the enemy has time and again blinded top officials to the limitations of Sigint—notably its inherent biases, lack of context, and incompleteness.
HISTORY, PUBLIC UNDERSTANDING, AND INTELLGENCE OPERATIONS
All three of these examples that I have drawn from the highly redacted release of American Cryptology During the Cold War are just glimmers. The documents referred to in the footnotes of this publication—the underlying primary sources—almost all remain classified. Scholars and writers outside of the intelligence community do not have access to them and are unlikely to any time soon under current policy. We have no way of evaluating the use of evidence in this released report; we have no way of seeing if the evidence would support other interpretations; we have no way to examine the evidence in greater depth.
And we can only guess at what other information crucial to the writing of a full and accurate history of war, diplomacy, and presidencies of this era still remains hidden altogether. The disconcerting thing is that we know enough even from these glimmers to have our confidence thoroughly shaken about the value of historical judgments made in the absence of the Sigint record. For many key events of the last fifty years, we literally still in all likelihood do not yet even really know what happened, why it happened, and what lessons we can correctly draw about the military, diplomatic, and presidential decisions that were made in the course of these events. Basic judgments about pivotal questions of war and peace, negotiating strategies, military tactics, allocations of budgets, still await being informed by the Sigint record. And the people who someday will be making these crucial judgments, who someday will be the crucial users of intelligence, are today reading histories that fail to convey a realistic sense of the role intelligence has played in past decisions.
It is difficult not to believe that there are many important historical events of the last fifty years about which our understanding would be fundamentally altered by a fuller access to the Sigint record. For example: Was it the strategic bombing of specific targets in Serbia, or was it the threat of a ground invasion, that finally induced Slobodan Milosevic to capitulate to NATO pressure in 1999? Were there other key turning points in the Vietnam War where Sigint played a crucial role in shaping White House policies? How did Sigint affect US policies on arms negotiations with the Soviets, the US response to the rise of the Solidarity movement in Poland, and other decisive events during the Cold War? We have had extremely selective releases on these topics. The writing of a true history awaits a full release.
Now one obvious reply to this argument is that the people who need to know do know; that members of Congressional committees and their staffs have the necessary clearances, as of course do key military and White House officials, so it is ‘only’ us, the members of the public, who are being misled or kept in the dark.
The problem with this argument is that it is a profoundly dangerous mistake to underestimate the power of popular understanding of history in a democratic society. Popular understanding—or misunderstanding—of the causes and conduct of World War I, World War II, the Vietnam War shaped the beliefs of generations, and not just among ordinary Americans but also military officers, top-ranking White House advisors, and indeed presidents themselves. Some dangerously wrong or overly simplistic ‘lessons learned’ from past wars have had tragic and enduring consequences. The specter of war triggered by the failure of disarmament haunted the post–World War I generation; the specter of Munich haunted the post–World War II generation; the specter of American military impotence haunted the post–Vietnam War generation. Getting the history right is not just an exercise in satisfying academic curiosity. The understanding of history has profound and real consequences.
Getting it right matters also to the work of the intelligence enterprise itself. The number of collection successes that were then undermined by analysis failures or political failures is perhaps the most striking feature of Thomas Johnson’s history of NSA in the Cold War. The Chinese intervention in Korea, the Soviet invasion of Hungary, and arguably the Soviet invasion of Afghanistan were all major events for which Sigint provided plausible warnings that somehow or other failed to be heeded (17). To this we might well add the 9/11 terrorist attacks.
It is also striking how the same organizational issues arise again and again in the history of intelligence operations: a lack of linguists; a dysfunctional disconnect between collection and analysis; turf battles and compartmentalization impeding coordination of intelligence sources; a mutual mistrust or incomprehension between users of intelligence and producers of intelligence.
This is history that the next generation of citizens, but even more the next generation of politicians, government officials, military officers needs to absorb. Without a ‘Sigint-history-mindedness’ among consumers of intelligence that is second nature—exactly the way that the study of battle from Thermopylae to Afghanistan instills a sense of the unchanging lessons of tactics and terrain that is second nature to every generation of rising military officers-to-be—we are likely to keep seeing these same mistakes in the use of intelligence, and in the organization of the intelligence enterprise, repeated.
Here again, secrecy is the great enemy. Almost every single one of the historical lessons that emerges from the post–World War II Sigint history that we do have access to is a lesson that, were it more openly and widely disseminated, would actually serve the interests of intelligence; would help the intelligence community do its job more effectively; would counter some of the simplistic and knee-jerk ‘solutions’ that tend to be imposed when things go wrong; would increase the likelihood that senior government officials who receive Sigint make effective, wise, and judicious use of it; would help users and producers of intelligence speak the same language; and might even have averted many of the self-inflicted wounds suffered by the intelligence community in the last half century. It is still difficult to fathom how anyone with a historical memory of the revelations of the Church Committee in 1975, and the effects they had, could have imagined that the Bush Administration’s decision to evade the requirements of statutory law in ordering domestic surveillance would not inevitably become public knowledge, and inevitably produce a backlash as they did.
(It is also worthwhile to consider that the public most often learns about the inner workings of the intelligence agencies in connection with scandal and failure. A more realistic, ongoing picture of how intelligence agencies operate, and of the difficulties and complexities and constraints that the process of intelligence gathering and analysis faces, might do much to avert a rush to judgment when the process proves to be fallible—as it always will on occasion. And in this sense, the manner in which historical releases have been cherry-picked by NSA to emphasize past triumphs like the Battle of Midway has arguably been counter-productive, in giving an unrealistic sense of seamless omnipotence, infallibility, and easy success. A final point worth considering about the damaging consequences of secrecy is, as the journalist Paul McMasters observes, ‘The government’s obsession with secrecy creates a citizen’s obsession with conspiracy’ (18).)
There is one other important conclusion that emerges from Thomas Johnson’s study: and that is the manifest fact—given what NSA agreed to declassify in this report—that the crucial issues posed by recent intelligence history can for the most part be openly discussed without compromising vital sources and methods. As Senator Ron Wyden, a member of the Senate Intelligence Committee, recently remarked: ‘Most members of the public do not expect to have detailed information about how intelligence collection is actually conducted, but they do expect to understand the boundaries of what the law does and does not allow, so that they can ratify or reject the decisions that public officials make on their behalf’ (19).
Daniel Moynihan, rather more pithily, once said, ‘Secrecy is for losers’ (20). I would not go quite that far myself, but I wish at least a bit of that sentiment was somewhere a part of the system. Today, all of the institutional inertia is on the side of secrecy. (No one ever got in trouble for not releasing a document.) We know the costs of indiscretion when it comes to the compromise of vital secrets. But we also somehow need to find a way to be equally mindful of the costs of secrecy. Because there is a cost that we all pay for needless secrecy, in the sacrifice of historical understanding that results.
But the main thought I want to end with is that intelligence history is not just vital for history: it is vital for intelligence, too.
Notes
1. Josef L. Althoz, Damian McElrath, and James C. Holland (eds.) The Correspondence of Lord Acton and Richard Simpson (Cambridge: Cambridge University Press 1973), p. I: 114.
2. Daniel Patrick Moynihan, Secrecy (New Haven, Conn.: Yale University Press 1998), p. 8.
3. A number of self-critical CIA and NSA internal histories have become publicly known either through leaks or through official declassification: these include CIA Inspector General Lyman B. Kirkpatrick Jr.’s examination of the Bay of Pigs fiasco, discussed in Michael Warner, ‘The CIA’s Internal Probe of the Bay of Pigs Affair’, Studies in Intelligence (Winter 1998–99); a CIA account of a bungled 1952 spy mission to China, discussed in Nicholas Dujmovic, ‘Two CIA Prisoners in China, 1952–73’, Studies in Intelligence 50/4 (2006)); and Thomas R. Johnson’s history of NSA activities in the Cold War, discussed below.
4. Nicholas Dujmovic, ‘Review of “Legacy of Ashes: The History of the CIA”’, Studies in Intelligence 51/3 (2007).
5. See, for example, the discussion of the deleterious effects of ‘intelligence barriers’ on Cold War and counterterrorism intelligence operations in Michael Herman, ‘Counter-Terrorism, Information Technology and Intelligence Change’, Intelligence and National Security 18/4 (Winter 2003), pp. 40–58.
6. Gaddis quoted in Margaret MacMillan, Dangerous Games: The Uses and Abuses of History (New York: Modern Library 2009), p. 141.
7. See, for example, the observations of General O. P. Weyland, the Far East Air Forces commander in Korea, quoted in Robert F. Futtrell, The United States Air Force in Korea, 1950–1953 (New York: Duell, Sloan and Pearce 1961), p. xvii.
8. David Dilks (ed.) The Diaries of Sir Alexander Cadogan, O.M. 1938–1945 (London: Cassell 1971), p. 21.
9. NSA’s very selective releases of post–World War II Sigint history have largely come in the form of brochures and monographs published by its Center for Cryptologic History; these include David A. Hatch and Robert Louis Benson, The Korean War: The SIGINT Background (2000); Thomas R. Johnson and David A. Hatch, NSA and the Cuban Missile Crisis (1998); and Robert Louis Benson and Michael Warner, VENONA: Soviet Espionage and the American Response, 1939–1957 (1996).
10. USAF Historical Division, United States Air Force Operations in the Korean Conflict, 1 November 1950–30 June 1952 (Washington 1955), pp. 107–10, 116–18.
11. Thomas R. Johnson, American Cryptology During the Cold War, 1945–1989 (Center for Cryptologic History, National Security Agency 1995), pp. I: 48–49.
12. Ibid., pp. I: 43–46.
13. Ibid., pp. II: 353–54.
14. Ibid., pp. II: 516–23.
15. Ibid., pp. II: 559–61.
16. Ibid., p. II: 565.
17. Ibid., pp. I: 43–46, 235, III: 253–54.
18. McMasters quoted in Moynihan, Secrecy, p. 218.
19. Congressional Record, 25 March 2010, pp. S2108–2109.
20. Moynihan, Secrecy, p. 1.
STEPHEN BUDIANSKY
Continued secrecy surrounding signals intelligence during the Cold War has profoundly distorted the historical record, making a full and accurate military, diplomatic, and presidential history of the last half century impossible for now. Not only history, but the ongoing intelligence enterprise itself, would be better served by significantly greater openness. This article is adapted from the Schorreck Memorial Lecture in Cryptologic History, presented by the author at the National Cryptologic Museum, Ft. Meade, Maryland, 24 May 2010, and the National Security Agency, 26 May 2010
The work of intelligence, like the work of war and diplomacy it supports, is a complex and risky endeavor, in which failure and blame are always lurking just around the corner. At times the story of modern intelligence seems but a trail of woe, an endless narrative of mistakes.
It is an endeavor burdened all the more by the secrecy that is part and parcel to it. It is no slight upon the professionalism, dedication, or morality of practitioners of intelligence to point out, as did Lord Acton a century and a half ago, that ‘everything secret degenerates’ (1). Or, as Daniel Patrick Moynihan elaborated on this same point somewhat more recently, secrecy insulates us ‘from our own miscalculations’—and insures that our mistakes are ‘not open to correction’ (2).
That is another way of saying that history matters—and, in a democratic society, open access to historical knowledge matters especially. In fairness to the US intelligence agencies, while there are myriad examples one can point to of mistakes that were covered up and lessons that went unheeded, there are also a surprising number of instances in which the agencies forthrightly confronted their errors and endeavored to learn from them. Historical studies conducted by NSA and CIA reveal a willingness—at least at times—to engage in unsparing and self-critical assessments of past blunders and misconduct (3). Credit for this willingness to confront the past should be largely given to the courage and objectivity of the professional historians on their staffs. As CIA historian Nicholas Dujmovic noted, he is not infrequently ‘accused of dwelling too much . . . on the failures’ of his agency; but he has found that ‘most of us in the profession take these cases very much to heart, endeavoring to learn as much from them as we can’ (4).
There is also a caveat that must preface any argument for the importance of intelligence history: namely, that there is only so much that history can inform us about when it comes to the day-to-day operations of such a highly technical enterprise as modern intelligence gathering has become, in which technology, targets, and methods change at a dizzying and accelerating pace.
But there is one aspect of intelligence history that is always relevant, no matter how much technology and the world may change. Intelligence is one of those peculiar human endeavors in which merely getting the right answer is an insufficient condition for success, and much that happens beyond the walls of NSA and CIA for example determines the success or failure of the entire American intelligence enterprise. This aspect of intelligence history has less to do with how to conduct operations than with the institutions, ideas, and leadership that determine whether the right answer gets to the right people, and whether it is used by them (5). It involves questions such as:
• how the intelligence enterprise is organized so that what is collected is recognized and analyzed and relayed in time to the people who need it;
• whether those who need it—military, political, and diplomatic officials—are equipped to understand what intelligence can and cannot do, understand both its importance and its limitations;
• and, inescapably in a democracy, whether the public has realistic expectations for what intelligence can and cannot do, and whether it remains willing to extend the considerable trust to its government to exercise extraordinary powers in extraordinary secrecy.
And the number of times the same stories occur over and over in this part of intelligence history gives one more than a little pause. I would not suggest anything so trivial as ‘those who ignore history are condemned to repeat it’, nor even that history is reducible to a bullet-point list of ‘lessons learned’. And I am mindful of the historian John Lewis Gaddis’s cautionary advice that a person who looks only to history in choosing his course is like someone trying to drive a car with his eyes glued to the rearview mirror: It’s good to know where you’ve been, but it’s also a good way to end up in the ditch (6).
Still, it is striking how many times a phrase along the lines, ‘We had to relearn all of the lessons of . . .’ appears in the memoirs of military, diplomatic, and political leaders who have grappled with a crisis in intelligence organization, particularly that of signals intelligence—Sigint (7). The same mistakes of obstinate disregard of Sigint (by military commanders in particular) have often been replayed; likewise the same mistakes of overenthusiastic and contextless embrace of Sigint (by political leaders in particular). The same scandals have arisen; the same complaints have been uttered by the users and producers of intelligence about one another’s obtuseness; the same unrealistic expectations of success have led to the same hasty overreactions in the aftermath of notorious failures; the same fights to unify collection and analysis and to break down institutional barriers to the sharing of information have been fought.
All of these mistakes were, at least in part, a result of historical blindness. All could arguably have been averted had there been a more robust and open effort made to study, analyze, and disseminate the history of Sigint operations and the history of how Sigint has been incorporated into military, diplomatic, and presidential decision making, for good and for ill.
SIGINT HISTORY IN THE LARGER HISTORICAL CONSCIOUSNESS
Shortly after the end of World War II the British diplomat Sir Alexander Cadogan remarked that intelligence was the ‘missing dimension’ in international history (8). Historians knew that there was an elephant in the room, but it was a largely invisible elephant. With David Kahn’s book The Codebreakers and then the first trickles of revelations about Ultra beginning in the late 1970s came the emergence of cryptologic history, or more accurately Sigint history, as a whole new sub-field of historical inquiry. The release by NSA in 1996 of 1.3 million pages of Sigint material from World War II turned the trickle into a flood. It was what allowed me to write my history of Allied cryptography in World War II, Battle of Wits, and which launched my continuing interest in the subject. And for a while we saw a boom (or at least a ‘boomlet’) in Sigint studies, with conferences, new scholarly organizations, journals, academic and popular books. If probably 90 percent of the scholarship and writing was focused on World War II, still it seemed to promise bigger things to come.
Yet with only a few exceptions, Sigint history—by which I principally mean, à la Cadogan, the study of the role that signals intelligence has played in shaping major, international historical events—never really succeeded in penetrating the larger historical consciousness. The ‘missing dimension’ of signals intelligence is still not reflected in the way most professional historians think and write about history, certainly not to a degree that even remotely matches the evidently immense importance that Sigint has had in shaping the events of our age. It is not reflected in the military, diplomatic, and presidential histories that are read by the general public, by military leaders and future military leaders, and even by presidents and future presidents, and which form widely held conceptions and understanding of history in our society.
And it is no exaggeration to say that we are at a crisis point in the study of intelligence history in general, and signals intelligence history in particular: because there is a very real question of whether any serious historians and writers outside of the intelligence community are going to continue trying to research and understand and write about this subject at all. The cause of this crisis is simple: The practice and craft of history is a document-driven business, and the documents are not available. With the exception of a few highly selective releases (notably on limited aspects of Venona, the Korean War, and the Cuban missile crisis), the entire history of post–World War II signals intelligence in the US and Britain remains a firmly closed book (9). My fellow intelligence historian David Alvarez recently remarked to me that David Kahn may have the unique distinction of having created an entire new field of study, watched it blossom, and lived to see its demise. A search of the Historical Abstracts database and several other databases of published articles in the humanities and social sciences turned up fewer than two dozen articles in the last five years that even mentioned in passing cryptanalysis, Sigint, Comint, signals intelligence, or other related terms. Even the Journal of Military History, over the last decade, has published only four articles even mentioning these subjects; there were two in Foreign Affairs; there were literally zero in any of the premier journals of American or diplomatic history.
Military historians have long complained of being marginalized in the professional study of history; but they have nonetheless enjoyed a vibrant and flourishing field thanks not just to widespread public interest but also to no shortage of primary materials of all kinds that support new research, fresh interpretations, and different perspectives. Military history may get no respect in some parts of the academic world, but surely no one would call war the ‘missing dimension’ in international history.
But signals intelligence history is once again retreating into the shadows. The critical mass of scholars willing to invest the considerable energy required to master the technicalities of a complex and often difficult-to-understand subject is dwindling in the face of the impossibility of making a career in a field where the primary sources—notably nearly all documents relating to the post–World War II period—are locked away and no longer forthcoming.
THE ENDURING HISTORICAL DISTORTIONS OF SECRECY
World War II provides us with a vivid case study of what happens to the writing of history in the absence of access to the signals intelligence record. With hindsight we can see how the era before the Ultra revelations beginning in the 1970s were literally dark ages of World War II history.
The absence of the Sigint record did not merely leave a hole in this history: it left us with a downright fraudulent history, replete with deliberately fabricated cover stories that had been created at the time—and maintained for decades afterwards—to conceal the very existence of codebreaking and signals intelligence. We are still digging our way out from under this. Literally thousands upon thousands of contemporaneous documents (as well as nearly every single one of the memoirs of important military commanders and statesmen of this era) were written in ways deliberately intended to mislead. Hundreds and probably thousands of both popular and academic books and monographs written in the decades after the war repeated these deliberate distortions. Probably the greatest distortion that secrecy caused was to make a number of allied commanders look like intuitive military geniuses for their uncanny ability to anticipate the enemy’s plans—when in fact it was Sigint that really deserved the credit.
The dark ages that beset World War II history before the late 1970s still hold post–World War II history firmly in their grasp. The occasional glimmers that have peeked through the cracks have only emphasized the depth and vastness of the darkness that still enshrouds everything else. One glimmer came in 2008, when George Washington University’s National Security Archive posted on the Internet a highly redacted but nonetheless still fascinating internal NSA history that their researchers had obtained through a declassification request: Thomas R. Johnson’s American Cryptology During the Cold War, 1945–1989. It is especially fascinating for the hints it offers of what an accurate version of military, diplomatic, and presidential history of the post–World War II era might look like if historians actually had access to these crucial intelligence records. It also confirms how seriously the lack of access to these materials has indeed distorted our understanding of the history of the last half century.
For example: the battle for air superiority during the Korean War has been a subject of intense interest to air power historians. Korea was the first air superiority battle of the jet age, and in my book Air Power, I drew on several seemingly authoritative US Air Force studies which stated that the key turning point in the battle for air superiority over Korea came when fifty of the new, more-capable F-86 fighter jets were rushed to Korea to supplement the F-80s already in theater. Subsequently, US fighter pilots attained an extraordinary 13 to 1 kill ratio against MiG-15 jets, a result attributed to the superior performance and capabilities of the F-86 (10).
American Cryptology During the Cold War adds the missing dimension of Sigint. And it is not only an amazing story of how Sigint was employed in real-time on an ongoing operational basis, but it thoroughly rewrites the history of air power as well. In spring 1951 units of the Air Force Security Service began intercepting Soviet ground-control communications. These were communications from ground controllers to pilots telling them where to go and what to do, and they provided US commanders with a windfall of accurate, up-to-the-minute information on the whereabouts and numbers of enemy aircraft. Other signals were also being intercepted that helped supplement this information, including Chinese and Korean voice and Morse code messages from air defense units.
It was not until September 1951, however, that all of these intercept operations were brought together to a single location so the information could be rapidly correlated and the information communicated in real time to US tactical air controllers, who in turn were able to provide real-time warnings and targeting information to US fighter pilots in the air. This greatly extended the distance at which enemy aircraft could be located, well beyond the range provided by US radar, and it was a stunningly powerful force multiplier. Johnson quotes one official involved in the operation who stated at the time: ‘The present day top-heavy success of the F-86s against the MiG-15s dates almost from the day of the inception’ of this new integrated intercept and intelligence service (11).
Another example from the Korean War: It has been argued for some time by historians that General MacArthur and his G-2, Charles Willoughby, manipulated intelligence to conform to MacArthur’s desires and beliefs, and in particular MacArthur’s insistence that the Chinese would not intervene in Korea. David Halberstam’s 2007 book The Coldest Winter: America and the Korean War indeed contains a thoroughly damning indictment of MacArthur and Willoughby on this score. But it has certainly been a hotly debated point.
Again, the Johnson NSA history, drawing on the Sigint documentation, leaves little doubt where the truth lies. The evidence shows that beginning in June 1951 there was solid Sigint intelligence pointing to a massive movement of Chinese army forces to Manchuria. By July the Armed Forces Security Agency had identified six Chinese field armies on the Korean border. Other Sigint made clear the Chinese were preparing to enter the war (12).
So, regarding one of the most important historical questions about the course of the Korean War, regarding MacArthur’s competence as a commander and his arguably lifelong propensity for insubordination, regarding one of the most important lessons of modern history about the dangers of separating collection from analysis, and the dangers of manipulated intelligence—all this lies in documents that historians still do not have the full access to they need to produce a full and accurate account.
A third example from American Cryptology During the Cold War which points to the need for a sweeping revision of historical understanding involves the Vietnam War and Lyndon Johnson’s presidency: specifically LBJ’s fascination, fixation, obsession even, with raw Sigint and the confidence he placed in this source. It is clear from the NSA study that LBJ’s insistence on having a direct supply of Sigint to the White House was central to the daily decisions and the control that he exercised over the conduct of the war. (‘Lyndon Johnson was the most avid consumer of intelligence ever to occupy the White House,’ the study notes, and a ‘flood’ of raw Sigint was fed by NSA directly to the White House Situation Room (13).) From the Gulf of Tonkin through the Tet offensive, Sigint directed White House decision making, for better or worse. And in those two events that bookended the Johnson Administration’s tragic saga of Vietnam—the Gulf of Tonkin and Tet—it was clearly for the worse: the utter faith in Sigint led to grave mistakes in misreading enemy actions and intentions. In the Tonkin Gulf incident, LBJ, relying on an initial raw Sigint report, overruled the commander on the scene. The commander was dubious that North Vietnamese gunboats had in fact attacked his two destroyers during the previous night at all; he believed it may well have been a panicky false alarm on the part of the ships’ officers and crew. But an intercepted North Vietnamese message seemed to confirm that an attack had occurred, and within hours LBJ had ordered a retaliatory airstrike—and begun America’s fatal escalation in Vietnam. A few days later, however, it became clear that this intercepted message had been misinterpreted, that it referred to a different and much more minor incident of several days earlier. This is a crucial revelation that any history of the Vietnam War has to incorporate (14).
We now also know from this study that in the Tet offensive, Sigint in retrospect had only part of the picture; overreliance on this source by the White House and by General William Westmoreland led US forces to believe that the major blow would fall to the north. Too little effort was made to fill out the intelligence picture using other sources that might have given a truer prediction of what was to come (15).
As Thomas Johnson observes, ‘Sigint became the victim of its own success’ (16). That too is a recurring theme in the history of Sigint; the almost magical ability to read the actual words of the enemy has time and again blinded top officials to the limitations of Sigint—notably its inherent biases, lack of context, and incompleteness.
HISTORY, PUBLIC UNDERSTANDING, AND INTELLGENCE OPERATIONS
All three of these examples that I have drawn from the highly redacted release of American Cryptology During the Cold War are just glimmers. The documents referred to in the footnotes of this publication—the underlying primary sources—almost all remain classified. Scholars and writers outside of the intelligence community do not have access to them and are unlikely to any time soon under current policy. We have no way of evaluating the use of evidence in this released report; we have no way of seeing if the evidence would support other interpretations; we have no way to examine the evidence in greater depth.
And we can only guess at what other information crucial to the writing of a full and accurate history of war, diplomacy, and presidencies of this era still remains hidden altogether. The disconcerting thing is that we know enough even from these glimmers to have our confidence thoroughly shaken about the value of historical judgments made in the absence of the Sigint record. For many key events of the last fifty years, we literally still in all likelihood do not yet even really know what happened, why it happened, and what lessons we can correctly draw about the military, diplomatic, and presidential decisions that were made in the course of these events. Basic judgments about pivotal questions of war and peace, negotiating strategies, military tactics, allocations of budgets, still await being informed by the Sigint record. And the people who someday will be making these crucial judgments, who someday will be the crucial users of intelligence, are today reading histories that fail to convey a realistic sense of the role intelligence has played in past decisions.
It is difficult not to believe that there are many important historical events of the last fifty years about which our understanding would be fundamentally altered by a fuller access to the Sigint record. For example: Was it the strategic bombing of specific targets in Serbia, or was it the threat of a ground invasion, that finally induced Slobodan Milosevic to capitulate to NATO pressure in 1999? Were there other key turning points in the Vietnam War where Sigint played a crucial role in shaping White House policies? How did Sigint affect US policies on arms negotiations with the Soviets, the US response to the rise of the Solidarity movement in Poland, and other decisive events during the Cold War? We have had extremely selective releases on these topics. The writing of a true history awaits a full release.
Now one obvious reply to this argument is that the people who need to know do know; that members of Congressional committees and their staffs have the necessary clearances, as of course do key military and White House officials, so it is ‘only’ us, the members of the public, who are being misled or kept in the dark.
The problem with this argument is that it is a profoundly dangerous mistake to underestimate the power of popular understanding of history in a democratic society. Popular understanding—or misunderstanding—of the causes and conduct of World War I, World War II, the Vietnam War shaped the beliefs of generations, and not just among ordinary Americans but also military officers, top-ranking White House advisors, and indeed presidents themselves. Some dangerously wrong or overly simplistic ‘lessons learned’ from past wars have had tragic and enduring consequences. The specter of war triggered by the failure of disarmament haunted the post–World War I generation; the specter of Munich haunted the post–World War II generation; the specter of American military impotence haunted the post–Vietnam War generation. Getting the history right is not just an exercise in satisfying academic curiosity. The understanding of history has profound and real consequences.
Getting it right matters also to the work of the intelligence enterprise itself. The number of collection successes that were then undermined by analysis failures or political failures is perhaps the most striking feature of Thomas Johnson’s history of NSA in the Cold War. The Chinese intervention in Korea, the Soviet invasion of Hungary, and arguably the Soviet invasion of Afghanistan were all major events for which Sigint provided plausible warnings that somehow or other failed to be heeded (17). To this we might well add the 9/11 terrorist attacks.
It is also striking how the same organizational issues arise again and again in the history of intelligence operations: a lack of linguists; a dysfunctional disconnect between collection and analysis; turf battles and compartmentalization impeding coordination of intelligence sources; a mutual mistrust or incomprehension between users of intelligence and producers of intelligence.
This is history that the next generation of citizens, but even more the next generation of politicians, government officials, military officers needs to absorb. Without a ‘Sigint-history-mindedness’ among consumers of intelligence that is second nature—exactly the way that the study of battle from Thermopylae to Afghanistan instills a sense of the unchanging lessons of tactics and terrain that is second nature to every generation of rising military officers-to-be—we are likely to keep seeing these same mistakes in the use of intelligence, and in the organization of the intelligence enterprise, repeated.
Here again, secrecy is the great enemy. Almost every single one of the historical lessons that emerges from the post–World War II Sigint history that we do have access to is a lesson that, were it more openly and widely disseminated, would actually serve the interests of intelligence; would help the intelligence community do its job more effectively; would counter some of the simplistic and knee-jerk ‘solutions’ that tend to be imposed when things go wrong; would increase the likelihood that senior government officials who receive Sigint make effective, wise, and judicious use of it; would help users and producers of intelligence speak the same language; and might even have averted many of the self-inflicted wounds suffered by the intelligence community in the last half century. It is still difficult to fathom how anyone with a historical memory of the revelations of the Church Committee in 1975, and the effects they had, could have imagined that the Bush Administration’s decision to evade the requirements of statutory law in ordering domestic surveillance would not inevitably become public knowledge, and inevitably produce a backlash as they did.
(It is also worthwhile to consider that the public most often learns about the inner workings of the intelligence agencies in connection with scandal and failure. A more realistic, ongoing picture of how intelligence agencies operate, and of the difficulties and complexities and constraints that the process of intelligence gathering and analysis faces, might do much to avert a rush to judgment when the process proves to be fallible—as it always will on occasion. And in this sense, the manner in which historical releases have been cherry-picked by NSA to emphasize past triumphs like the Battle of Midway has arguably been counter-productive, in giving an unrealistic sense of seamless omnipotence, infallibility, and easy success. A final point worth considering about the damaging consequences of secrecy is, as the journalist Paul McMasters observes, ‘The government’s obsession with secrecy creates a citizen’s obsession with conspiracy’ (18).)
There is one other important conclusion that emerges from Thomas Johnson’s study: and that is the manifest fact—given what NSA agreed to declassify in this report—that the crucial issues posed by recent intelligence history can for the most part be openly discussed without compromising vital sources and methods. As Senator Ron Wyden, a member of the Senate Intelligence Committee, recently remarked: ‘Most members of the public do not expect to have detailed information about how intelligence collection is actually conducted, but they do expect to understand the boundaries of what the law does and does not allow, so that they can ratify or reject the decisions that public officials make on their behalf’ (19).
Daniel Moynihan, rather more pithily, once said, ‘Secrecy is for losers’ (20). I would not go quite that far myself, but I wish at least a bit of that sentiment was somewhere a part of the system. Today, all of the institutional inertia is on the side of secrecy. (No one ever got in trouble for not releasing a document.) We know the costs of indiscretion when it comes to the compromise of vital secrets. But we also somehow need to find a way to be equally mindful of the costs of secrecy. Because there is a cost that we all pay for needless secrecy, in the sacrifice of historical understanding that results.
But the main thought I want to end with is that intelligence history is not just vital for history: it is vital for intelligence, too.
Notes
1. Josef L. Althoz, Damian McElrath, and James C. Holland (eds.) The Correspondence of Lord Acton and Richard Simpson (Cambridge: Cambridge University Press 1973), p. I: 114.
2. Daniel Patrick Moynihan, Secrecy (New Haven, Conn.: Yale University Press 1998), p. 8.
3. A number of self-critical CIA and NSA internal histories have become publicly known either through leaks or through official declassification: these include CIA Inspector General Lyman B. Kirkpatrick Jr.’s examination of the Bay of Pigs fiasco, discussed in Michael Warner, ‘The CIA’s Internal Probe of the Bay of Pigs Affair’, Studies in Intelligence (Winter 1998–99); a CIA account of a bungled 1952 spy mission to China, discussed in Nicholas Dujmovic, ‘Two CIA Prisoners in China, 1952–73’, Studies in Intelligence 50/4 (2006)); and Thomas R. Johnson’s history of NSA activities in the Cold War, discussed below.
4. Nicholas Dujmovic, ‘Review of “Legacy of Ashes: The History of the CIA”’, Studies in Intelligence 51/3 (2007).
5. See, for example, the discussion of the deleterious effects of ‘intelligence barriers’ on Cold War and counterterrorism intelligence operations in Michael Herman, ‘Counter-Terrorism, Information Technology and Intelligence Change’, Intelligence and National Security 18/4 (Winter 2003), pp. 40–58.
6. Gaddis quoted in Margaret MacMillan, Dangerous Games: The Uses and Abuses of History (New York: Modern Library 2009), p. 141.
7. See, for example, the observations of General O. P. Weyland, the Far East Air Forces commander in Korea, quoted in Robert F. Futtrell, The United States Air Force in Korea, 1950–1953 (New York: Duell, Sloan and Pearce 1961), p. xvii.
8. David Dilks (ed.) The Diaries of Sir Alexander Cadogan, O.M. 1938–1945 (London: Cassell 1971), p. 21.
9. NSA’s very selective releases of post–World War II Sigint history have largely come in the form of brochures and monographs published by its Center for Cryptologic History; these include David A. Hatch and Robert Louis Benson, The Korean War: The SIGINT Background (2000); Thomas R. Johnson and David A. Hatch, NSA and the Cuban Missile Crisis (1998); and Robert Louis Benson and Michael Warner, VENONA: Soviet Espionage and the American Response, 1939–1957 (1996).
10. USAF Historical Division, United States Air Force Operations in the Korean Conflict, 1 November 1950–30 June 1952 (Washington 1955), pp. 107–10, 116–18.
11. Thomas R. Johnson, American Cryptology During the Cold War, 1945–1989 (Center for Cryptologic History, National Security Agency 1995), pp. I: 48–49.
12. Ibid., pp. I: 43–46.
13. Ibid., pp. II: 353–54.
14. Ibid., pp. II: 516–23.
15. Ibid., pp. II: 559–61.
16. Ibid., p. II: 565.
17. Ibid., pp. I: 43–46, 235, III: 253–54.
18. McMasters quoted in Moynihan, Secrecy, p. 218.
19. Congressional Record, 25 March 2010, pp. S2108–2109.
20. Moynihan, Secrecy, p. 1.
Subscribe to:
Posts (Atom)