Original Research ARTICLE
Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis
- 1University of North Carolina at Chapel Hill, United States
- 2London School of Economics and Political Science, United Kingdom
- 3Imperial College London, United Kingdom
- 4School of BioSciences, University of Melbourne, Australia
- 5School of Historical and Philosophical Studies, University of Melbourne, Australia
Background. The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq's Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the US Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products.
Method. Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments.
Results. Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability.
Conclusions. Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.
Keywords: intelligence analysis, intelligence failures, Intelligence reform, IRTPA , ICD203 , Tradecraft standards , inter-rater reliability
Received: 31 Aug 2018;
Accepted: 07 Dec 2018.
Edited by:Nathan Dieckmann, Oregon Health & Science University, United States
Reviewed by:Daniel Benjamin, University of Southern California, United States
James Kajdasz, United States Air Force Academy, United States
Copyright: © 2018 Marcoci, Burgman, Kruger, Silver, McBride, Thorn, Fraser, Wintle, Fidler and Vercammen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
* Correspondence: Dr. Alexandru Marcoci, University of North Carolina at Chapel Hill, Chapel Hill, 27599, North Carolina, United States, firstname.lastname@example.org