Article originally published in The California Tech, January 26, 2015
Why do smart individuals form stupid organizations?
The Dunning-Kruger effect can be paraphrased as "too stupid to realize it." In more depth, an individual displaying the Dunning-Kruger effect is so ignorant of their ignorance, their self-assessed competence is wildly discordant with reality. Everyone displays this effect to some extent in areas beyond their interests or expertise, but examples of universal affliction are mercilessly rare, especially at an institution like Caltech, and especially not in YouTube comments. In general, people have the ability to incorporate new information and to self-assess and self-correct as necessary.
If we look at organizations, however, this generality becomes a rarity. There are many reasons for this. Top-down organizations often use a simply-connected org chart and have many single points of failure. Throw in an incentive scheme that actively impedes important information moving up the chain and it's frequently surprising that large organizations can function at all. As an example, problems in the supply chain outsourcing Boeing employed to build the 787 were predicted more than 15 years ago, and it took at least a decade before senior management did anything about it. Sony has been hacked half a dozen times with no apparent change in corporate culture. The most recent Iraq war was initiated on transparently false grounds, and none of our checks and balances were able to pull the US back from the brink. On the other hand, Google holds weekly company-wide meetings where anyone can ask a question. Elon Musk (CEO of SpaceX and Tesla Motors) has recommended to "really pay attention to negative feedback, and solicit it, especially from friends."
Last week I wrote about Media FBI burglary. The FBI, run for nearly half a century by J. Edgar Hoover, became primarily obsessed with furthering its own agenda to the detriment of its supposed mission. The actions and labor of its individual agents were co-opted by an organization that was never accountable to the people it served. With no avenue for internal dissent, the FBI could and did run roughshod across the freedoms it was supposed to protect. In the end, only the actions of a handful of motivated activists and journalists were able to restore the flow of communication and enable the continuance of democratic functionality.
But it is too easy to point the finger and blame the police. Often, we read a portion of daily outrage at the conduct of some state-empowered law enforcement. Often, it seems that the actions of a few bad cops are enabled by corrupt or dysfunctional leadership. Often, it seems the idealistically motivated majority are hamstrung by the same institutional issues.
Law enforcement is more complicated than this. The golden rule of "do unto others" breaks down when it comes to punishing bad people. My personal experiences in places with no law enforcement has convinced me that police are an essential element of any society of nontrivial size. Yet law enforcement must involve secrecy to protect privacy and the innocent. Law enforcement must deal with the fact that it is primarily concerned with the dispensation of state-sanctioned violence. Law enforcers are to perform difficult, dangerous, and undervalued labor for the common good. Designing an organization to work in this environment without succumbing to corruption or self-interest is highly nontrivial and requires participation, constant maintenance, and oversight.
Bonnie Raines, one of the FBI burglars coming to campus on Tuesday, reminds us that all organizations and agencies are composed of individual people: "The average FBI agent felt pretty good about the job he was trying to do. A large number were Irish Catholic young men. They did have idealism about trying to do good. But it can be so easily manipulated. It is so easy to put blinders on. Some of them, we think, probably weren't all that smart. They didn't anticipate a break in." They certainly didn't anticipate becoming part of a notorious organization bent on crushing the civil rights movement and opposition to the Vietnam War. It's too easy to point out, but almost no one sets out to do evil.
On Dec. 9, 2014, the Committee Study of the Central Intelligence Agency's Detention and Interrogation Program, or CIA Torture Report, was finally released. In comparison with the FBI abuses perpetrated during the 1960s and 1970s, it is illustrative to see how certain patterns are repeated in our own time. After Sept. 11, 2001, all 17 US intelligence agencies frantically worked to try to understand and prevent a future terrorist attack. Some of them (notably not the FBI in this instance) went so far as to employ torture, sometimes referred to euphemistically as Enhanced Interrogation Techniques, or EITs. Similarly, prisoners were referred to as detainees to avoid any confusion with the Geneva Convention. While the CIA was likely not the worst practitioner, it is the only agency thus far to be held to any kind of account and thus forms our case study.
In summary, the key findings of the report were:
The CIA's use of torture was not an effective means of acquiring intelligence or gaining cooperation from prisoners, and justifications based on effectiveness were thus void.
The interrogations and conditions of imprisonment were brutal and far worse than the CIA told policymakers responsible for oversight, and included rape and hypothermia.
The CIA repeatedly lied to and obstructed the Department of Justice, congressional committees, the White House, and their own Office of Inspector General.
The program hindered and harmed national security.
The CIA leaked selected or false information to the media in support of the program.
The CIA's management of the program was fundamentally flawed throughout its duration, including outsourcing, lacking internal accountability, improvising torture methods on the fly, and failing to keep track of who, why, and how many people were kept.
The program was unsustainable, extremely costly, and significantly damaged the world standing of the US.
The CIA marginalized and ignored numerous internal critiques, criticisms, and objections concerning the operation and management of the CIA's Detention and Interrogation Program.
None of these findings are particularly surprising. The inefficacy of torture at eliciting reliable information has been recognized since the beginning of history. The science of effectively and ethically interrogating prisoners has been well understood in the West since at least the second world war. Even if torture worked it would still be unethical. There are plenty of examples of effective security techniques, such as universal pre-emptive imprisonment, that are nonetheless unethical and should be avoided, especially if you're fighting a war on moral or ethical grounds. Torture is illegal, and prosecution for it is mandatory under international law. Finally, torture and other acts that degrade the law of war endanger the US's own troops. As the most powerful country on Earth economically and militarily, the US has the most to lose if it seeks to normalize fighting dirty. This logic applies domestically as well, where the public's faith in government and law enforcement is the most powerful tool for law and order, a faith compromised by pointless and relatively rare actions by a handful of malefactors enabled by institutional incompetence.
Perhaps the most interesting conclusion of the torture report is the last one. The CIA, like any other agency, is staffed primarily by motivated experts who have a solid moral compass and the interests of the country at heart. Yet despite that, none of these majority stakeholders within the CIA were able to prevent or confront the torture program. In a system where incompetence can be hidden through secrecy and classification, it becomes even harder to ensure just action.
Between Hoover's FBI, the CIA's catastrophic lapse, and the NSA's mass surveillance and deception of Congress, it is now clear that corruption stemming from secrecy and unaccountability is the norm, not the exception. This danger was recognized by the Founding Fathers when Jefferson wrote, "In questions of power, let no more be heard of confidence in man, but bind him down from mischief by the chains of the Constitution."
What can we learn from these examples? Organizations exist to tackle big, complex problems that can be solved no other way. They perform vital functions and must deal with a constantly changing environment in a tightly coupled system. The most important tool an organization possesses in dealing effectively with pre-existing and emergent challenges is internal communication.
Effective internal communication is the antidote to chaos at every level, from family dynamics to Caltech student leadership to ensuring ethical action within secretive organizations to state, national, and international law and politics. Everyone on Earth participates in numerous organizations, and the actions of those organizations, mediated by communication, are the responsibility of their respective participants, whether that be speaking, listening, or relaying information.
Today, it seems unlikely that legislation will ever catch up with technological innovation. The political climate of the future is being determined in the present by the ideology of technology companies that connect people, conduct business, and solve particular problems. The prime example is the ongoing transition of political power from government to the big technology companies, for whom expertise in signals intelligence (sometimes known as targeted advertising) is core business. Whereas domestic and foreign intelligence actions have been historically separated to safeguard freedom, today both effectively sit in the hands of private, non-democratically regulated multinational corporations. To whom, if not their own engineers, will these entities be accountable when their power is absolute?