Structured Analytic Techniques (SATs) are collaborative methods designed to instil more rigour in your thinking. They’re the slow-paced version of the five habits of the master thinker. For the sake of showcasing five such techniques, we’re going to apply them to scenarios from modern classic movies. We’ll explore the Key Assumptions Check (The Village), Structured Self-Critique (No Country for Old Men), Devil’s Advocacy (12 Angry Men), Deception Detection (The Usual Suspects) and Alternative Future Analysis (Back to the Future). Here’s how Structured Analytic Techniques can be used to break down narratives.
Table of Contents
- What Are Structured Analytic Techniques?
- 5 Structured Analytic Techniques
- 2. Structured Self-Critique
- 3. Devil’s Advocacy
- 4. Deception Detection
- 5. Alternative Futures Analysis
What Are Structured Analytic Techniques?
Structured Analytic Techniques are systematic approaches to improving the quality of your thinking when wrestling with painful problems or grappling with tough decisions. They’re typically used in intelligence analysis, business and academia. But with a bit of adaptation, they can help explore pretty much any issue, diagnose or reframe any problem, support decision-making or anticipate future events.
The qualitative methods improve analytic rigour by organising information, uncovering uncertainties and reducing the chance of making spectacular mistakes. Cognitive biases distort our perception and thinking, impairing rational analysis and judgment. Engrained mindsets make us perceive what we expect to perceive, limiting our ability to think critically. SATs improve our ability to make judgments by helping to identify and challenge them.
While they can be used by a lone analyst, most methods are designed with cooperation in mind. Structured Analytic Techniques are most effective when you can draw on the expertise of specialists and “outsiders” without special subject knowledge alike. When people with different perspectives come together to complement each other. Collaborative application of these techniques ensures a more comprehensive analysis and more reliable conclusions.
My main sources for these methods are the Handbook of Analytic Tools and Techniques by Randy Pherson and the CIA’s Tradecraft Primer. You’ll find both on the reading list you get when signing up for my free 3 Ideas in 2 Minutes newsletter. But you don’t need to be an intel analyst to benefit from Structured Analytic Techniques. To demonstrate their suitability for daily use, we’ll be applying each method to a scenario from a modern cinematic classic.
5 Structured Analytic Techniques
What Structured Analytic Technique you should use depends on the kind of problem you’re facing. Here are five methods that can be employed individually or in combination with each other. [Spoilers Ahead!]
1. Key Assumptions Check
M. Night Shyamalan’s period thriller The Village is set in an isolated 19th-century village in Pennsylvania. Maybe. Throughout much of the film, both the audience and most of the characters assume the villagers’ lives take place in a secluded settlement surrounded by woods. Conveniently, the residents cannot explore or leave their little world since the woods are inhabited by mysterious and dangerous creatures. However, the plot twist reveals the audience’s key assumptions to be incorrect.
The Village is not set in the 1800s but in the present day. The isolated community turns out to be carefully planned and constructed by a former history professor. In an attempt to escape the violence and pain of contemporary society, he recruited like-minded people to found the village. Needless to say, the mysterious creatures are fake, too; a myth created to keep the new generation from exploring the outside world. Essentially, the thriller’s thrill rides on the audience to buy into the key assumptions of the setting.
A Key Assumptions Check is a diagnostic technique designed to prevent such misinterpretations in real life. Assumptions are dangerous knowledge; ideas we consider to be true, consciously or unconsciously and without evidence. They can act as gap fillers for the unknown, too. Since we’ve accepted them as genuine, we don’t challenge them. A key assumption, such as the one about the movie’s setting, forms the basis of a judgment. If the assumption collapses, the entire assessment collapses. So it seems like a good idea to identify and question them.
Step 1: Identify Your Underlying Assumptions
We begin by putting ourselves in the shoes of overly analytic film enthusiasts who can’t just sit back and enjoy a film. Our starting point for the Key Assumption Check is to verbalise the working conclusion and break it down into its underlying assumptions. We ask W-Questions (who, what, when, how etc.) and listen for phrases that indicate undue certainty (it must be, will always etc.). We make those assumptions explicit by writing them down.
Everything about The Village makes us think we’re watching a historical fantasy thriller. This spawns several key assumptions about the storyline: The story is set in a fictional version of the 1800s. In this isolated world, paranormal beings exist. Shyamalan’s plot twist will occur within this framing. Who are those creatures? What motivates them to terrorise the village? Chances are, the average moviegoer accepts similar assumptions as true and suspends disbelief.
Step 2: Challenge Your Assumptions
Once the key assumptions are identified, we assess them for their validity. Remember, for our conclusion to be true, our assumptions have to be correct. So it’s necessary to challenge them by asking good questions: What makes you think they’re true? Under what conditions would they fall apart? Can their validity change over time? If they’re wrong, how would this impact our conclusion? Using these guiding questions, we evaluate each assumption as being supported (S), correct with caveats (C) or unsupported (U).
For The Village to be a historical fantasy thriller all of our assumptions have to be mostly true. But there’s reason to doubt the story’s setting in the 1800s. Anachronisms in language, behaviour and clothing put caveats on this claim. The same goes for the paranormal beings. Nobody ever gets attacked by the slow-moving creatures. The elders’ conversations hint at them being aware of some kind of plot to keep the settlement isolated. Then there’s our assumption about the nature of the expected plot twist. If we’re being honest, it’s unsupported by any evidence.
Step 3: Refine Your List
In the third step of our Key Assumptions Check we’re reworking our list. We create three bins with supported, caveated and unsupported assumptions – and delete the latter ones. It’s perfectly possible that new assumptions emerge from this exercise. So we add and evaluate those. We continue to refine our list until we’re only left with assumptions likely to be true. This way, we’ve successfully avoided intuitive traps such as an overreliance on first impressions or the all-too-common confirmation bias.
In our example, we’re throwing out our assumption about the plot twist. We’re left with the anachronisms, which could be a result of the director’s sloppiness. And the creatures, which could be either symbolic or literal. Throughout the story, new evidence emerges and our focus shifts to a new assumption about the community leaders: They’ve created some kind of a fake world.
Sure, running an Structured Analytic Techniques at the movies might ruin the film. But it demonstrates how we can easily elevate our thinking from intuitive guesswork to systematic analysis with a bit of methodical questioning.
2. Structured Self-Critique
There are no do-overs in life. Unless you’re Phil Connors (Bill Murray) in the classic comedy Groundhog Day. Unfortunately for Llewelyn Moss (Josh Brolin) he’s not Phil but a character in the crime thriller No Country for Old Man. The hunter stumbles upon the aftermath of a drug deal gone violently wrong. From there, the curious, guarded and resourceful war veteran follows a track that leads him to yet another dead body. And a suitcase full of money. Should he take that case?
Whenever we’re about to make a high-stakes decision, a Structured Self-Critique can be used to evaluate it before the law of unintended consequences strikes. It can help us anticipate weaknesses in our analysis before decisions are actioned. The method is an evolution of Premortem Analysis, which was originally conceived by cognitive psychologist Gary Klein. Developed by Randy Pherson, Structured Self-Critique legitimises dissent and turns analysts into their staunchest critics.
The classic Premortem asks the group to imagine spectacular failure in the future. Their impending decision turned out wrong. What happened? Structured Self-Critique changes the incentive in a similar way. Decision-makers are now “judged by their ability to find weaknesses” in their thinking. I’ve condensed Pherson’s approach into four areas for questioning and review. Let’s apply it to Llewelyn’s conundrum. He’s about to take that suitcase. How could that turn out to be a poor choice?
Step 1: Sources of Uncertainty
Is there a single correct answer or more than one? Is the situation evolving or in flux? Have alternative hypotheses been considered including the possibility of deception?
There are plenty of sources of uncertainty around the decision to take the money. Someone will most likely be looking for it. A survivor saw Moss’ face earlier. The situation seems volatile, more criminals or law enforcement could arrive at any time. It’s unclear what powers are even involved in this. The one right thing to do is most likely to leave it and get out. But it could also be a trap and become a problem either way since Llewelyn is now a witness. The veteran seems to think he knows what he’s doing. But who says his counterparts aren’t even smarter than him?
Step 2: Critical Assumptions
Were all key assumptions identified and challenged? If it turned out to be wrong, which assumption would have the greatest impact on the outcome?
Llewelyn won’t run a comprehensive Key Assumptions Check. But he can make his most obvious assumptions more explicit: Nobody will find out I took the case. Should I have to, I can outsmart the criminals and the police. I can get away with this. His assumptions have to be mostly true for his decision to not be a life-ending one. They seem the main source of potential failure should he keep the money. But we’ve already uncovered several sources of uncertainty. Moss may overestimate himself. And we’re only halfway through our analysis.
Step 3: Evidence
What is our evidence that we made the right decision? Is there information or expertise we don’t have but would want to see to make an informed decision? What evidence would change our minds?
It appears straightforward. The skilled veteran is alone in the desert and nobody can see him take the money. But he doesn’t know who he is dealing with and what their capabilities are. If Llewelyn knew he was being tracked, he certainly would not take that case. He would need more information out there in the desert before making his decision. But No Country for Old Men is set in the 1980s. There’s no Starlink yet and he can’t google Mexican cartels. Moss has to make a call based on very limited information.
Step 4: Changes in Environment
What external factors (social, political, legal, competitive, ethical etc.) not under our control would change the viability of our decision?
There’s a lot not under Llewelyn’s control and a lot that can change. Moss’ car could break down or the weather could change trapping him in the desert. As we mentioned earlier, he’s unaware of the social and legal context of the whole situation. Depending on who owns the money, the ethics of his theft might also change.
Thinking through the four steps of the Structured Self-Critique could’ve helped our protagonist to reconsider his decision. But Llewelyn doesn’t think for long. He takes the cash. And nothing happens. Until he’s being hunted by an aging sheriff, a psychopathic hitman and the Mexican cartel; the latter of which gets to him first.
3. Devil’s Advocacy
Devil’s advocacy is central to the 1957 courtroom drama 12 Angry Men. The entire story is set in a jury room where twelve men are tasked with deliberating the guilt or innocence of a young defendant charged with murdering his father. All but one juror finds the defendant guilty. Juror 8 (Henry Fonda) is not convinced. He uses devil’s advocacy to challenge his fellow jurors’ verdict, revisit the evidence and consider reasonable doubt.
When a group faces a false consensus due to groupthink or a single, strongly held belief prevails, Devil’s Advocacy can be used to challenge them. As a contrarian Structured Analytic Technique, it’s less collaborative and more confrontational. Usually, a brave member of the team builds a strong case that contradicts the prevailing view. This is why it should be used with caution and for high-stakes decisions. That is, whenever making the wrong decision has dire consequences.
Step 1: Outline the Current Thinking
The weaknesses of a judgement tend to lie in flawed assumptions, faulty logic, questionable evidence and the lack of plausible alternative hypotheses. So our first step is to outline what those are under the dominant view. In 12 Angry Men, all but one juror agrees: The murder weapon, a knife, is rare and therefore proves the son’s involvement. An old man heard the defendant threatening his father. The timeline also points to the defendant being guilty. His alibi seems unbelievable, too, as he cannot remember much of the movie he supposedly saw during the time of the murder.
Step 2: Select and Review
It’s best to focus on a few aspects of a judgement that are most susceptible to challenge. Is the evidence valid? Is information missing? Is deception at play? In the case of our murder trial, Juror 8 questions the assumption that the knife is unique or rare. He bought one just like it in the same neighbourhood. The same goes for the witness. The old man’s hearing and eyesight are not that great anymore, which puts his testimony into question. Did he really hear the defendant say: “I’m going to kill you?” And how much do people typically remember of a movie they just saw?
Step 3: Present Alternative Hypotheses
Now it’s time to lay out and present the evidence for alternative hypotheses to the rest of the group. You make the case for why their assumptions are wrong, their evidence is of poor quality or they might have been deceived. Depending on your context, this can happen verbally or by drafting a contrarian paper. It’s worth making a clear distinction between the prevailing view and your position as a devil’s advocate. Juror 8’s little devil’s advocate spiel does not miss the mark as it establishes reasonable doubt about the alleged murderer’s guilt.
By the way, the Tenth Man Rule is a similar method worth considering. Born out of the movie World War Z, it’s an institutionalised form of devil’s advocacy. Yet another way to take this technique further is Red Team Analysis.
4. Deception Detection
Throughout the 1995 thriller The Usual Suspects, Detective Dave Kujan (Chazz Palminteri) assumes that Roger “Verbal” Kint (Kevin Spacey) is just a low-level con artist. A pawn in a larger criminal scheme. One of the “usual suspects” involved in a heist gone wrong. Kujan believes he is getting the true story out of Verbal during his interrogation. This includes valuable information about the identity of one Keyser Söze, a ruthless and mysterious figure orchestrating crimes.
It’s only after Verbal is let go that the detective has a shocking realisation: Verbal Kint is, in fact, Keyser Söze. The entire story he narrated during the interrogation was a carefully crafted fabrication to deceive the authorities. Söze successfully manipulated the perceptions of the detectives (and the audience), allowing him to escape justice. The detective’s interrogation strategy failed. Because it rested on a flawed key assumption fed to him by the perpetrator himself.
Deception Detection is a structured analytic checklist that would’ve helped Detective Kujan to uncover Verbal’s misdirection. You should use it whenever a counterpart has a history of deceptive behaviour, you have to rely on a single piece of information or the stakes are too high to get things wrong. The checklist consists of four acronyms. We’ll go through them in condensed form. But if you’re keen on more detail, check out my long-form essay Deception Detection: How to Anticipate Deceit.
Step 1: Motive, Opportunity & Means (MOM)
We begin by assessing if the potential deceiver has a motive, the opportunity and the means to engage in misdirection. This includes the communication channels they have at their disposal as well as the risks and rewards of engaging in such behaviour. It’s also worth considering if your counterpart has feedback mechanisms through which they could monitor if their deception works. If a suspect doesn’t have the MOM, that pretty much clears them of deception.
Kujan is dealing with a known con artist. Yet, it seems like the detective grossly underestimates Verbal’s motive, opportunity and especially his means to tell a lie for personal gain. The criminal has the ear of the police and can monitor in real-time how his fictional story about Keyser Söze is received. At the time, Verbal is the police’s only witness and source of information. This alone should’ve made them more suspicious of the con artist’s story.
Step 2: Past Opposition Practices (POP)
When looking at Past Opposition Practices, we take a deeper dive into our counterparts’ track records. Do they have a history of engaging in deception? Does their current behaviour fit an existing pattern? Are there any historical precedents worth reflecting on? Even if there is no history, could the circumstances have changed to turn an honest person into a swindler?
Granted, we don’t know much about Verbal’s criminal record, so he may seem harmless. But he was picked up at a crime scene that involved explosions and a murdered Hungarian crime mob. During the interview, it also becomes clear that Verbal’s role in the events is much more significant than he initially lets on. It seems careless to not even consider the deception hypothesis. After all, criminals have been kind of historically known to mislead law enforcement by playing dumb.
Step 3: Manipulability of Sources (MOSES)
In analytical terms, a source refers to any piece of information, document or witness that is used to support or provide evidence for an assessment. Manipulability of sources is concerned with people’s access to them and their susceptibility to tampering. How reliable is the source? Has it been proven correct in the past? Who has direct access to it? How vulnerable to manipulation and control is the source?
Again, it doesn’t look too good for the detective’s career. The only thing that might make Mr Kint a reliable source is his seeming harmlessness and submissive cooperativeness. However, he has no known track record of providing accurate information. His claims are not verified by a second source. He can (and absolutely does) come up with any narrative he wants as long as the police are naive enough to believe it and let him go.
Step 4: Evaluation of Evidence (EVE)
Now it’s time to evaluate the evidence. We examine the whole chain from the collection of the data or information to its analysis and evaluation to feed into an assessment. Is there any conflicting information out there? Are there other sources, both human or otherwise, that can provide corroborating evidence? Or is any information noteworthy by its absence?
At the end of the movie, Verbal simply walks out of the building. As Detective Kujan reflects on Verbal’s testimony, the movie cuts to the criminal whose disability disappears. While it dawns on the officer that he has been misled, Kint’s shy demeanour turns into smug confidence. The only thing noteworthy of its absence was the complete lack of any evaluation of the evidence. Verbal was released because the key assumptions about his intentions and identity remained unchallenged.
5. Alternative Futures Analysis
Back to the Future is a 1985 science fiction comedy about teenager Marty McFly (Michael J. Fox) whose scientist friend, the wacky Doc Brown (Christopher Lloyd), builds a time machine in a DeLorean sports car. Due to unfortunate events, Marty travels to the year 1955 where he accidentally prevents his mom from falling in love with his dad. Since his mother is now keen on her future son, Marty must help his young father to right the wrong so he will not never be born. He succeeds, which is helpful as it allows the plot of two more sequels to happen.
Throughout the trilogy, time travel doesn’t always go to plan. From the perspective of 1955, several alternate timelines and future scenarios are created; for better or for worse. In the original timeline, Marty’s parents and siblings are tragic abject failures. But after Marty’s intervention, their future turns out bright. There’s the film’s futuristic conception of life in 2015; with hovering skateboards and flying cars. But in another timeline, the movie’s villain Biff Tannen (Thomas F. Wilson) has turned their hometown into a hellhole. Who could’ve predicted that?
Alternative Futures Analysis is an approach to anticipate how situations with heightened complexity and uncertainty might develop. It’s ideal to uncover unknown unknowns. The thinking technique allows you to generate multiple future scenarios; all vastly different. It works best when many factors and actors at play need to be distilled down into workable scenarios. Without the need for plutonium-powered time-travelling sports cars. Here’s how to develop some alternative future scenarios for time travel.
Step 1: Develop the Focal Issue
The first step in generating alternative future scenarios is finding our focal issue. This is usually done by talking to those most knowledgeable about a subject. SATs can take time, effort and commitment of hours or days. So resources should be focused on the most relevant issue or part of it.
The focal issue of our choice today is dictated by Doc and Marty’s mishaps. They weren’t prepared for what they unleashed. So we’ll do what they failed to do. We’ll attempt to anticipate the potential consequences of developing and using time travel technology.
Step 2: Brainstorm Factors
Our next step is to brainstorm the drivers of our focal issue, that is factors, forces, actors and events that will likely shape it and its outcomes. We then select the two, four, six, eight etc. most critical or uncertain ones. It’s important to select those based on clear criteria, which will depend on the context of our focal issue.
There are many factors at play when it comes to time travel. Advancements in physics are a given. The development of relevant technology is another big driver. Then there are the ethics of going back to the past, potentially altering history. What also factors in is the impact the whole time travel enterprise will have. For our example, we pick the frequency of using time travel technology and its impact as two key drivers.
Step 3: Create a Spectrum
Now that we have our key drivers, we conceptualise them as a spectrum and define their two endpoints. For example, if a key factor is the advancements in physics, the two endpoints could be rapid and stagnating. We do this for all the key factors we’ve identified.
For the sake of our case study, we limit our key drivers to two. When it comes to the frequency of use of technology, we define occasional and frequent as our two endpoints. The two ends of the spectrum for the impact of time travel technology are limited and severe.
Step 4: Create a Future Matrix
Now we pair our key drivers and turn them into matrices. Each spectrum is either put on the x-axis or the y-axis. This creates four quadrants that will be filled with four alternative future scenarios.
In our example, we’re crossing the frequent to occasional use of time travel technology with the severe to limited impact of time travel. But this is just an intermediate step leading to the fun part of storytelling.
Step 5: Tell Colourful Stories
Each quadrant represents one possible future scenario. It’s now time to fill those scenarios with life. This is best done by telling stories of how they might turn out. These stories should be thought-provoking but plausible.
In our first scenario, time travel is only used under exceptionally tragic circumstances but with severe impact. A purple alien, who sounds suspiciously like Llewelyn Moss, from No Country for Old Men, must be stopped from wiping out half of life in the universe. The second scenario pictures a world in which time travel is so ubiquitous that we need a government agency to regulate it. The impact is so severe that constant damage control is necessary. We could even call those agents time cops and make a mediocre movie about them starring Jean-Claude Van Damme.
Scenario number three explores the consequences of an occasional low-impact application of time travel technology. It could see usage for personal gains by teenagers and home scientists such as Marty and Doc Brown. The last alternative future should also sound familiar. In a world where time travel is frequent but the impact is limited, we’re potentially looking at time travel as a personal development tool. Just like Phil Connors in Groundhog Day, who experiences the same day over and over to learn how to be a better person.
Since I said the scenarios should be “thought-provoking but plausible” you may want to apply this SAT to more serious issues. Then you should also know that the method is part of the much more comprehensive Foresight Structured Analytic Technique. It combines several SATs and lets you generate scenarios from which you can then choose representative ones. The development of indicators rounds it all off as they can be used to monitor which of your scenarios is unfolding.
Closing Thoughts
While this article can only serve as a (tongue-in-cheek) primer, Structured Analytic Techniques are an excellent way to help us overcome our cognitive limitations and improve the quality of our thinking. Even in mundane situations such as ruining movie night with unreasonable movie critique. They may seem overly intellectual and onerous at first. But they’re surprisingly easy to use and practical. And the more we use them, the more we develop the daily habits of a master thinker.
