Sloppy Thinking Is Always Wrong

Not too long ago I was at dinner with a forensic dentist, a forensic mathematician, and a forensic biologist. One member of the group mentioned that his wife was terrified to go camping because she feared abduction by UFOs. Being a pack of know-it-alls, we all had a good snicker at her absent expense.

The biologist piped up at this point:

While her fear sounds silly, it's very likely that there's life on other planets, and somewhat likely that there's intelligent life. So, therefore-

The forensic mathematician put his head in his hands and interrupted with a groan.

You have no idea how offensive that statement is to me. How can you say you know anything about the probability of finding life on other planets without knowing something about the frequency of life on other planets? And how can you determine this frequency of life on other planets without actually visiting them or somehow performing an experiment to test these planets for life? Unless, of course, you are trying to tell me that you're from another planet?

The mathematician's rebuke is harsh but fair. People carelessly assign probabilistic statements to events without any information other than their own wishes that something be true.

When we say something is "likely" we are in fact making a numerical judgment that the chances are greater than 50%. When we say that something is very likely, we make the statement that the chances are even greater than our estimate of "likely".

The fact that we refuse to be precise in our estimate does not excuse us from reaching a conclusion without any supporting information.

Someone might rejoinder that there's a chance our fuzzy-headed biologist might just be correct in his statement. Unfortunately, he's no more correct than the stopped clock that's right twice a day. "Correct" in this case is based on luck instead of knowledge.

A More Complex Example

At first blush, forensic thinking may appear to strip us of all investigative power and abandon us in murky waters where every conclusion is predicated on weasel-worded assumptions. This is absolutely not true. To illustrate, let's return to our dice example again. Let's assume we roll six sixes in a row on what appears to be a normal six-sided die. This makes us suspect a trick die that will only roll a six. The chance of rolling six sixes on a trick die that always lands on six is 1, because that is the only result we'll get. The chance of rolling a six six times on normal die is:

(1/6)x(1/6)x(1/6)x(1/6)x(1/6)x(1/6), or approximately 0.0000214.

Therefore, our odds of rolling six sixes when we compare the assumption of a trick die versus a normal die becomes:

1 divided by 0.0000214 or approximately 46,700 to one.

Which is a very strong argument in favor of a trick die, as the six sixes in a row are 46,700 times more likely with that die.

The advantage to this type of thinking is that it separates our assumptions from our analysis and draws them out into the open. Even if your assumptions are wrong, you can still remain confident in your analysis. However, a change in your assumptions will often require a different conclusion.

New to this blog?

I'd first read this posting:

What is Forensic Thinking?

Followed by

A Simple Example

Enjoy.

What is Forensic Thinking?

Put plainly, forensic thinking is a way of considering complex problems in an honest and forthright fashion. Forensic thinking requires us to be explicit in our assumptions and only make definitive statements about facts that we can verify. Ideally, any good forensic scientist approaches their work in this manner.

When thinking about forensic analysis, the average person might understand the thought process to go something like this:

I extracted DNA from semen found inside a rape victim. The DNA profile from that semen matches a suspect. This DNA profile is so rare it is found in only one out of ten billion people, and therefore unlikely to be found in anyone else. Therefore, the chances against the suspect being innocent are approximately ten billion to one.

This sounds plausible, but is incorrect and not even remotely close to forensic thinking. A forensic scientist would instead present the question this way:

I extracted DNA from semen found inside a rape victim. The DNA profile from that semen matches a suspect. This DNA profile is so rare it is found in only one out of ten billion people, and therefore unlikely to be from anyone else. How relatively likely is it that I'd find this DNA profile when I compare two possibilities: The suspect contributed the semen found inside the rape victim versus some random, unrelated person contributed the semen?

The correct answer is approximately ten billion to one.

These two ways of thinking sound similar, but they are worlds apart. The first example mixes the analysis with the assumptions by leaping to the conclusion that the likely donor of the semen is the likely rapist. The suspect matching the DNA profile may be the victim's consensual partner and not a real assailant. At the same time the suspect could be both a frequent consensual partner of the victim AND the perpetrator of the rape. The DNA profile alone cannot distinguish between these two possibilities.

Making a definitive statement about events we were not around to observe is extremely perilous. The forensic scientist can only be certain of the evidence and analysis that he or she performed or examined directly. In this case, that would be the DNA profile resulting from the scientist's careful analysis.

One can only make informed statements about the probability of an event that has been observed or measured objectively. To do otherwise would be like trying to guess how likely a die roll will end up "6" without knowing how many sides the die has.

Forensic Thinking: A Simple Example

One can only make informed statements about the probability of an event that has been observed or measured objectively. To do otherwise is like trying to guess how likely a die roll will end up "6" without knowing how many sides the die has.

The uncertainty behind such guesswork is obvious. The probability of rolling "6" is 1/6 for a six sided die, 1/20 for a 20 sided die, and zero for a four-sided die. Not knowing anything about the die renders it quite impossible to determine how likely a result would be, the probability would range anywhere from zero to 1/6. The die might even be a trick piece designed to roll "6" more often than any other number, which means the probability is even higher than 1/6.

Now imagine how far off you'd be with more complex problems, like how likely a crime scene DNA profile matches the "real killer" when you weren't present for any of the events that deposited the DNA in the first place.

However all hope is not lost. Let's return to the die example. We need to determine how likely a "6" would be when we roll this unseen die. Let's say that we know that the die in question either has six sides or twenty sides. We can compare the probabilities of rolling a "6" on a six versus a twenty sided die by thinking about the problem like this:

How likely am I to roll a six assuming a six-sided die versus rolling a six assuming a twenty-sided die?

Which would be:

1/6 divided by 1/20 = 20/6 or approximately 3.333

Assuming we have no other evidence regarding six or twenty sided dice being chosen for our experiment, the odds are approximately 3.333 to one in favor of rolling a "6" with a six sided die. Certain results are even more informative than the "6". For example, had we rolled a "7", the probability of getting that result assuming a six-sided die are zero, which would exclude the six sided die from our consideration.

We're not home yet, however. Odds of 3.333 to 1 mean you'll expect to be wrong approximately 23% of the time. What if we rolled the die three times and always came up with numbers of six or less? We'd certainly begin to suspect a six-sided versus a twenty-sided die, But how likely is it?

The probability of rolling any number on a six-sided die is 1/6. However, in this case we're asking the probability of rolling numbers of six or less three times in a row. Since the die can only produce numbers of six or less, the probability for each roll is 1.

1 x 1 x 1

The probability of rolling any number on a twenty-sided die is 1/20. However in this case, we're asking what the probability of rolling any number of six or less on a twenty sided die three times in a row. That becomes

(1/20) + (1/20) + (1/20) + (1/20) + (1/20) + (1/20) = 6/20

(6/20) x (6/20) x (6/20) = 0.027

If we divide the probability of getting numbers of six or less after three rolls on a six-sided die by the probability of the same result on a twenty sided die, we get

1/0.027 = 37.037

Or odds of about 37 to 1. Approximately 97% of the time you'd get this result with a six-sided die versus a twenty-sided die. After these results, we'd be reasonably confident of guessing that the die has six sides and the probability of rolling a "6" is 1/6.

Forensic Scientists on TV do not utilize forensic thinking.

On television shows, forensic scientists generally fall into two types. Let's call the first Sherlock. Relying more on instinct and experience than any scientific skill, Sherlock only uses science to confirm what he or she already knows. It's almost as if Sherlock solved the crime at first glance, and the science is merely an amusing afterthought. Denzel Washington's character in "The Bone Collector" is a Sherlock. He sees things that no one else sees, and is always one step ahead of the science. He picks as his successor a beat cop with zero forensic experience who just happens to be born with the same instincts as Sherlock Washington.

The second type we can call the Deep Space Probe. The Deep Space Probe works in the lab and through miraculous techniques can determine the exact substance that was on the suspect's shoe and link it to the exact place of the crime. Results are produced immediately and eliminate all other possible explanations leaving only one possible culprit. CSI is a big fan of the Deep Space Probe forensic scientist, often paired up with a Sherlock working in the field. Sherlock's initial hunches are only disproved when the Deep Space Probe extracts miraculous (and impossible) amounts of information from the evidence.

Neither the Sherlock nor the Deep Space Probe utilize forensic thinking.

Confirmation Bias

Sherlock suffers from what is commonly known as confirmation bias.

Imagine a crime where thieves break into a donut shop and steal money from the register. Examination of the scene reveals fingerprints not belonging to the owner or current employees. Some prints at the scene match a man on parole for a felony burglary. Should we focus only our examination on this one man, or is it a mere coincidence?

Confirmation bias is a pernicious hound, stalking the intellect and striking when we least anticipate falling into its clutches. In its most basic form, confirmation bias is a form of bigotry. A very clear sign of confirmation bias are cases where a theory is proven regardless of the results of our investigation. If fingerprints matching our suspect are found, that means the suspect committed the crime. If fingerprints were not found, that means that only an experienced burglar such as our suspect committed the crime.

If you've seen the film In the Heat of the Night one of the opening scenes has a classic rendition of confirmation bias. A man is found murdered. A short distance a way, a black man is found trying to leave town on the next train with a large amount of cash in his wallet. The police are certain that they have their man because he fits their expectations and experience. They're horribly wrong, yet it takes great effort to convince them otherwise.

Politics are rife with confirmation bias. When the public votes for our favorite candidate, that means that the candidate is the right person for the job. When the public votes against our favorite candidate, the candidate is still the right person for the job, but the public is too stupid to recognize this.

Overzealous Interpretation of Evidence

The Deep Space Probe suffers from overzealous interpretation of evidence. A single hair or mote of dust is expanded into the entire fabric of the crime. In reality, minute traces of evidence found by the Deep Space Probe can rarely point to only one source. For example, a brown hair from the head of a Caucasian will not be easily confused with a hair from someone of African descent. However, one cannot look at a hair under a microscope and uniquely determine which person the hair came from.

DNA analysis can in many cases determine which person left a hair, bloodstain, or other biological evidence at a crime, but that does not mean the person committed the crime. Forensic DNA analysis is an excellent way to determine identity, but we cannot determine when the evidence was left at the scene. If a person sneezes on a tissue and leaves in a trashcan where someone is assaulted later on, overzealous interpretation of the DNA profile obtained from the tissue may unfairly implicate the sneezer in the crime.