The Scientific Method in Investing

Would you rather have the fifty most important pieces of information when making an investment decision, or only the five most important pieces of information?

Most people would say fifty – more is better than less, right?

Well, not necessarily, as we can see from a study on horse betting:

How could this possibly be? Does it apply to investing as well as horse betting? Surely knowing more about a company one is considering investing in is critical in coming to the right decision (i.e. buy/hold/sell)? How could more investment research lead to worse investment performance (due to no improvement in accuracy combined with an unwarranted increase in confidence)?

We’ll explore this paradox and see what role the Scientific Method can play in making better decisions.

Outline for this post:

1. Back-to-Basics – What is the Mosaic Theory?
2. What are the problems with Mosaic Theory?
3. Back-to-Basics – What is the Scientific Method?
4. How can the Scientific Method be applied to investing?
5. Why isn’t the Scientific Method used more frequently at investment firms?
6. Summary
7. Areas for further thought & research

1: Back-to-Basics – What is the Mosaic Theory?

The mosaic theory:

“involves collecting information from different sources, public and private, to calculate the value of [a] security… An analyst gleans as many pieces of information as possible, determines if they tell a story that makes sense, and decides whether to recommend a trade.”

The idea is that if you gather all the information available and put the pieces together like a mosaic or jigsaw puzzle, then a clear picture of reality will emerge. This is how Sherlock Holmes does things, right?

Actually, not quite.

2: What are the problems with Mosaic Theory?

As Richard Heuer explains, there are several problems with the Mosaic Theory approach:

1. “Instead of a picture emerging from putting the pieces together, analysts typically form a picture first and then select the pieces to fit.”
2. Pieces of information often fit into multiple jigsaws. The fact that a piece fits one particular jigsaw does not disprove the validity of other jigsaws.

The biggest challenge to the Mosaic Theory is that we all have biases. Yes, even you, and certainly me. (Although maybe not Sherlock).

We seek out information that confirms our pre-existing beliefs, and down-play (or outright ignore) disconfirming evidence. We torture pieces of evidence into fitting our preferred puzzle, even when the piece may fit just as well into other (contradictory) puzzles or when the data point isn’t statistically significant.

Gathering additional information that conforms to our existing beliefs increases our confidence in those beliefs, but if the additional information doesn’t disprove the validity of other beliefs then it won’t improve the accuracy of our forecasts. This can cause us to bet more money than we ought to.

These fallacies of Mosaic Theory are at the core of why horse bettors weren’t better off with fifty pieces of information than they were with the five most important pieces of information.

3: Back-to-Basics – What is the Scientific Method?

From Wikipedia:

“The process of the scientific method involves making conjectures (hypotheses), deriving predictions from them as logical consequences… Scientists then test [these] hypotheses by conducting experiments or studies. A scientific hypothesis must be falsifiable, implying that it is possible to identify a possible outcome of an experiment or observation that conflicts with predictions deduced from the hypothesis; otherwise, the hypothesis cannot be meaningfully tested.”

This is how most medical diagnoses are done. Effective doctors:

1. Spend years training in anatomy, medicine, surgery, etc
2. Observe indicators (symptoms) in a patient of what is happening
3. Use their specialized knowledge to develop multiple hypotheses that might explain these observations
4. Conduct tests and collect additional information to evaluate the hypotheses
5. Then, and only then, make a diagnosis.

This isn’t to say that medical diagnoses are infallible to error. Things go wrong every day in the world of medicine (many of which can be overcome by the simple use of checklists).

But the fundamental approach of the Scientific Method is more robust than the Mosaic Theory approach.

As Heuer explains:

“This medical analogy focuses attention on the ability to identify and evaluate all plausible hypotheses. Collection is focused narrowly on information that will help to discriminate the relative probability of alternate hypothesis… the medical analogy attributes more value to analysis and less to collection than the mosaic metaphor.” [emphasis my own]

This goes a long way to explain why more information isn’t necessarily better. One ought to concentrate on collecting information that disproves a hypothesis, or that determines the relative probability of alternative hypotheses. That is the best return on time.

4: How can the Scientific Method be applied to investing?

Instead of painting a picture of a company and then setting out to find the supporting pieces, analysts ought to:

1. Spend years learning about business models & competitive advantage, reading corporate filings, practicing valuation methodologies, learning about specific businesses and industries, speaking to industry veterans, etc
2. Observe business and industry data points, and concurrently observe price-implied-expectations
3. Use their specialized knowledge to develop multiple hypotheses that might explain these observations, including what one would have to believe to be a bull, a bear, or neutral on a stock/industry
4. Conduct tests (such as speaking to industry insiders/surveying customers, analyzing the financials, studying base rates, etc) and collect additional information to evaluate each hypothesis
5. Then, and only then, determine if a stock is a buy, a hold, or a sell.

Step 1 is a substantial portion of an analysts' training. Each part takes years and skills must be refined & improved over time (there is no such thing as perfecting this step, or any step for that matter). The learning process is iterative, arduous, and humbling.

The key in step 2 is to focus in on the key drivers of the business. The key drivers include the factors that go into revenues (volume & pricing), income statement costs (such as COGS and SG&A), balance sheet costs (such as capex and working capital investment), and the number of shares outstanding. These in turn can be broken down into smaller parts, and are driven by fundamental factors such as secular trends, competitive advantages, management acumen and the customer value proposition.

For step 3, it's critical to develop multiple hypotheses. This can be interpreted in the investment world as coming up with true bear, base and bull case hypotheses. It's usually easy to find the bull case on a stock (management & the sell-side typically provide this willingly!) The bear case can be harder to find. It's possible to "invent" the bear case: e.g., what would one have to believe for the stock to go nowhere (or decline) over the next five years? Alternatively, just ask people who know the stock but don't own it what has prevented them from owning it. Center oneself around the bull/bear arguments surrounding the key drivers of the business. Heuer:

Analysis starts with a full set of alternative possibilities, rather than with a most likely alternative for which the analyst seeks confirmation. This ensures that alternative hypotheses receive equal treatment and a fair shake.

For step 4, focus on analyses that can disprove each hypothesis. Do not ignore any hypothesis. Additionally, focus on the 'diagnosticity' of the evidence. Chapter 8 of Heuer's book gives a demonstration of this stage ("Analysis of Competing Hypotheses").

Analysis identifies and emphasizes the few items of evidence or assumptions that have the greatest diagnostic value in judging the relative likelihood of the alternative hypotheses.

Step 5 sounds simple enough given the above steps. However, there are two main misperceptions around this stage:

The most probable hypothesis is usually the one with the least evidence against it, not the one with the most evidence for it.

Distinguish an unproven hypothesis from a disproved hypothesis. An absence of evidence does not necessarily disprove the hypothesis... A principal lesson is this: Whenever an analyst is tempted to write the phrase: "there is no evidence that...", the analyst should ask this question: If this hypothesis is true, can I realistically expect to see evidence of it?

There will always be nuances around what your own style of investing is/how you think about sizing/how much exposure you already have to a sector/geography etc. These will impact your decisions for step 5.

I’ll walk through an example of this in a future post.

5: Why isn’t the Scientific Method used more frequently at investment firms?

There are several impediments to the usage of the Scientific Method at most investment firms.

Firstly, analysts are under pressure to produce actionable ideas. Taking the time to learn about a bunch of businesses, develop hypotheses about them and then test these hypotheses can be at odds with the pressure to produce actionable ideas now.

Secondly, actively seeking disconfirming evidence goes against human nature. We naturally take shortcuts, and actively seeking to disprove our beliefs is very unnatural for us all.

Thirdly, in economics (and in investing in particular), there are few sure things. Many hypotheses are not falsifiable, and can only be known with the passage of time. The Scientific Method can still be applied as a way of thinking, however. And many things that do not seem to be falsifiable at first blush are in fact so if you take the time to structure the question correctly.

For example, asking “Will this company win the market for X?” will only be known with the passage of time. However, “Are product reviews on Amazon improving over time?” and “Is the ROI more attractive for customers than other solutions?” are more falsifiable. Analysts should cluster hypotheses such as these.

6: Summary

"Declarations of high confidence mainly tell you that an individual has constructed a coherent story in his/her mind, not necessarily that the story is true." - Daniel Kahneman.

Our brains crave order, but the world is messy & often inconsistent. Uncertainty should be our default state, and multiple perspectives should be contemplated. Doubt should be celebrated!

The Scientific Method involves making explicit multiple hypotheses which are falsifiable, and then actively seeking to disprove these hypotheses. This is not a natural process for most of us, and the process faces institutional barriers to adoption at investment firms of all types.

Having biases is unavoidable, but it doesn’t necessarily have to be a bad thing as long as you’re aware of your own biases i.e. the lens through which you view the world. Analysts ought to make explicit every assumption they have, and they ought to phrase each of these assumptions as a hypothesis to be tested, not a treasure to be held.

My key recommendation is to learn about a bunch of businesses really well that are in your circle of competence, and to put them into a “Wish List”. Observe their financials & customer feedback etc over time, develop multiple hypotheses to explain any interesting observations and to demonstrate what it takes to be a bear/bull/neutral, and actively seek to disprove each hypothesis you come up with/that the market throws at you.

Creativity is required in coming up with multiple hypotheses – not just the one or two that sound best to you. This is easily the hardest, and most overlooked, part of this process. See Range.

Discipline is required to actively seek disconfirming evidence for each hypothesis.

Skill is required in performing the correct analytics.

Temperament and humility are required in updating one’s views like a Bayesian, especially when it makes your past beliefs/forecasts look foolish (as it does for all of us from time to time!).

Therein lies the challenge and the opportunity.

Doing the right work > doing the work right. And picking the right games > improving at the game itself.

7: Areas for further thought & research

Note that Slovic’s research was only on eight horse bettors – which is not a robust sample size. But this effect has been proven in other domains with the same outcomes. I just picked the horse-betting example because I like the graph!

Importantly, these eight bettors were experts in their domain. One of the reasons the graph looks so flat is because the bettors were pretty good at identifying the most important five traits they’d want to know to predict the outcome of a race. They satisfied part 1 of the S.M. process that I outlined above, which is critical to knowing what to observe in step 2, and to identifying the right hypotheses to prove/disprove in step 3.

Questions I'm contemplating:

• How does one come up with the right hypotheses? Should one use price implied expectations as the assumed base case hypothesis to be tested?
• Should one prioritize the hypotheses surrounding the most important one/two variables behind a company or try to cover more ground on the first swoop? For example, should one just focus on the hypotheses around volumes & pricing for growthy businesses for the first cut at steps 3-4?
• Should one limit the number of hypotheses to as few as possible – say, three? Or should one come up with as many hypotheses as possible? Should one focus on ‘clustering’ of hypotheses & observations or be more focused?
• How does one know when one knows an industry/company well enough to start on step 2?
• What’s the best way to test this on real manager returns data?
• Are “Red-Teams” and “Pre-Mortems” a natural extension of this process or do they need to be practiced separately by someone else?
• What is the best way to implement the "Analysis of Competing Hypotheses" (chapter 8 of Heuer's book) within an investment memo?