Focus on the facts, data, and probability. Always backtest everything


Too many investors and traders are “guessing”. Their “analysis” is no better than random guessing based on what they deem “logical”. As we demonstrate on Bull Markets, conventional ways of trading the stock market are often no better than tossing a coin.
Whenever you think about ANYTHING in the markets, whether it be a belief, a strategy, or a market outlook – you always want to backtest it.

What is “backtesting”?

At any given point in time, there will be a certain set of market conditions + your market outlook
For example, the S&P’s 14 day RSI is at 20 (set of market conditions), hence I think this is short term bullish for the stock market (your market outlook).
Backtesting is when you do historical analysis to see if your market outlook stands up to the light of evidence.
E.g. “The S&P’s 14 day RSI is at 20, hence I think this is short term bullish for the stock market.” Historically, is this true? When the S&P’s 14 day RSI was at 20 in the past, did the stock market actually go up? Or did the stock market keep going down?
In other words, when a similar set of conditions appeared in the past, what did the stock market do next in the short term, medium term, and long term?
In order to properly do backtesting, you need to use data instead of “eyeballing” a chart. Data is objective. Your eyes are subjective, which means that you tend to see what you want to see.

Why shoud we backtest everything?

Because we cannot trust our eyes, “intuition”, or “gut feeling”.
When you do qualitative market analysis and use DISCRETIONARY (non-quantitative) trading strategies such as staring at a stock chart with support/resistance and indicators, you tend to see what you want to see. As a result, discretionary trading strategies work very well with 20/20 hindsight. But in real-time, discretionary trading strategies are often no better than random guessing. Worse, it’s impossible to know how well a discretionary strategy worked in the past because it’s impossible to backtest it.
When most people start trading/investing, they do so with discretionary strategies because it’s much easier to create a discretionary strategy than it is to build a quantitative strategy.
A lot of people will use qualitative strategies once or twice. And when it works, they’ll think “HOLY SHIT THIS IS AWESOME! THIS STRATEGY IS THE SECRET TO INVESTING AND TRADING!” (That’s pretty much the standard reaction among people who are first introduced to technical analysis, until they experience a few failures). And then they wonder why it fails the next time.
The failures really weren’t a mystery.
These investors and traders simply didn’t backtest their strategies long enough to see EXACTLY how well these strategies works over a long period of time, during bull markets and bear markets.
In other words, many investors and traders will stumble across a strategy, blindly believe that it works, and then proceed to use it.
Here’s the problem.
Even bad strategies work well 50% of the time. Hence, a lot of traders will stumble across a bad strategy when it’s working well, and think “wow, this is terrific!” And then they use the strategy for a long period of time but find out that it doesn’t work. Then they wondered “what happened? why isn’t this working?” Well the other 50% of the time happened – the 50% of the time when the bad strategy doesn’t work.

Don’t be an eyeball trader

I call investors and traders who don’t backtest “eyeball traders”, because all they do is “eyeball” a chart, stroke their chin, and then invent some rationale for their investment decision or market outlook. Sometimes they’ll use cliché phrases to justify what they’re doing, such as “I’m short because the 200 day moving average is like a magnet for the S&P”. (For the record, there is no such thing as a “magnet” in the stock market).
Here’s a very simple example. Looking at chart patterns is a common discretionary strategy. Chart patterns work terrifically in 20/20 hindsight, but in real time aren’t often better than a coin toss. Chart patterns have so many false signals that when you add the successes + the failures, they are often not much more useful than a coin toss.
Here’s another kind of discretionary analysis that has no predictive value. Eyeball traders who look at support/resistances will often say “if the stock breaks below $100, it’ll go to $95”.
That’s like saying “if the temperature goes above 75 degrees Farenheit, it’ll probably go to 80 degrees Farenheit, give or take a few degrees.”
Thanks Sherlock. That’s how numbers work. The big question is IF.

Here’s a simple example. Many eyeball traders use “head and shoulders” patterns for identifying tops. Head and shoulders work great with 20/20 hindsight.
But in real time? These patterns have far too many false signals.


The common answer to dealing with false breakouts and breakdowns is “you need to wait for confirmation”.
But by the time you get confirmation, half of the market’s movement is already over! And even after “confirmation”, there are still a lot of false signals.
But what if you really want to trade using chart patterns?
That’s ok! Learn computer programming, learn AI, and then OBJECTIVELY QUANTIFY each chart pattern so that your computer can objectively recognize it (e.g. how to quantify “head and shoulders” instead of eyeballing it). That way you’ll know exactly how well these things work, and exactly what counts (and doesn’t count) as a “head and shoulders pattern”.

Why don’t people backtest everything?

Most investors and traders don’t backtest their analysis for 1 of 3 reasons:

  1. Most people just don’t know how to backtest.
  2. Most people don’t have time. Backtesting takes time. It’s not as easy as staring at a chart and guessing “this looks bullish to me!” or “this looks bearish to me!”
  3. Some people have the time, but they’re just too lazy (this is the case with many professional investors, traders, and fund managers). It’s much easier and more fun to fabricate market analysis out of randomly guesses.

There’s another reason (beyond plain laziness) why most professionals in this industry prefer guessing over backtesting.
It’s because of the way financial professionals are educated.
Most people in the financial industry come from a social science background. They probably studied economics, business, accounting, finance, etc. They don’t come from an engineering background. (This is actually something that my wife brought up – she comes from a civil engineering background).
People from a social science background are taught to “guess” using theory. In school, all you need to do is regurgitate a bunch of economic theory. I know, because I majored in economics.
You know the best thing about young people? They’re impressionable. Older people are harder to mold and change. But for young people, if they spend 4 years of their life learning how to guess and memorize theories based on guessing, then that’s what they’ll do in their future career and life. They’ll probably make decisions based on what “seems” logical to them (i.e. guessing) or what economic theory supports (i.e. guessing).
*This is also why employers prefer hiring engineers over social arts students straight out of school. Engineers are trained with a systematic way of thinking.
This was one of the funniest things I noticed when studying economics in university. Economics professors “prove” theory with theory! They often come up with their pet theory first and then “find evidence” to “prove” their pet theory. That’s like fitting the data at its worst.
Here’s a very simple example. Economic theory states that rising interest rates is bad for the economy and stock market. But economic students NEVER bother to actually backtest this. They just regurgitate the theory. They are never asked in college/university to see what happens next to the economy and stock market in 1 month, 1 year, 3 years etc when interest rates go up.
This stuff is simple. A lot of it is just correlation and regression analysis.
People with a social science background are not taught to QUESTION EVERYTHING. They’re taught to blindly believe. XYZ expert says ABC, so you should memorize, repeat, and regurgitate ABC. These experts never show you the EMPIRICAL PROOF for where ABC theory came from.
That’s why people with STEM backgrounds tend to be better investors and traders. People with STEM backgrounds are taught to think in terms of systems, data, and facts.
People with STEM backgrounds are trained to have the discipline to test things for themselves. Theory is great, but it needs to stand up to the light of evidence. Engineers are trained to never assume something. Can you imagine if a civil engineer builds a bridge without factually knowing all the loads and measurements? That would be disastrous.
Yet investors and traders do that all the time! They make their decisions based on a bunch of theories, yet they never bother to backtest if the theory is even true or not. In economics and finance, a lot of things are based on assumptions. “Assuming XYZ, assuming ABC…, assuming ceteris paribus”. And 50% of the time, these assumptions are wrong. In engineering and STEM fields, people don’t ASSUME things.
In school, engineering students like my wife had to do a ton of experiments, labs, and research. Do you know what “economic labs” look like? A bunch of people regurgitating theory and models that have little historical data to support them. Jim Rogers thinks that you shouldn’t study economics or finance to become a successful investor because the way modern economics is taught is terrible. Instead, you should study history and philosophy. History doesn’t repeat itself, but it does rhyme.
Humans have the tendency to let their pre-existing belief guide their thinking. Hence, they see and focus on things that support their thinking. This is called “seeing what you want to see”, which is a bias that’s impossible for humans to avoid without using quantitative data.
That’s why I use quantitative models to trade. I don’t stare at an economic data chart and say “this ‘looks like’ the economy is improving to me”. “Looks like” is an eyeball trader’s favorite phrase. Instead, I quantitatively describe the phrase “the economy is improving/deteriorating”, and then let the computer objectively tell me in which periods the economy was improving and in which periods the economy was deteriorating.
Never, ever blindly believe in an ideology, strategy, or idea. Backtest everything, especially “conventional wisdom” to see if it is true or not.

Become a systems trader

By backtesting everything, you end up creating a SYSTEM. A quantitative SYSTEM.
You need a system to be successful.
This is true in investing as well as in business. A successful business doesn’t rely on wanton discretionary decisions. Big successful businesses have systems in place so that things run smoothly. Think about a factory as an example. Can a factory run efficiently and smoothly without systems? Of course not! It would be chaos, and everyone would waste a lot of time DECIDING what to do. Factories with systems don’t waste time on deciding what to do. They just spend time on doing it or they spend time on improving their systems. Systems allow you to consistently replicate a process so that you have continued success in the long run.
The rational behind using systems applies to individuals in every endeavor of their life, and it also applies to countries. I am continuously fascinated with the rise and fall of great nations. In particular, I am very interested in massive changes over time. For example, did you know that just 500 years ago, the West was a backwater compared to the East? (e.g. Middle East, ancient China, etc). What changed so rapidly?
The west’s greatest innovation was the creation of SYSTEMS. It started with the ancient Greeks and Romans. Unbiased systems bring stability, predictability, and repeatability. I’ll give you a very simple example. Travel enough around the world and you’ll see why developed countries are the way they are, and why developing countries are the way they are. Developed countries have systems: the rule of law. Everyone follows and lives under a system. In developing countries, the systems are often all over the place. They’ll often have systems, but no one uses the system. Everyone uses a backdoor.
Democracy isn’t necessarily the best political system. It can be slow, there can be a lot of red tape. For example, a good king can do things 100x faster than a Congress filled with 500 men. But a democracy is the most stable long term system. Democracy ensures that everything in government follows systems, such as transition of power systems. In many developing nations, politics is still mostly a function of internal fighting and heredity. Many developing nation “democracies” are shams (just look at Putin).
Effective western systems are notable elsewhere, such as western healthcare systems. In developed western nations, you line up at the hospital, get a number, and wait for your number to be called. In many developing countries, everyone rushes to the hospital, people try to call on connections/favors with doctors, etc. It’s chaos. If you don’t believe me, try visiting a hospital in many parts of Asia.
Man is falleable. Systems are infalleable, IF designed properly. Man is prone to processing error. Machines don’t make errors, unless errors are coded into them in the first place.

  1. Humans make errors when deciding on rules about how to act. Humans also make errors when executing those rules.
  2. Machines can have errors if those errors were coded into the system in the first place. But machines will not have execution errors.

The biggest problem with not backtesting is that you don’t think like a systems trader. 
When people just stare at a chart – whether it be technical indicators, fundamental indicators, chart patterns – they tend to see what they want to see. The mind plays tricks on people to confirm their pre-existing bias.
A lot of times you’ll look at a chart, see an indicator, and think “wow, this is terrific! It timed the market’s turning points very well”. But when you backtest ALL THE DATA and run the numbers, the result ends up being very different. Yes, there are a lot of successes. But there are also a ton of failures, and some of those failures can be devastating (in terms of losses or missed opportunities).
Once again, this is the biggest problem with “eyeball” investors. They see what they want to see (from Psychology Today)
Quantitative backtesting eliminates this bias. The numbers are unbiased and equations are impartial. 1 = 1, and 2 = 2. When you use “eyeballing”, 1 might look like 2, and 2 might look like 1. The eye is not PRECISE.
This is why I don’t like Elliot Wave theory. Eyeball traders start counting Elliot waves. And then when it doesn’t work, they say “oh, let’s combine these 2 waves together!” And then when that doesn’t work, they eyeball the chart again, and say “oh, I guess these waves were part of a bigger wave”.
If Elliot Wave worked, you would be able to QUANTIATIVELY describe it using pure math. When people describe it from a discretionary standpoint by combing and separating waves, all they’re doing is looking in the 20/20 rear mirror.
But what if you just love Elliot Wave theory?
That’s ok! Learn computer programming, learn AI, and then OBJECTIVELY QUANTIFY Elliot Waves. Then you will know exactly how well Elliot Waves work.
Herein lies the biggest problem with any trading strategy that relies on eyeballing a chart. People adjust a chart in a billion ways to fit their pre-existing market outlook.
For example, you’ll often see technicians draw 5-10 different trendlines on a chart. They end up “proving” what you want to see, and it works no better than a coin toss in real time.
It’s just madness.
This is also the problem with Fibonacci retracements and extensions. Some people think of Fibonacci retracements (e.g. 38.2, 50%, 61.8%) as these MAGICAL numbers and lines that the stock market “must touch”. What a load of BS. Fibonacci retracements and extensions are an approximation. There is nothing mystical about them.

  1. Fibonacci retracements start off as being a reflection of the human psyche. E.g. if the stock market crashes “too much”, it’ll probably bounce 38.2% – 61.8%. That’s how the universe works. If you stretch an elastic band too far in one direction, it’ll snap back in the other direction. That’s called mean reversion and cycles. There are cycles in everything – the weather, your career, life/death.
  2. Then the HFT algos pick up on these patterns (e.g. a 50% bounce) and reinforce them.
  3. The “magical” Fibonacci retracement lines are BS, because the market rarely retraces EXACTLY 38.2%, 50%, or 61.8%. So when you look at a fibonacci trader who says “the market will retrace 50% or 61.8%”, the market will rarely retrace exactly 50% or 61.8%. 50% and 61.8% are not “magical numbers”. The market might retrace 55%, 57%, 60% 65%, or 70%. Then when the market retraces eg 59%, these eyeball traders will say “see, I told you it would retrace 61.8%!” You could just as easily set any number as a “magical Fibonacci retracement”, and it would work. For example, set the 50% retracement at 51%, and the 61.8% retracement at 63%.
  4. 38.2% = a little more than 1/3, 50% = half, 61.8% = a little more than 2/3, 76.4% = ¾
  5. It’s madness how traders use Fibonacci stuff as if it were a science. It’s not a science. It’s an approximation.

Many technical traders draw a ton of support and resistances. They always say “see, the support/resistance held! It worked” Well when you draw 10 different lines on a chart, one of them will always “work well” as support/resistance.
This is just madness.
Much of the trading community revolves around drawing random trendlines on a chart, which they can never objectively backtest to see how useful these are in real-time. Most traders in this industry never once ask the most important question. “Is the whole drawing trendlines and staring at chart patterns activity just a load of BS that works great with 20/20 hindsight, but in real time is no better than monkeys throwing darts at a chart?”

1 comment add yours

  1. Hi Troy,
    Good article! Kudos.
    By the way, the correct spelling is infallible not infalleable – (I automatically proofread everything – kind of eyeball-backtesting).
    Happy New Year!
    Mordecai

Leave a Comment