Bias Hunter
  • Home
  • Blog
  • Resources
  • Contact

Deeper Look at the Rationality Test

15/10/2015

0 Comments

 
Okay, so I promised to reveal my own results for the last week’s Rationality test, and also take a deeper look at the questions while I’m at it. So here goes.

You might guess that – based on the fact that rationality is a big theme of this blog – I would receive the score “Rationalist”. Well, you’d be half right. When I was trying out the beta version of the test (with slightly different questions I think), I got “Skeptic”. Also, the more rationalist result of the second try was a lot due to the fact that for some questions I knew what I was supposed to answer. I guess this shows there’s still room for improvement. Anyway, so what are you supposed to answer, and why? I’ll go through the test question-by-question, providing short explanations and links to relevant themes or theories. At the end, I’ll show how the questions relate to the skills and sub-categories.

Question-by-question analysis

1. LA movie
Well, this is just a classic case of sunk costs. You’ve invested time and money into something, but it’s not bringing the utility (or profit, or whatever) that you thought. If you abandon it, you abandon all chance of getting any utility out from the investment. However, as far as rationality is concerned, past costs are not relevant, since you can’t affect them anymore. The only thing that matters is the opportunity cost: should you stay in the movie, or rather do something else. If that something else brings more benefits, rationally you should be doing that.

2. Music or job
This problem is a classic case of how to confuse your opponents in debates. You can see this in politics, like suppose you’re talking about school shootings with a hard-line conservative: “Either we give guns to teachers or we lower the right to bear arms to 8 years old!”. Well, of course you see right away that you’re being presented a false dichotomy: there are many other ways to prevent shootings – like banning all arms altogether. But to skew the debate and try to put you in a position in which you have to accept something you don’t like, your opponent tries to lure you into the these-are-the-only-two-options trap.

3. Doughnut-making machine
Now, this question is basically just simple arithmetic. However, the trick here is that the answer that immediately comes to mind is incorrect, ie. a false System 1 response. Instead, what you need to do is to mentally check that number, see it is wrong, and use System 2 to provide the right answer. The question itself is just a rephrase of one question in the classic Cognitive Reflection Test.

4. Fixating on improbable frightening possibilities
I’m a little puzzled about this question. Sure, I understand the point is that if you’re always fixating on really unlikely bad things, you’re doing something wrong. Still, I find it hard to see anyone would actually be like this!

5. The dead fireman
Now, the point in this question was to see how many possible causes you would think of before deciding on one. The idea is, naturally enough, confirmation bias. We’re too often thinking of a certain explanation, and then immediately jumping to look for confirming evidence. In complex problems, this is a special problem since as we all know, if you torture the numbers enough with statistics, you can make them confess to anything.

6. Things take longer
Well, I presume this simple self-report question is just measuring your susceptibility to the planning fallacy.

7. Bacteria dish
This question has the same idea as the Doughnut-making machine. You get a System 1 answer, suppress it, and (hopefully) answer correctly with System 2. This question is also a rephrase of a question from the Cognitive Reflection Test.

8. Refuting arguments
Being able to argue against other people is a clear sign of rhetorical skills and logical thinking.

9. Budgeting time for the essay
This question checks the planning fallacy. Often, we’re way too optimistic about the time that it takes to complete a project. For example, I’m still writing a paper I expected to be ready for submission in May! In this question, you were awarded full points for assigning at least 3 weeks for the essay, ie. the average of the previous essay completion times.

10. Learning from the past
This is a simple no-trick question that honestly asks whether you learn from your mistakes. I honestly answered that I often do, but sometimes I end up repeating mistakes.

11. BacoNation sales and the ad campaign
This checked your ability to use statistical reasoning. True enough, sales have risen compared to the previous month, but all in all the sales have varied enough to make it plausible that the ad campaign had no effect. In fact, if you pnorm(44.5, mean(data), sd(data)), you get 0.12167, which implies that it’s plausible the September number comes from the same normal distribution. This makes the effect of the ads only somewhat likely.

12. Sitting in the train
So this is the first of the two questions that check how much you value your time. Of course, the point here is that you ought to be consistent. Unfortunately, there may be valid arguments for claiming that you value time on holiday and at home differently, due to differing opportunity costs. See question 20 below for more explanation.

13. Value of time
This question simply asks whether you find it easy or difficult to value your time. Unsurprisingly, the easier you find it the higher your points.

14. One or two meals
Would you rather have one meal now or two meals in a year? This is measuring the discounting of time. Assuming that you’re not starved of food, you presumably should discount meals in the same way as money, since money can obviously buy you meals. See question 21 below for a longer explanation.

15. Continue or quit
Another one of those self-report questions, this is basically asking whether you have fallen into the sunk cost trap.

16. 45 or 90
Here’s another question about time discounting, this time with money. The same assumptions hold as before: we’re assuming you are not in desperate need of money. If that holds, you should discount the same way over all time ranges.

17. Certainty of theory
Can a theory be certain? If you’re a Bayesian (and why wouldn’t you be, right?), you can never set a theory to be 100% certain (let’s ignore tautologies and contradictions here). In a Bayesian framework this would mean that no matter what evidence you observe, the theory can never be proven wrong, because a prior of 1 discounts any evidence for or against it.

18. 100 vs 200
Another discounting question, this time with slightly different amounts of money. Once again, you should discount the same way and choose whatever you chose before. Note that here we are also assuming that 100/200 amounts are close enough to the 40/90 decision – if we had amounts in the millions, that might make a lot of impact.

19. Big assignment
The big assignment vs. small assignments is just a self-report measure to investigate your planning skills.

20. Paying for a task
This question is a sister question to the one where you’re sitting in the train. I presume that the point is that your valuation of one hour should be the same in both question. However, we can question whether the situations are really the same. In one, you’re one holiday, and sitting in a train in a new city has positive value for me. What’s more, on holiday the opportunity costs are different. I’m not really trading off time for working hours, because the point of the holiday mindset is precisely setting aside the possibility of work, so I can enjoy whatever I’m doing – like sitting in a train in a new city. In this question, you’re trying to avoid a task at home, where the opportunity costs of one hour may certainly be different than when you’re on holiday. For example, if you have a job you can do from home, you could be working, or going out with friends, etc.

21. 45 or 90
Well, this is of course part of the other time discounting questions. Here we have the same 45/90 amounts, but the time has been shifted for one year to the future. Again, you should choose whatever you chose before.

All these questions had the similar format:
A dollars in time t vs. B dollars in time t+T

If you’re perfectly rational, you should discount in the same way between times [now, 3 months] and [1 year, 1 year 3 months]. The reason is quite simple: if you’re now willing to wait for the extra three months but not when the lower amount is immediate, you will in the future end up changing your decision. And, if you already know you will change it, why wouldn’t you choose that option already. Hence, you should be consistent. (if you really need an academic citation, here is a good place to start)
 
The score
 So how do these questions make up your score?
If you look at the URL of your results report, you probably see something like
https://www.guidedtrack.com/programs/1q59zh4/run?normalpoints=33&sunkcost=4&planning=3&explanationfreeze=3&probabilistic=4&rhetorical=4&analyzer=3&timemoney=4&intuition=14&future=14&numbers=16&evidence=14&csr=8&enjoyment=0&reportshare=0
You can use that to look at your score by category, for example in my case:
Picture
That’s all for this week, happy rationality (or rationalizing?) to all of you! :)
0 Comments

Basic Biases: The Planning Fallacy

27/10/2014

0 Comments

 
I figured it would probably take a bit more than an hour to write up this post – reality intervened and two hours were exceeded. This is probably an experience many people share, whether from school, work, or personal life. Everything takes much longer than expected. Even though I’ve written countless essays, blog posts and papers – and they’ve all seemed to take commonly longer than estimated – I still didn’t see this coming. There’s a term in decision research for this phenomenon: the planning fallacy. In short, it means we are quite idiotic planners: we never see the delays and problems coming.
Picture
The planning fallacy is ubiquitous to the point of being scary. As Jonathan Baron recounts in his exquisitely fantastic book Thinking and Deciding (pp. 387-388):
Projects of all sorts, from massive works of construction to students’ papers to professors’ textbooks, are rarely completed in the amount of time originally estimated. Cost overruns are much more common than underruns (to the point where my spelling checker thinks that only the former is an English word). In 1957, for example, the Sydney (Australia) Opera House was predicted to cost $17 million and to be completed in 1963. A scaled-down version opened in 1973, at a cost of $102 million! 
When it comes to our predictions of completion times, there seems to be no such thing as learning from history. It seems that every project we begin is considered only as a single instance, not as belonging to a category.

In fact, herein lies one of the keys to overcoming the problem. Relating a single project to a reference class of projects encourages statistical, data-driven thinking and reveals the delays to us. In an experiment (Buehler et al., 1994), students were much better at predicting when they were going to complete assignments when they were told to think about the past and connect it to the present assignment.

In my opinion, there are two kinds of errors underlying the planning fallacy:

1)      Thinking we will have more time or energy in the future
2)      Thinking we have prevented the errors that have happened before

The first case is especially related to personal projects. All the time there are several projects demanding attention. When we predict the future, we tend to believe that we will have more free time in the future than now. Since all those meetings, unpredictable surprises and events we just want to be at our not yet in view, the future looks promisingly empty. However, once it arrives it is very likely to look much like the present: the Sunday afternoon I was supposed to use for fixing my bike and catch up on reading was spent over a delightful brunch that we agreed on just a few days in advance. And so on. It never turns out like you planned.
Picture
Plan vs. reality
In the second case – and I think this might be more relevant in complex projects – we believe that we have in fact learned from history, that we’ve plugged the gaps that proved our downfall in previous projects. Unfortunately, this is exactly what we thought previously, too! Surprising problems in project are surprising exactly because they are not the same errors as before. As before, just round the corner is something we didn’t think of, something that catches us off guard. If we only remembered that it’s always been like this…

Of course, I’m deliberately painting a gloomy picture here. We do learn from our mistakes and tend not to make the same mistakes again. But the point is that there will probably always be new mistakes – and this ought to be reflected in planning. I don’t think anybody is better off from the fact that projects run over their deadlines almost certainly. For example, by some estimates only 1% of military purchases are delivered on time and budget (Griffin & Buehler, 1999)! There’s certainly room for improvement!

And that improvement is possible. According to Buehler et al. (1994), others are less susceptible to the planning fallacy than the actors themselves. In simple terms: those not invested in the plan are more likely to have a conservative view of the completion time. Unfortunately, they are also likely to err in the opposite direction by stating too slow completion times! So, it’s probably a feasible idea to estimate the actual time by averaging your estimate and the outsider’s estimate. This is hardly optimal, but it’s better than relying on a single biased estimate.

So the recipe for better plans seems quite straightforward:

-          stopping to think
-          considering past projects and their results
-          getting an outsider opinion and aggregating estimates
0 Comments

    RSS Feed

    Archives

    December 2016
    November 2016
    April 2016
    March 2016
    February 2016
    November 2015
    October 2015
    September 2015
    June 2015
    May 2015
    April 2015
    March 2015
    February 2015
    January 2015
    December 2014
    November 2014
    October 2014
    September 2014
    August 2014

    Categories

    All
    Alternatives
    Availability
    Basics
    Books
    Cognitive Reflection Test
    Conferences
    Criteria
    Culture
    Data Presentation
    Decision Analysis
    Decision Architecture
    Defaults
    Emotions
    Framing
    Hindsight Bias
    Improving Decisions
    Intelligence
    Marketing
    Mindware
    Modeling
    Norms
    Nudge
    Organizations
    Outside View
    Phd
    Planning Fallacy
    Post Hoc Fallacy
    Prediction
    Preferences
    Public Policy
    Rationality
    Regression To The Mean
    Sarcasm
    Software
    Status Quo Bias
    TED Talks
    Uncertainty
    Value Of Information
    Wellbeing
    Willpower

Powered by Create your own unique website with customizable templates.