Skip to content

Archive for

Jun 29

8 Comments

#52 Top thirteen topics

After I present an activity on a particular statistical topic while conducting a workshop for teachers, I often say something like: I think this is one of the top ten things for students to learn in introductory statistics.  Naturally enough, a workshop participant always asks me to provide my complete “top ten” list.  My sheepish response has always been to beg off, admitting that I have never taken the time to sit down and compile such a list*.

* Workshop participants have always been too polite to ask how, in that case, I can be so sure that the topic in question is actually on that imaginary list of mine.

To mark the 52nd post and one-year milestone for this weekly blog, I have finally persuaded myself to produce my list of most important topics for students to learn in introductory statistics.  I hope you will forgive me for expanding the number of topics to a lucky thirteen*.  Commenting on this list also provides an opportunity for me to reflect on several earlier posts from my year of blogging. * I also recommend the “top seven” list produced by Jessica Utts in an article for The American Statistician in 2003 (here), to which she added an additional four topics at an ICOTS presentation in 2010 (here).

Unlike previous posts, this one poses no questions for students to appear in italics.  Instead I focus on the question that has often been asked of me: What are the most important topics for students to learn in introductory statistics?

1. Identifying observational units and variables points the way.

In post #11 (Repeat after me, here), I repeated over and over again that I ask students to identify observational units and variables for almost every example that we study throughout the entire course.  Along with identifying the variables, I ask students to classify them as categorical or numerical, explanatory or response.  Thinking through these aspects of a statistical study helps students to understand how the study was conducted and what its research questions were.  These questions also point the way to knowing what kind of graph to produce, what kind of statistic to calculate, and what kind of inference procedure to conduct.  I have found that identifying observational units and variables is more challenging for students than I used to think.

One of my favorite examples to illustrate this concerns the murder trial of Kristen Gilbert, a nurse accused of being a serial killer of patients.  The following data were presented at her trial:

The observational units here are hospital shifts, not patients.  The explanatory variable is whether or not Gilbert was working on the shift, which is categorical and binary.  The response variable is whether or not a patient died on the shift, which is also categorical and binary.  Students need to understand these basic ideas before they can analyze and draw conclusions from these data.

2. Proportional reasoning, and working with percentages, can be tricky but are crucial.

I suspect that at least two-thirds of my blog posts have included proportions or percentages*.  Proportions and percentages abound in everyday life.  Helping students to work with percentages, and to recognize the need for proportional reasoning, is a worthy goal for introductory statistics courses.

* This very sentence contains a proportion, even if it is only a guess.

Look back at the table of counts from the Kristen Gilbert trial.  Students who do not think proportionally simply compare the counts 40 and 34, which suggests a small difference between the groups.  But engaging in proportional reasoning reveals a huge discrepancy: 40/257 ≈ 0.156 and 34/1384 ≈ 0.025.  In other words, 15.6% of shifts on which Gilbert worked saw a patient death, compared to 2.5% of shifts on which Gilbert did not work.  These percentages are displayed in the segmented bar graph:

What’s so tricky about this?  Well, converting the proportions to statements involving percentages is non-trivial, particularly as these are conditional percentages.  More challenging is that many students are tempted to conclude that the death rate on Gilbert shifts is 13.1% higher than the death rate on non-Gilbert shifts, because 0.156 – 0.025 = 0.131.  But that’s not how percentage difference works, as I ranted about at length in post #28 (A pervasive pet peeve, here).  The actual percentage difference in the death rates between these groups is (0.156 – 0.025) / 0.025 × 100% ≈ 533.6%.  Yes, that’s right: The death rate on a Gilbert shift was 533.6% higher than the death rate on a non-Gilbert shift!  This gives quite a different impression that the incorrect claim of a 13.1% difference.

The importance of proportional reasoning also arises when working with probabilities.  I strongly recommend producing a table of hypothetical counts to help students work with conditional probabilities.  For example, I used that technique in post #10 (My favorite theorem, here) to lead students to distinguish between two conditional probabilities: (1) the probability that a person with a positive test result has the disease, and (2) the probability that the test result is positive among people who have the disease, as shown in the table:

3. Averages reveal statistical tendencies.

The concept of a statistical tendency is a fundamental one that arises in all aspects of life.  What do we mean when we say that dogs are larger than cats?  We certainly do not mean that every dog is larger than every cat.  We mean that dogs tend to be larger than cats.  We also express this idea by saying that dogs are larger than cats on average.  We can further explain that if you encounter a dog and a cat at random, it’s more likely than not that the dog will be larger than the cat*.

Understanding statements of statistical tendencies, and learning to write such statements clearly, is an important goal for introductory statistics students.  Is this an easy goal to achieve?  Not at all.  I mentioned in post #37 (What’s in a name? here) that psychologist Keith Stanovich has described this skill, and probabilistic reasoning more generally, as the “Achilles Heel” of human cognition.

The dogs and cats example is an obvious one, but averages can also help us to see a signal in the midst of considerable noise.  Post #9 (Statistics of illumination, part 3, here) about the infamous 1970 draft lottery illustrates this point.  The scatterplot on the left, displaying draft number versus day of the year, reveals nothing but random scatter (noise) on first glance.  But calculating the median draft number for each month reveals a clear pattern (signal), as shown on the right:

You might be thinking that students study averages beginning in middle school or even sooner, so do we really need to spend time on averages in high school or college or courses?  In post #5 (A below-average joke, here), I argued that we can help students to develop a deeper understanding of how averages work by asking questions such as: How could it happen that the average IQ dropped in both states when I moved from Pennsylvania to California?

4. Variability, and distributional thinking, are fundamental.

Averages are important, but variability is at the core of statistical thinking.  Helping students to regard a distribution of data as a single entity is important but challenging.  For example, post #4 (Statistics of illumination, part 2, here) described an activity based on data about readability of cancer pamphlets.  I ask students to calculate medians for a dataset on pamphlet readability and also for a dataset on patient reading levels.  The medians turn out to be identical, but that only obscures the more important point about variability and distribution.  Examining a simple graph reveals the underlying problem that many patients lack the skill to read the simplest pamphlet:

In posts #6 and #7 (Two dreaded words, here and here), I suggested that we can help students to overcome their dread of the words standard deviation by focusing on the concept of variability rather than dreary calculations that are better performed by technology.  I also argued in post #8 (End of the alphabet, here) that z-scores are an underappreciated idea that enable us to compare proverbial apples and oranges by taking variability into account.

5. Visual displays of data can be very illuminating.

In light of the graphs presented above, I trust that this point needs no explanation.

6. Association is not causation; always look for other sources of variability.

Distinguishing causation from association often prompts my workshop comment that I mentioned in the first sentence of this post.  I want students to emerge from their introductory statistics course knowing that inferring a cause-and-effect relationship from an observed association is often unwarranted.  Posts #43 and #44 (Confounding, here and here) provide many examples.

The idea of confounding leads naturally to studying multivariable thinking.  Post #3 (Statistics of illumination, part 1, here) introduced this topic in the context of graduate admission decisions.  Male applicants had a much higher acceptance rate than female applicants, but the discrepancy disappeared, and even reversed a bit, after controlling for the program to which they applied.  For whatever reason, most men applied to the program with a high acceptance rate, while most women applied to the program with a very low acceptance rate.

Post #35 (Statistics of illumination, part 4, here) continued this theme in the context of comparing lung capacities between smokers and non-smokers.  Surprisingly enough, smokers in that study tended to have larger lung capacities than non-smokers.  This perplexing result was explained by considering the ages of the people, who were all teenagers and younger.  Smokers were much more likely to be older than younger, and older kids tended to have larger lung capacities than younger ones.  The following graph reveals the relationships among all three variables:

7. Randomized experiments, featuring random assignment to groups, allow for cause-and-effect conclusions.

Some students take the previous point too far, leaving their course convinced that they should never draw cause-and-effect conclusions.  I try to impress upon them that well-designed randomized experiments do permit drawing cause-and-effect conclusions, as long as the difference between the groups turns out to be larger than can plausibly be explained by random chance.  Why are the possible effects of confounding variables less of a concern with randomized experiments?  Because random assignment of observational units to explanatory variable groups controls for other variables by balancing them out among the groups.

Post #20 (Lincoln and Mandela, part 2, here) describes a class activity that investigates the psychological phenomenon known as anchoring by collecting data from students with a randomized experiment.  Students are asked to guess the age at which Nelson Mandela died, but some students first see the number 16 while others see the number 160.  The following graph displays the responses for one of my classes. These data strongly suggest that those primed with 160 tend to make larger guesses than those primed with 16:

Posts #27 and #45 (Simulation-based inference, parts 2 and 3, here and here) also featured randomized experiments.  We used simulation-based inference to analyze and draw conclusions from experiments that investigated effects of metal bands on penguin survival and of fish oil supplements on weight loss.

8. Random sampling allows for generalizing, but it’s very hard to achieve.

Random sampling is very different from random assignment.  These two techniques share an important word, but they have different goals and consequences.  Random sampling aims to select a representative sample from a population, so results from the sample can be generalized to the population.

I described how I introduce random sampling to my students in post #19 (Lincoln and Mandela, part 1, here).  In this activity, students select samples of words from the Gettysburg Address.  First students select their sample simply by circling ten words that appeal to them.  They come to realize that this sampling method is biased toward longer words.  Then they use genuine random sampling to select their sample of words, finding that this process is truly unbiased.  The following graphs (from the applet here) help students to recognize the difference between three distributions: 1) the distribution of word lengths in the population, 2) the distribution of word lengths in a random sample from that population, and 3) the distribution of sample mean word lengths in 1000 random samples selected from the population:

I emphasize to students that while selecting a random sample of words from a speech is straight-forward, selecting a random sample of human beings is anything but.  Standing in front of the campus library or recreation center and selecting students in a haphazard manner does not constitute random sampling.  Even if you are fortunate enough to have a list of all people in the population of interest from which to select a random sample, some people may choose not to participate, which leaves you with a non-random sample of people for your study.

9. Analyzing random phenomena requires studying long-run behavior.

There’s no getting around the fact that much of statistics, and all of probability, depends on asking: What would happen in the long run?  Such “long run” concepts are hard to learn because they are, well, conceptual, rather than concrete.  Fortunately, we can make these concepts more tangible by employing the most powerful tool in our pedagogical toolbox: simulation!

Post #17 (Random babies, here) presents an activity for introducing students to basic ideas of randomness and probability.  Students use index cards to simulate the random process of distributing four newborn babies to their mothers at random.  Then they use an applet (here) to conduct this simulation much more quickly and efficiently.  Post #18 (What do you expect? here) follows up by introducing the concept of expected value.  The following graph shows how the average number of correct matches (of babies to mothers) changes for the first 1000 repetitions of simulating the random process, gradually approaching the long-run average of 1.0:

The usefulness of simulation for studying and visualizing randomness permeates all of the posts about probability.  For example, post #23 (Random rendezvous, part 1, here) presents the following graph of simulation results to display the probability that two people successfully meet for lunch, when their arrival times are independent uniform distributions and they agree to wait fifteen minutes for each other:

10. Sampling distributions lay the foundation for statistical inference.

One of the questions posed by prospective teachers in post #38 (here) asked me to identify the most challenging topic for introductory statistics students.  My response was: how the value of a sample statistic varies from sample to sample, if we were to repeatedly take random samples from a population.  Of course, for those who know the terminology*, I could have answered with just to words: sampling distributions.  I expanded on this answer in posts #41 and #42 (Hardest topic, here and here).

* Dare I say jargon?

Understanding how a sample statistic varies from sample to sample is crucial for understanding statistical inference. I would add that the topic of randomization distributions deserves equal status with sampling distributions, even though that term is much less widely used.  The difference is simply that whereas the sampling distribution of a statistic results from repeated random sampling, the randomization distribution of a statistic results from repeated random assignment.  In his classic article titled The Introductory Statistics Course: A Ptolemaic Curriculum? (here), George Cobb argued that statistics teachers have done a disservice to students by using the same term (sampling distributions) to refer to both types, which has obscured the important distinction between random sampling and random assignment.

You will not be surprised that I consider the key to studying both sampling distributions and randomization distributions to be … drumroll, please … simulation!

11. Confidence intervals estimate parameters with a margin-of-error.

The need for interval estimation arises from the fundamental idea of sampling variability, and the concept of sampling distributions provides the underpinning on which confidence interval procedures lie.  I described activities and questions for investigating confidence intervals in a three-part series of posts #14, #15, and #46 (How confident are you? here, here, and here).

In post #15 (here), I argued that many students fail to interpret confidence intervals correctly because they do not think carefully about the parameter being estimated.  Instead, many students mistakenly interpret a confidence interval as a prediction interval for an individual observation.  Helping students to recognize and define parameters clearly is often overlooked but time well spent.

As with many other topics, interactive applets can lead students to explore properties of confidence intervals.  The following graph, taken from post #14 (here) using the applet here, illustrates the impact of confidence level, while revealing that confidence level refers to the proportion of intervals (in the long run, under repeated random sampling) that successfully capture the value of the population parameter:

12. P-values indicate how surprising the sample result would be if a hypothesized model were true.

The p-value has been the subject of much criticism and controversy in recent years (see the 2019 special issue of The American Statistician here).  Some have called for eliminating the use of p-values from scientific inquiry and statistical inference.  I believe that p-values are still essential to teach in introductory statistics, along with the logic of hypothesis testing.  I think the controversy makes clear the importance of helping students to understand the concept of p-value in order to avoid misuse and misinterpretation.

Yet again I advocate for using simulation as a tool for introducing students to p-values.  Many posts have tackled this topic, primarily the three-part series on simulation-based inference in posts #12, #27, and #45 (here, here, and here).  This topic also featured in posts #2 (My favorite question, here), #9 (Statistics of illumination, part 3, here), and #13 (A question of trust, here).

The basic idea behind a p-value is to ask how likely an observed sample result would be if a particular hypothesis about a parameter were true.  For example, post #12 (here) described a study that investigated whether people are more likely to attach the name Tim (rather than Bob) to the face on the left below:

When I asked this question of my students in a recent class, 36 of 46 students associated Tim with the face on the left.  A simulation analysis of 10,000 coin flips (using the applet here) reveals that such an extreme result would happen very rarely with a 50-50 random process, as shown in the graph below.  Therefore, we conclude that the sample result provides strong evidence against the 50-50 hypothesis in favor of the theory that people are more likely to attach the name Tim to the face on the left.

13. Statistical inference does not reveal, or account for, everything of interest.

It’s imperative that we statistics teachers help students realize that statistical inference has many, many limitations.  This final topic on my list is a catch-all for many sub-topics, of which I describe a few here.

I mentioned the importance of interval estimates earlier, but margin-of-error does not account for many things that can go wrong with surveys.  Margin-of-error pertains to variability that arises from random sampling, and that’s all.  For example, margin-of-error does not take into account the possibility of a biased sampling method.  I described one of my favorite questions for addressing this, with an admittedly ridiculous context, in post #14 (How confident are you? Part 1, here).  If an alien lands on earth, sets out to estimate the proportion of humans who identify as female, and happens upon the U.S. Senate as its sample, then the resulting confidence interval will drastically underestimate the parameter of interest.

Margin-of-error also fails to account for other difficulties of conducting surveys, such as the difficulty of wording questions in a manner that does not influence responses, and the distinct possibility that some people may exaggerate or lie outright in their response.

The distinction between statistical significance and practical importance is also worth emphasizing to students.  One of my favorite questions for addressing this is a somewhat silly one from post #16 (Questions about cats, here).  Based on a large survey of households in the U.S., the proportion of households with a pet cat differs significantly from one-third but is actually quite close to one-third.  Reporting a confidence interval is much more informative than simply producing a p-value in this context and many others.

Another misuse of p-values is to mindlessly compare them to 0.05 as a “bright line” that distinguishes significant results from insignificant ones.  In fact, the editorial (here) in the special issue of The American Statistician mentioned above calls for eliminating use of the term statistical significance in order to combat such “bright line” thinking.

A related and unfortunately common misuse is the practice of p-hacking, which means to conduct a very large number of hypothesis tests on the same dataset and then conclude that those with a p-value less than 0.05 are noteworthy.  A terrific illustration of p-hacking is provided in the xkcd comic here (with explanation here).

Writing this blog for the past year and compiling this list have helped me to realize that my own teaching is lacking in many respects.  I know that if I ever feel like I’ve got this teaching thing figured out, it will be time for me to retire, both from teaching and from writing this blog.

But I am far from that point.  I look forward to returning to full-time teaching this fall after my year-long leave*.  I also look forward to continuing to write blog posts that encourage statistics teachers to ask good questions. 

* I picked a very, shall we say, eventful academic year in which to take a leave, didn’t I?

In the short term, though, I am going to take a hiatus in order to catch my breath and recharge my batteries.  I am delighted to announce that this blog will continue uninterrupted, featuring weekly posts by a series of guest bloggers over the next couple of months.

Oh wait, I just realized that I still have not answered a question that I posed in post #1 (here) and promised to answer later: What makes a question good?  I hope that I have illustrated what I think makes a question good with lots and lots and lots of examples through the first 51 posts.  But other than providing examples, I don’t think I have a good answer to this question yet.  This provides another motivation for me to continue writing this blog.  I will provide many, many more examples of what I think constitute good questions for teaching and learning introductory statistics.  I will also continue to reflect on this thorny question (what makes a question good?), and I vow once again to answer the question in a later* post.

* Possibly much later

P.S. I greatly appreciate Cal Poly’s extending a professional leave to me for the past year, which has afforded me the time to write this blog.  

I extend a huge thanks to Beth Chance and Tom Moore, who have read draft posts and offered helpful comments every week*.

* Well, except for the weeks in which I was unable to produce a draft in time.

My final and sincere thanks go to all of you who have read this blog and encouraged me over the past year.

Jun 22

1 Comment

#51 Randomness is hard

I enjoy three-word sentences, such as: Ask good questions. I like cats*. What about lunch**?  Here’s another one: Randomness is hard.

* See post #16 (Questions about cats, here).

** I borrowed this one from Winnie-the-Pooh (see here).

What do I mean when I say that randomness is hard? I mean several things: Randomness is hard to work with, hard to achieve, hard to study.  For the purpose of this post, I mean primarily that randomness is hard to predict, and also that it’s hard to appreciate just how hard randomness is to understand.

Psychologists have studied people’s misconceptions about randomness for decades, and I find these studies fascinating.  I try not to overuse class examples that emphasize misunderstandings, but I do think there’s value in helping students to realize that they can’t always trust their intuition when it comes to randomness.  Applying careful study and thought to the topic of randomness can be worthwhile.

In this post, I discuss some examples that reveal surprising aspects of how randomness behaves and lead students to recognize some flaws in most people’s intuition about randomness.  As always, questions that I pose to students appear in italics.

I ask my students to imagine a light that flashes every few seconds.  The light randomly flashes a green color with probability 0.75 and red with probability 0.25, independently from flash to flash.  Then I ask: Write down a sequence of G’s (for green) and R’s (for red) to predict the colors for the next 40 flashes of this light.  Before you read on, please take a minute to think about how you would generate such a sequence yourself.

Most students produce a sequence that has 30 G’s and 10 R’s, or close to those proportions, because they are trying to generate a sequence for which each outcome has a 75% chance for G and a 25% chance for R.  After we discuss this tendency, I ask: Determine the probability of a correct prediction (for one of the outcomes in the sequence) with this strategy.

We’ll figure this out using a table of hypothetical counts*.  Suppose that we make 400 predictions with this strategy.  We’ll fill in the following table by assuming that the probabilities hold exactly in the table:

* For more applications of this method, see post #10 (My favorite theorem, here).

First determine the number of times that the light flashes green and the number of times that the light flashes red:

Now fill in the counts for the interior cells of the table.  To do this, remember that the strategy is to predict green 75% of the time and to predict red 25% of the time, which gives:

Fill in the remaining totals.  This gives:

How many times is your prediction correct?  You correctly predict a green light 225 times (top left cell of the table), and you correctly predict a red light 25 times (bottom right), so you are correct 250 times.  These counts are shown in bold here:

For what proportion of the 400 repetitions is your prediction correct?  You are correct for 250 of the 400 repetitions, which is 250/400 = 5/8 = 0.625, or 62.5% of the time. 

Here’s the key question: This is more than half the time, so that’s pretty good, right?  Students are tempted to answer yes, so I have to delicately let students know that this percentage is actually, well, not so great. 

Describe a method for making predictions that would be correct much more than 62.5% of the time.  After a few seconds, I give a hint: Don’t overthink.  And then: In fact, try a much more simple-minded approach.  For students who have not yet experienced the aha moment, I offer another hint: How could you be right 75% of the time?

This last question prompts most students to realize that they could have just predicted green for all 40 flashes.  How often will your prediction be correct with this simple-minded strategy?  You’ll be correct whenever the light flashes green, which is 75% of the time.  Fill in the table to analyze this strategy.  The resulting table is below, with correct predictions again shown in bold.  Notice that your prediction from this simple-minded strategy is correct for 300 of the 400 repetitions:

I learned of this example from Leonard Mlodinow’s book The Drunkard’s Walk: How Randomness Rules Our Lives.  I recount for my students the summary that Mlodinow provides: “Humans usually try to guess the pattern, and in the process we allow ourselves to be outperformed by a rat.”  Then I add: Randomness is hard*.

* At least for humans!

What percent better does the simple-minded (rat) strategy do than the guess-the-pattern (human) strategy?  Well, we have determined these probabilities to be 0.750 for rats and 0.625 for humans, so some students respond that rats do 12.5% better.  Of course, that’s not how percentage change works*.  The correct percentage difference is [(0.750 – 0.625) / 0.625] × 100% = 20.0%.  Rats do 20% better at this game than humans.

* I discussed this at length in post #28 (A persistent pet peeve, here).

For more mathematically inclined students taking a probability course, I often ask a series of questions that generalizes this example: Now let p represent the probability that the light flashes green.  Let’s stipulate that the light flashes green more often than red, so 0.5 < p < 1.  The usual (human) strategy is to guess green with probability p and red with probability (1 – p).  Determine the probability of guessing correctly with this strategy, as a function of p.

We could use a table of hypothetical counts again to solve this, but instead let’s directly use the law of total probability, as follows:

Graph this function.  Here’s the graph:

Describe the behavior of this function.  This function is increasing, which makes sense, because your probability of guessing correctly increases as the lop-sidedness of the green-red breakdown increases.  The function equals 0.5 when p = 0.5 and increases to 1 when p = 1.  But the increase is more gradual for smaller values of p than for larger values of p, so the curve is concave up.

Determine the probability of a correct guess for our rat friends, as a function of p.  This one is easy, right?  Pr(correct) = p.  That’s all there is to it.  Rats will always guess green, so they guess correctly at whatever probability green appears.

Graph these two functions (probability of guessing correctly for humans and rats) on the same scale.  Here goes, with the human graph in black and the rat graph in blue:

For what values of p does the rat do better (i.e., have a higher probability of success) than the human?  That’s also easy: All of them!*  Randomness is hard.

* Well, okay, if you want to be technical: Rats and humans tie at the extremes of p = 0.5 and p = 1.0, in case that provides any consolation for your human pride.

Where is the difference between the human and rat probabilities maximized?  Examining the graph that presents both functions together, it certainly looks like the difference is maximized when p = 0.75.  We can confirm this with calculus, by taking the derivative of p2 + (1-p)2 – p, setting the derivative equal to zero, and solving for p.

The “rats beat humans” example reminds me of a classic activity that asks students: Produce a sequence of 100 H’s and T’s (for Heads and Tails) that you think could represent the results of 100 flips of a fair coin. 

Your prediction will be correct 50% of the time no matter how you write your sequence of Hs and Ts.  This activity focuses on a different aspect of randomness, namely the consequence of the independence of the coin flips.  Only after students have completed their sequence do I reveal what comes next: Determine the longest run of consecutive heads in your sequence.  Then I have students construct a dotplot on the board of the distribution of their values for longest run of heads.

How can we investigate how well students performed their task of producing a sequence of coin flip outcomes?  Yet again the answer I am fishing for is: Simulate!  The following graph displays the resulting distribution of longest runs of heads from simulating one million repetitions of 100 flips of a fair coin:

The mean of these one million results is 5.99 flips, and the standard deviation is 1.79 flips.  The maximum value is 25.  The proportion of repetitions that produced a longest run of 5 or more flips is 0.810, and the proportion that produced a longest run of 8 or more flips is 0.170.

How do you anticipate students’ results to differ from simulation results?  Student-generated sequences almost always have a smaller mean, a smaller standard deviation, and a smaller proportion for (5 or more) and for (8 or more).  Why?  Because people tend to overestimate how often the coin alternates between heads and tails, so they tend to underestimate the average length for the longest consecutive run of heads.  In other words, people generally do a poor job of producing a plausible sequence of heads and tails.  Randomness is hard.

As a class activity, this is sometimes conducted by having half the class generate a sequence of coin flips in their head and the other half use a real coin, or a table of random digits, or a calculator, or a computer.  The instructor leaves the room as both groups put a dotplot of their distributions for longest runs of heads on the board.  When the instructor returns to the room, not knowing which graph is which, they can usually make a successful prediction for which is which by guessing that the student-generated graph is the one with a smaller average and less variability.

As another example that illustrates the theme of this post, I ask my students the “Linda” question made famous by cognitive psychologists Daniel Kahneman and Amos Tversky: Linda is 31 years old, single, outspoken, and very bright.  She majored in philosophy.  As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.  Which is more probable? (1) Linda is a bank teller. (2) Linda is a bank teller and is active in the feminist movement.

Kahneman and Tversky found that most people answer that (2) is more probable than (1), and my students are no exceptions.  This is a classic example of the conjunction fallacy: It’s impossible for the conjunction (intersection) of two events to be more probable than either of the events on its own.  In other words, there can’t be more feminist bank tellers in the world than there are bank tellers overall, feminist or otherwise.  In more mathematical terms, event (2) is a subset of event (1), so (2) cannot be more likely than (1).  But most people respond with the impossibility that (2) is more likely than (1).  Randomness is hard.

When I present these examples for students, I always hasten to emphasize that I am certainly not trying to make them feel dumb or duped.  I point out repeatedly that most people are fooled by these questions.  I try to persuade students that cognitive biases such as these are precisely why it’s important to study randomness and probability carefully. 

I also like to think that these examples help students to recognize the importance of humility when confronting randomness and uncertainty.  Moreover, because randomness and uncertainty abound in all aspects of human affairs, I humbly suggest that a dose of humility might be helpful at all times. That thought gives me another three-word sentence to end with: Let’s embrace humility.

P.S. I learned of the activity about longest run of heads from Activity-Based Statistics (described here) and also an article by Mark Schilling (here).

I highly recommend Daniel Kahneman’s book Thinking: Fast and Slow and also Michael Lewis’s book about Kahneman and Tversky’s collaboration and friendship, The Undoing Project: A Friendship that Changed Our Minds.

Jun 15

2 Comments

#50 Which tire?

Perhaps you and your students have heard the campus legend* about two students who miss an exam due to excessive partying, but they tell their professor that they had a flat tire.  They realize that this sounds like a flimsy excuse, so they are pleasantly surprised when the professor accepts their explanation and offers a make-up exam on the following morning.  When they arrive for the make-up exam, they are sent to two separate rooms.  They find question 1, worth 5 points, to be quite straight-forward.  Then they turn the page to find question 2, worth 95 points: Which tire was it?

* I first heard of this story from the “Ask Marilyn” column in the Parade section of the Sunday newspaper on March 3, 1996.  Laurie Snell wrote about this for Chance News (here).  Laurie wrote to the professor involved, a chemist named Dr. Bonk at Duke University.  Dr. Bonk confirmed that something of the sort had happened, but he could not remember the details and suspected that they had been embellished over time.

I ask my students to imagine themselves in this nerve-wracking situation, even though I know that none of them would ever tell a lie to a professor.  I ask them to think about which tire they would say – left front, left rear, right front, or right rear – and write down their answer.  Before I continue with this post, let me ask you to decide on your answer.

Then I predict that one particular tire tends to be selected more often than random chance would expect – the right front tire.  Next we gather data on their response with a simple show of hands.  Telling the story and collecting the data takes less than five minutes of class time.

Here’s the great thing: In addition to getting a laugh from a fun story, you can use these data to introduce or review several topics in an introductory statistics course.  Below I will present and describe six extended exercises, all with different learning objectives, based on this fun and quick data collection exercise.  The topics* of these exercises are:

  1. Simulation-based inference
  2. Binomial distribution
  3. Sample result in opposite direction from conjecture
  4. One-proportion z-test and z-interval
  5. Impact of sample size on p-value, confidence interval
  6. Chi-square goodness-of-fit test

* If you do not have time to read all of these, I recommend #3 and #5 as the least routine.

As always, questions that I pose to students appear in italics.

The first exercise uses class data from this activity to practice applying simulation-based inference with a null hypothesis other than 50/50 in which to apply simulation-based inference, unlike the studies on blindsight and facial prototyping (as in post #12 here) and choice of Halloween treats (as in post #13 here).

1. In the spring quarter of 2018, 17 of 44 students in my class selected the right front tire.

  • a) Identify the observational units and variable.  Also classify the variable as categorical or numerical.
  • b) State (in words) the null and alternative hypotheses to be tested.
  • c) Calculate the sample proportion of students who selected the right front tire.
  • d) Specify the input values for a simulation analysis to assess the strength of evidence for my claim provided by the data.
  • e) Run the simulation analysis, and describe the resulting null distribution of the sample proportion.
  • f) Report and interpret the approximate p-value.
  • g) Summarize your conclusion, and explain how it follows from the p-value.

As I described in post #11 (Repeat after me, here), I like to ask part (a) repeatedly (as I described in post #Z).  The observational units are students, and the variable is which tire they pick.  This variable is categorical, not binary except that my conjecture treats it as binary.  The null hypothesis is that 25% of all students would pick the right front tire, in other words that there’s nothing special about the right front tire.  The alternative hypothesis is that more than 25% of all students would pick the right front tire, that there’s something special about the right front tire that makes it pop into minds first.  The sample proportion who selected the right front tire is 17/44 ≈ 0.386.

To conduct a simulation analysis, the input values are a success probability of 0.25, sample size of 44, and a large number such as 1000 or 10,000 repetitions.  Using an applet (here) to run this simulation produces an approximate null distribution as shown on the left:

The graph on the right reveals that the approximate p-value is 0.0311.  This means that if students only had a 25% chance of picking the right front tire, then there’s a little more than a 3% chance that 17 or more of 44 students would have picked the right front tire.  Less than 0.05 but greater than 0.01, this is a fairly small, but not very small, p-value.  We conclude that the sample data provide fairly strong, but not very strong, evidence for the theory that students pick the right front tire more than would be expected by random chance.

I also use these data to give students experience with recognizing and applying the binomial probability distribution.

2. Let the random variable X represent the number of students in a class of 44 who select the right front tire.  Assume that each student makes their selection independently. Also assume (for now) that each student selects randomly from the four tire options.

  • a) Describe the probability distribution of X by giving its name and specifying its parameter values.
  • b) Calculate Pr(X ≥ 17).  Feel free to use software or a calculator.  Show how you calculate this.
  • c) In the spring quarter of 2018, 17 of 44 students in my class selected the right front tire.  What conclusion would you draw?  Explain how this conclusion follows from the probability in (b).

The probability distribution of X is binomial with parameters n = 44 and p = 0.25.  We can calculate Pr(X ≥ 17) by taking 1 – Pr(X ≤ 16) ≈ 1 – 0.9682 = 0.0318.  This is a fairly small probability, so the observed result would be fairly surprising if the assumption that p = 0.25 were true, so the sample data provide fairly strong evidence that students have a higher probability than 0.25 for selecting the right front tire.

The “which tire” in-class data collection activity is not foolproof, in that the sample data does not always turn out as predicted.  But a disappointing result can provide an opportunity for a worthwhile lesson.

3. At a recent workshop for college professors, 8 of 36 workshop participants selected the right front tire.

  • a) Explain why it’s not necessary to carry out calculations for a hypothesis test of whether the sample data provide strong evidence that people select the right front tire more often than would be expected by random chance.
  • b) Without performing an analysis, what can you say about how the p-value would turn out? 
  • c) Based on this sample result, would you reject the null hypothesis?  Explain.
  • d) Based on this sample result, would you accept the null hypothesis?  Explain.

Before we jump in to perform a full hypothesis test, I encourage students to look at the sample result: Only 8/36 ≈ 0.222 of the sample selected the right front tire.  This is less than one-fourth, so this is in the wrong direction from our conjecture and the alternative hypothesis.  In light of this, we already know (of course!) that the sample data do not provide strong evidence to suggest that more than one-fourth of all people would select the right front tire.  There’s no need to conduct a formal test to realize and conclude this. 

I think this is a fruitful conversation to have with students, who are often tempted to follow a procedure, or plug into a formula, without thinking things through in advance.

If we were to calculate a p-value here, we know that it would be greater than 0.5.  (In fact, the binomial p-value turns out to be 0.710.)  We certainly do not reject the null hypothesis.  But we also cannot accept the null hypothesis, because there are many other potential values of the parameter than are also consistent with the sample result.

I also like to use a larger sample size, by combining results across several classes, to give students practice with applying a one-proportion z-test.

4. In the winter quarter of 2017, 56 of 120 students across my several classes selected the right front tire.

  • a) Calculate the proportion of these students who selected the right front tire.  Is this a parameter or a statistic?  Explain.
  • b) Write a sentence describing the parameter of interest.
  • c) State the null and alternative hypotheses to be tested.
  • d) Check whether the sample size conditions for a one-proportion z-test are satisfied.
  • e) Calculate and interpret the value of the test statistic.
  • f) Summarize your conclusion, and provide justification based on the test statistic.
  • g) Calculate and interpret a 95% confidence interval for the parameter.
  • h) To what population would you feel comfortable generalizing the results of this analysis?

The proportion who selected the right front tire is 56/120 ≈ 0.467.  This is a statistic, because it’s based on the sample of students in my classes.  The parameter of interest is the proportion of all students at my university who would select the right front tire.  I’m assuming here that the population of interest is all students at my university, but you could also take the population to be a broader group.  Of course, the students in my class were not randomly selected from any population, so we should be cautious about generalizing the results of this analysis.

The null hypothesis is that one-fourth of all students would select the right front tire.  The alternative hypothesis is that more than one-fourth would select the right front tire.  The sample size condition is satisfied because 120×1/4 = 30 and 120×3/4 = 90 are both larger than 10.  The test statistic is:

The observed value of the sample proportion who selected the right front tire (0.467) is about 5.5 standard deviations above the hypothesized value of 0.25.  Being 5.5 standard deviations away is a huge distance that would almost never occur by random chance.  There’s no need to consult a z-table or use software to know that the p-value is extremely close to zero.  The sample data provide extremely strong evidence that the population proportion who would select the right front tire is greater than 0.25.

A 95% confidence interval for the population proportion who would say right front is:

This calculation becomes 0.467 ± 0.089, which is the interval (0.378 → 0.556).  We can be 95% confident that between 37.8% and 55.6% of all students at the university would answer right front.  Notice that this interval lies completely above the value 0.25, consistent with our having rejected the null value of 0.25. 

Part (h) is an important question, as it prompts students to pause and consider that the sample of students in my class (or your class, if you try this activity) was not randomly selected from any population, so we should not take any of these inferences too seriously.  We should even be cautious about generalizing to all students at the university.  I recommend that students say that then results can be generalized only to a population of students similar to those in the sample.

I also use hypothetical data with the “which tire?” context to lead students to investigate the impact of sample size on hypothesis tests and confidence intervals.

5. Suppose that 30% of the people in a random sample from a population select the right front tire. 

  • a) What more information do you need to conduct a hypothesis test and determine a confidence interval?
  • b) Suppose that the sample size is n = 100.  Determine the value of the test statistic, p-value, and 95% confidence interval.
  • c) Repeat for a sample size of n = 500.
  • d) Summarize the role of sample size on these hypothesis tests and confidence intervals.  

Most students realize in part (a) that we need to know the sample size.  I encourage them to express this in context: We need to know how many people answered the “which tire” question.  I also encourage students to use technology (such as the applet here) for the calculations in parts (b) and (c), so they can focus on the underlying concept.

With a sample size of 100 in part (b), the z-test statistic is 1.15 with a p-value of 0.1241.  The sample result (30% saying “right front”) does not provide much evidence to conclude that right front would be selected more than by random chance.  The 95% confidence interval for the population proportion who would select the right front tire is (0.210 → 0.390), so we can be 95% confident that between 21.0% and 39.0% of all people would select the right front tire.  Notice that this interval includes the value 0.25.

With a sample size of 500 in part (c), the z-test statistic is 2.58 with a p-value of 0.0049.  The sample result (30% saying “right front”) provides strong evidence to conclude that right front would be selected more than by random chance.  The 95% confidence interval for the population proportion is (0.260 → 0.340), so we can be 95% confident that between 26.0% and 34.0% of all people would select the right front tire.  Notice that this interval is entirely above the value 0.25.

For part (d), I hope students say that when the sample result remains proportionally the same, a larger sample size produces a larger z-test statistic and smaller p-value.  This means that a larger sample size produces stronger evidence against the null hypothesis, in favor of the alternative that people tend to select the right front tire more often than would be expected by random chance.  A larger sample size also generates a more narrow confidence interval.

You have no doubt noticed that in the previous exercises, I converted the non-binary variable (which tire was picked) into a binary variable (right front or not).  The non-binary nature of the original variable provides a good opportunity for students to practice with applying chi-square goodness-of-fit tests.

6. Consider testing the null hypothesis that students are equally likely to select any of the four tires.  Here are the responses (counts) for my 120 students in the Winter quarter of 2017:

  • a) Determine the expected counts for testing this hypothesis.
  • b) Calculate the value of the test statistic.
  • c) Determine the p-value.
  • d) Summarize your conclusion.
  • e) Identify the category (tire) with the largest contribution to test statistic, and comment on what the data reveal about this tire.
  • f) Now test a new hypothesis: Students are twice as likely to select the right front tire as any other tire, and the rest are equally likely.  Report the hypothesis, test statistic, and p-value.  Summarize your conclusion.

The expected counts, under the null hypothesis of equal likeliness, are 120×(1/4) = 30 for each tire.  The chi-square test statistic turns out to be 0.533 + 5.633 + 22.533 + 2.700 = 31.4.  The p-value, based on 3 degrees of freedom, is 0.0000007.  With such a very small p-value, we conclude that the sample data provide overwhelming evidence to reject the hypothesis that students are equally likely to select among the four tire choices.  Not surprisingly, the largest contribution to the test statistic comes from the right front tire, where the observed count (56) considerably exceeds the expected count (30).  This reveals that the popularity of the right front tire is the biggest contributor to rejecting the null hypothesis of equal likeliness.

For part (f), students must first figure out that the proportions in the null hypothesis are now 0.4 for right front and 0.2 for each of the other tires.  Students then produce the following table as they conduct the test:

Now the p-value turns out to be 0.271, so the sample data do not provide convincing evidence to reject the 20-20-40-20 hypothesis.  Some students take this a step too far by concluding that the sample data provide evidence in favor of the 20-20-40-20 hypothesis*.  I like having students use a single dataset to test one hypothesis that produces a very small p-value and another that yields a not-so-small p-value. 

* See post #29 (Not enough evidence, here) for more examples and discussion about the perils of drawing conclusions when the p-value is not small.

This “which tire” question provides a fun context in which to gather data from students.  The data collection takes very little time.  You can then ask students to ponder several questions about the data that illustrate various aspects of statistical inference.

Before I close, I want to emphasize a concern that I mentioned, but only briefly, above: Needless to say, students in your class constitute only a convenience sample of students from your school.  You could make a strong case that performing statistical inference on such data is inappropriate.  I do think it’s important to draw students’ attention to this issue and caution them not to take their findings too seriously or generalize their results very broadly.  Nevertheless, I think this is a fun context that can be memorable for students, while allowing you to ask good questions about important topics in statistical inference.

Jun 8

#49 My favorite problem, part 3

In the past two posts (here and here), I have described my favorite problem and how I present it to students at many levels.  The problem is to determine the optimal strategy for hiring an employee subject to several restrictive conditions, and also to examine what happens to the probability of successfully choosing the best candidate as the number of candidates increases.  Here’s a reminder of the outline for this three-part series:

In part 1, we experienced a revelation that I termed the Key Insight: We can optimize our probability of choosing the best by using information about the quality of candidates who appear early in the interviewing process.  We followed this up in part 2 by deriving a probability function for any number of candidates, and we used some R code to evaluate this function for several specific values.  What we learned provided a hint of a Remarkable Result, in that the probability of success remained as high as 0.368 even with 5000 candidates to choose among, as shown in the following table:

As we conclude this series in this post, we will explore how the optimal strategy, and its probability of success change as the number of candidates becomes extremely large.  To do this, we’ll use some ideas and tools from single-variable calculus.  For students who have not studied calculus, I ask them to follow along as best they can.  They can still experience the Remarkable Result even if they do not follow all of the mathematical details that confirm it.

As always, questions that I pose to students appear in italics.

8. Approximating the probability function with calculus

Here’s the probability function that we derived (recall that n represents the number of candidates, and (r – 1) is the number of candidates that you let go by before you start to consider hiring one):

For very large values of n, we have many terms in that sum, and we also need to evaluate this function for many values of r.  These calculations can take a while even with a fast computer.  It would be very helpful to find a simpler function that could approximate our exact probability function well.

What calculus tool can we use to approximate that sum?  When students need a hint, I ask: What are the primary calculus “things” that you learn about in the first term or two of your calculus series?  Most students identify derivatives and integrals as two of the most prominent things they learn about in calculus, and some realize that an integral can be thought of as the continuous version of a discrete sum.  We can use the following approximation:

Evaluate this integral.  Some students remember that this integral leads to a natural log function, as follows:

Use this result to approximate the probability function.  Substituting this log function for the sum gives:

How well does this function approximate the exact probability function?  I used R to evaluate both functions, for all values of r, with a few different values of n.  The graphs below display the exact probabilities with a solid dot, the approximate probabilities with an open diamond:

The graphs in the top row show that the approximation does poorly when n = 12, better but not great when n = 50.  The graphs in the bottom row reveal that the approximation performs very well when n = 500 and n = 5000.  We can feel quite comfortable using the approximation for our purpose, which involves much larger values than n = 5000.

9. Confirming the Remarkable Result

Remember our goal: For a given number of candidates (n), we want to determine the value of r that maximizes the probability of successfully choosing the best.  Now we have an approximate function for this probability, which we need to maximize.  What calculus tool can we use to determine the value of r that maximizes this function?  At this point, most of my students know that we can use the derivative for this purpose.  Consider again this function:

Take the derivative of this function with respect to r.  When students give me a blank stare, I ask: What derivative rule do we need to use?  If they still do not respond, I follow up with: Remind me what the × symbol means.  Some students roll their eyes and say times.  I ask for a slightly bigger word and wait for someone to say product, and then someone calls out that we need the product rule.  When students get to the part with the natural log, I again ask what rule we need and wait until someone says the chain rule.  Finally, we need the quotient rule to work with the argument of the natural log function*.

* I think it’s a lot of fun that we get to use the product, quotient, and chain rules to take this derivative.  Not all of my students agree.

I give students a few minutes to evaluate this derivative and check their answers with each other.  After applying the product, quotient, and chain rules, the derivative simplifies to:

What do we do next with this derivative?  Some students need a reminder: Why did we bother to evaluate this derivative in the first place?  We’re trying to determine the value of r that maximizes this function.  What do we do with the function’s derivative to determine where the function is maximized?  Set the derivative equal to zero.  And then …?  Solve.  Solve for what?  Solve for the value of r.  Then I give students a few minutes to do this.

Setting this derivative equal to zero produces:

Solving for r gives:

Based on this result, describe the optimal strategy with a very large number of candidates.  Remember that (r – 1) is the number of candidates that we let pass before we consider hiring one.  This result says that the optimal strategy is to let n/e of the candidates go by.  Because 1/e ≈ 0.3679, this means that we should let 36.79% go by, and then hire the first candidate you see who is the best so far.

Describe how to determine the probability of successfully choosing the best with this optimal strategy.  We can approximate this probability very well by plugging the optimal value of r into the function f(r).  In other words, we need to evaluate f(n/e + 1).

Evaluate this probability.  This function gives:

Interpret this probability.  By using the optimal strategy, even with an extremely large number of candidates, you have a 36.79% chance of successfully choosing the best.  In other words, if you use this strategy over and over again, you will successfully choose the best candidate in about 36.79% of all job searches.

Is this finding remarkable?  Yes!!*  I urge students not to allow all of this mathematics and calculus to divert their attention from what we’ve just discovered: The probability of successfully choosing the best candidate does not approach zero as the number of candidates gets very large.  In fact, this probability does not even come close to zero.  Instead, it approaches, and never dips below, 1/e, which is about 0.3679.  This is very close to the optimal probability with 5000 candidates that we discovered at the end of the previous post.  So, the probability that you successfully choose the best candidate is essentially the same whether you have 5 thousand candidates or 7.8 billion candidates.  Yes, most emphatically, that’s remarkable!

* Earning full credit for this answer requires using at least two exclamation points!

This finding is all the more amazing when you consider that there’s about a 37% chance that the best candidate will be in the initial group that you do not even consider hiring.  The conditional probability that you successfully choose the best candidate, given that the best candidate does not appear in that initial group, is therefore .3679 / .6321 ≈ 0.582.  In other words, given that this strategy allows you any chance to find the best candidate, there’s a 58.2% chance that you will succeed.

10. Extensions, including how to find your soulmate in life

We’ve now completed the solution to my favorite problem, but I’m enjoying this too much to stop just yet.  Now I’ll present two fun applications to situations other than hiring an employee.  With the first one, you can amaze your friends.  The second one just might help you to find your soulmate in life.

Consider this game: Give 50 blank index cards to some friends, and ask them to write one number, unseen by you, on each card.  The numbers can be as small or as large as they like.  When they have the 50 cards with 50 numbers, they need to remember what the largest number is and then shuffle the cards thoroughly.  The game is that your friends will reveal the numbers on the cards to you one at a time, and your task is to tell them immediately when you think you’ve seen the largest number in the whole stack.  Before you start, make sure that your friends appreciate how remarkable it will be if you can succeed at this!  Remind them that you have no idea how large or small the numbers are, and you’re trying to identify the largest one at the very moment that you first see it.

What strategy should you use?  Well, this game is essentially the same as the “choosing the best” problem, right?  We found in the previous post (here) that the optimal value is r = 19 when n = 50.  So, you should let the first 18 numbers go by.  All you have to keep track of is the largest number you see among those first 18.  Then as soon as you see a larger number than that, immediately declare in dramatic fashion: That’s the largest number in the whole stack! 

What’s the probability that you’ll be right?  According to the R code we ran for the previous post, this probability is about 0.3743.  Granted, if you play this game repeatedly, you’ll be right less than half the time in the long run, but you’ll be right more than one-third of the time.  Considering how challenging the task sounds, that success rate should be often enough to amaze your friends. 

* If you’re so inclined, you could ask for 2-to-1 odds on a small wager, and you’ll come out with a positive profit if you play enough times.  You might even persuade your friends to offer higher odds than that.

Now consider a much more important challenge in life: finding your soulmate.  Everyone wants to find the very best person in the whole world for them, right?  Nobody sets out to find someone among the top 40% of all possible life partners; you want the very best.  Think about how this process plays out: You meet people one at a time, and you have to decide somewhat quickly about whether you’d like to spend substantial time with the person.  The optimal strategy that we discovered says that you should let the first 37% of potential soulmates go by, and then propose to the first one you find who is the best so far. 

One complication is that you don’t know in advance exactly how many potential soulmates you’ll meet.  But you might consider ages 18-36 to be your years for conducting this search, and 18/e ≈ 6.62 years.  So, you should let potential soulmates pass between the ages of 18 and 24.62 years.  After that age, once you find one who is the best so far, try to convince them that they should consider you as their soulmate.  Of course, this points out a second and more important complication: Even if you succeed in finding the very best person for you, they may or may not reciprocate your assessment.

Thanks very much for joining me on this very long* journey through my favorite problem.  I admitted at the outset that this problem is not particularly realistic, but I find it a lot of fun to explore.  I particularly enjoy that we employed many aspects of probabilistic and mathematical thinking.  We used “brute force” enumeration and counting to analyze small cases, which led to the Key Insight that propelled us through the rest of the analysis.  Then we used more counting principles to figure out the general case, and we wrote some code that enabled us to tackle a large number of cases.  The resulting graphs pointed to a Remarkable Result, which we confirmed by applying some calculus.

* This series has exceeded 9000 words.

P.S. I mentioned in section 1 of this series that I first heard about this problem from Morrie DeGroot.  You can read more about this problem in his textbooks Optimal Statistical Decisions and Probability and Statistics.  I also recommend the article “Who Solved the Secretary Problem?” by Thomas Ferguson (here).

Jun 1

3 Comments

#48 My favorite problem, part 2

Now we continue with the analysis of my favorite problem, which I call “choosing the best,” also known as the “secretary problem.”  This problem entails hiring a job candidate according to a strict set of rules.  The most difficult rules are that you can only assess candidates’ quality after you have interviewed them, you must decide on the spot whether or not to hire someone and can never reconsider someone that you interviewed previously, and you must hire the very best candidate or else you have failed in your task.

Here’s a reminder of the outline for this three-part series:

In the previous post (here), we analyzed some small cases by hand and achieved the Key Insight that led to the general form of the optimal solution: Let a certain number of candidates go by, and then hire the first candidate you see who is the best so far.  The question now is how many candidates to let pass before you begin to consider hiring one.  We’ll tackle the general case of that question in this post, and we’ll consider cases as large as 5000 candidates. 

I tell students that the derivation of the probability function in section 4 is the most mathematically challenging section of this presentation.  But even if they struggle to follow that section, they should be able to understand the analysis in sections 6 and 7.  This will provide a strong hint of the Remarkable Result that we’ll confirm in the next post.

Before we jump back in, let me ask you to make predictions for the probability of successfully choosing the best candidate, using the optimal strategy, for the numbers of candidates listed in the table (recall that the last number is the approximate population of the world):

As always, questions that I pose to students appear in italics.

4. Deriving the probability function

We need to figure out, for a given number of candidates, how many candidates you should let pass before you actually consider hiring one.  This is where the math will get a bit messy.  Let’s introduce some symbols to help keep things straight:

  • Let n represent the number of candidates.
  • Let i denote the position in line of the best candidate.
  • Let r be the position of first “contender” that we actually consider hiring.
    • The strategy is to let the first (r – 1) candidates go by, before you genuinely consider hiring one.

We will express the probability of successfully choosing the best candidate as a function of both n and r.  After we have done that, then for any value of n, we can evaluate this probability for all possible values of r to determine the value that maximizes the probability.

First we will determine conditional probabilities for three different cases.  To see why breaking this problem into cases is helpful, let’s reconsider the n = 4 situation that we analyzed in the previous post (here).  We determined that the “let 1 go by” strategy is optimal, leading to success with 11 of the 24 possible orderings.  What value of r does this optimal strategy correspond to?  Letting 1 go by means that r = 2 maximizes the probability of success when n = 4.

These 24 orderings are shown below.  The ones that lead to successfully choosing the best with the “let 1 go by” strategy are displayed in green:

Looking more closely at our analysis of the 24 orderings with the “let 1 go by” (r = 2) strategy, we can identify different cases for how the position of the best candidate (i) compares to the value of r.  I’ve tried to use cute names (in italics below) to help with explaining what happens in each case:

  • Case 1, Too soon (i < r): The best candidate appears first in line.  Because our strategy is to let the first candidate go by, we do not succeed for these orderings.  Which orderings are these?  A, B, C, D, E, and F.
  • Case 2, Just right (i = r): The best candidate appears in the first position that we genuinely consider hiring, namely second in line.  We always succeed in choosing the best candidate in this circumstance.  Which orderings are these?  G, H, M, N, S, and T.
  • Case 3a, Got fooled (i > r): The best candidate appears after the first spot at which we consider hiring someone.  But before we get to the best candidate, we get fooled into hiring someone else who is the best we’ve seen so far.  Which orderings are these?  O, P, R, U, V, W, and X.
  • Case 3b, Patience pays off (also i > r): Again the best candidate appears after the first spot at which we consider hiring someone.  But now we do not get fooled by anyone else and so we succeed in choosing the best.  Which orderings are these?  I, J, K, L, and Q.

As we move now from the specific n = 4 case to the general case for any given value of n, we will consider the analogous three cases for how the position of the best candidate (i) compares to the value of r:

  • Case 1: i < r, so the best candidate is among the first (r – 1) in line.  What is the probability that you successfully choose the best candidate in this case?  Remember that the strategy is to let the first (r – 1) go by, so the probability of success equals zero.  In other words, this is the unlucky situation in which the best candidate arrives while you are still screening candidates solely to gain information about quality.
  • Case 2: i = r.  What is the probability that you successfully choose the best candidate in this case? When the best candidate is in position r, then that candidate will certainly be better than the previous ones you have seen, so the probability of success equals one.  This is the ideal situation, because the best candidate is the first one that you actually consider hiring.
  • Case 3: i > r, so the best candidate arrives after you have started to consider hiring candidates*.  What is the probability that you successfully choose the best candidate in this case?  This is the most complicated of the three situations by far.  The outcome is not certain.  You might succeed in choosing the best, but you also might not.  Remember from our brute force analyses by enumeration that the problem is that you might get fooled into hiring someone who is the best you’ve seen but not the overall best.  What has to happen in order for you to get fooled like this?  In this situation, then you will succeed in choosing the best unless the best of the first (i – 1) candidates occurs after position (r – 1).  In this situation, you will be fooled into hiring that candidate rather than the overall best candidate.  In other words, you will succeed when the best among the first (i – 1) candidates occurs among the first (r – 1) that you let go by.  Because we’re assuming that all possible orderings are equally likely, the probability of success in this situation is therefore (r – 1) / (i – 1).

* This is the hardest piece for students to follow in the entire three-part post.  I always encourage them to take a deep breath here.  I also reassure them that they can follow along again after this piece, even if they do not understand this part completely.

The following diagrams may help student to think through these three cases.  The * symbol reveals the position of the best candidate.  The red region indicates candidates among the first (r – 1) who are not considered for hiring.  The blue region for case 3 contains candidates who could be hired even though they are not the very best candidate.

How do we combine these conditional probabilities to determine the overall probability of success?  When students need a hint, I remind them that candidates arrive in random order, so the best candidate is equally likely to be in any of the n positions.  This means that we simply need to take the average* of these conditional probabilities:

* This is equivalent to using the law of total probability.

This expression simplifies to:

The above works for values of r ≥ 2.  The r = 1 situation means hiring the first candidate in line, so the probability of success is 1/n when r = 1.  Using S to denote the event that you successfully choose the best candidate, the probability function can therefore be written in general as:

Our task is now clear: For a given value of n, we evaluate this function for all possible values of r (from 1 to n).  Then we determine the value of r that maximizes this probability.  Simple, right?  The only problem is that those sums are going to be very tedious to calculate.  How can we calculate those sums, and determine the optimal value, efficiently?  Students realize that computers are very good (and fast) at calculating things over and over and keeping track of the results.  We just need to tell the computer what to do.

5. Coding the probability function

If your students have programming experience, you could ask them to write the code for this task themselves.  I often give students my code after I first ask some questions to get them thinking about what the code needs to do: How many loops do we need?  Do we need for loops or while loops?  What vectors do we need, and how long are the vectors? 

We need two for loops, an outer one that will work through values of r, and an inner one that will calculate the sum term in the probability function.  We also need a vector in which to store the success probabilities for the various values of r; that vector will have length n.

I also emphasize that this is a different use of computing than we use in much of the course.  Throughout my class, we make use of computers to perform simulations.  In other words, we use computers to generate random data according to a particular model or process.  But that’s not what we’re doing here.  Now we are simply using the computer to speed up a very long calculation, and then produce a graph of the results, and finally pick out the maximum value in a list.

Here is some R code* that accomplishes this task:

* A link to a file containing this code appears at the end of this post.

We figured out the n = 4 case by analyzing all 24 orderings in the last post (here), so let’s first test this code for that situation.  Here’s the resulting output:

Explain how this graph is consistent with what we learned previously.  We compared the “let 1 go by” and “let 2 go by” strategies.  We determined the probabilities of successfully choosing the best to be 11/24 ≈ 0.4583 and 10/24 ≈ 0.4167, respectively.  The “let 1 go by” strategy corresponds to r = 2, and “let 2 go by” means r = 3.  Sure enough, the probabilities shown in the graph for r = 2 and r = 3 look to be consistent with these probabilities.  Why does it make sense that r = 1 and r = 4 give success probabilities of 0.25?  Setting r = 1 means always hiring the first candidate in line.  That person will be the best with probability 1/4.  Similarly, r = 4 means always hiring the last of the four candidates in line, so this also has a success probability of 1/4.

Let’s do one more test, this time with n = 5 candidates, which I mentioned near the end of the previous post.  Here’s the output:

Based on this output, describe the optimal strategy with 5 candidates.  Is this consistent with what I mentioned previously?  Now the value that maximizes the success probability is r = 3.  The optimal strategy is to let the first two candidates go by*; then starting with the third candidate, hire the first one you encounter who is the best so far.  This output for the n = 5 case is consistent with what I mentioned near the end of the previous post.  The success probabilities are 24/120, 50/120, 52/120, 42/120, and 24/120 for r = 1, 2, 3, 4, and 5, respectively.

* Even though I keep using the phrase “go by,” this means that you assess their quality when you interview those candidates, because later you have to decide whether a candidate is the best you’ve seen so far.

6. Practice with a particular case

Now consider the case of n = 12 candidates, which produces this output:

Describe the optimal strategy.  The value r = 5 maximizes the success probability when n = 12.  The optimal strategy is therefore to let the first 4 candidates go by and then hire the first one you find who is the best so far.  What percentage of the time will this strategy succeed in choosing the best?  The success probability with this strategy is 0.3955, so this strategy will succeed 39.55% of the time in the long run.  How does this probability compare to the n = 5 case?  This probability (of successfully choosing the best) continues to get smaller as the number of candidates increases.  But the probability has dropped by less than 4 percentage points (from 43.33% to 39.55%) as the number of candidates increased from 5 to 12.

To make sure that students understand how the optimal strategy works, I ask them to apply the strategy to the following 25 randomly generated orderings (from the population of 12! = 479,001,600 different orderings with 12 candidates).  This exercise can also be helpful for understanding the three cases that we analyzed in deriving the probability function above.  For each ordering, determine whether or not the optimal strategy succeeds in choosing the best candidate. 

I typically give students 5-10 minutes or so to work on this, and I encourage them to work in groups.  Sometimes we work through several orderings together to make sure that they get off to a good start.  With ordering A, the best candidate appears in position 3, so our let-4-go-by strategy means that we’ll miss out on choosing the best.  The same is true for ordering B, for which the best candidate is in position 2.  With ordering C, we’re fooled into hiring the second-best candidate sitting in position 6, and we never get to the best candidate, who is back in position 10.  Orderings D and E are both ideal, because the best candidate is sitting in the prime spot of position 5, the very first candidate that we actually consider hiring.  Ordering F is another unlucky one in which the best candidate appears early, while we are still letting all candidates go by. 

I like to go slowly through ordering G with students.  Is ordering G a winner or a loser?  It’s a winner!  Why is it a winner when the best candidate is the very last one in line?  Because we got lucky with the second-best candidate showing up among the first four, which means that nobody other than the very best would be tempting enough to hire.

The following table shows the results of this exercise.  The numbers in bold color indicate which candidate would be hired.  Green letters and numbers reveal which orderings lead to successfully choosing the best.  Red letters and numbers indicate orderings that are not successful.

In what proportion of the 25 random orderings does the optimal strategy succeed in choosing the best candidate?  Is this close to the long-run probability for this strategy?  The optimal strategy resulted in success for 10 of these 25 orderings.  This proportion of 0.40 is very close to the long-run probability of 0.3955 for the n = 12 case from the R output.

7. Analyzing graphs, with a hint of the Remarkable Result

Now let’s run the code to analyze the probability function, and determine the optimal strategy, for larger numbers of candidates.  Here’s the output for n = 50 candidates:

Describe the optimal strategy.  What is its probability of success?  How has this changed from having only 12 candidates?  The output reveals that the optimal value is r = 19, so the optimal strategy is to let the first 18 candidates go by and then hire the first one who is the best so far.  The probability of success is 0.3743, which is only about two percentage points smaller than when there were only 12 candidates.  How were your initial guesses for this probability?  Most students find that their initial guesses were considerably lower than the actual probability of success with the optimal strategy.

The graph on the left below shows how the optimal value of r changes as the number of candidates ranges from 1 to 50, and the graph on the right reveals how the optimal probability of success changes:

Describe what each graph reveals.  The optimal value of r increases roughly linearly with the number of candidates n.  This optimal value always stays the same for two or three values of n before increasing by one.  As we expected, the probability of success with the optimal strategy decreases as the number of candidates increases.  But this decrease is very gradual, much slower than most people expect.  Increasing the number of candidates from 12 to 50 only decreases the probability of success from 0.3955 to 0.3743, barely more than two percentage points.

Now consider output for the n = 500 (on the left) and n = 5000 (on the right) cases*:

* The n = 5000 case takes only one second on my laptop.

What do these functions have in common?  All of these functions have a similar shape, concave-down and slightly asymmetric with a longer tail to the high end.   How do the optimal values of r compare?  The optimal values of r are 185 when n = 500 and 1840 when n = 5000.  By increasing the number of candidates tenfold, the optimal value of r increases almost tenfold.  In both cases, the optimal strategy is to let approximately 37% of the candidates go by, and then hire the first you see who is the best so far.  How quickly is the optimal probability of success decreasing?  This probability is decreasing very, very slowly.  Increasing the number of candidates from 50 to 500 to 5000 only results in the success probability (to three decimal places) falling from 0.374 to 0.369 to 0.368.

The following graphs display the optimal value of r, and the probability of success with the optimal strategy, as functions of the number of candidates:

Describe what these graphs reveal.  As we noticed when we examined the graph up to 50 candidates, the optimal value of r continues to increase roughly linearly.  The probability of success with the optimal strategy continues to decrease at a very, very slow rate.

Some students focus so intently on answering my questions that they miss the forest for the trees, so I ask: Do you see anything remarkable here?  Yes!  What’s so remarkable?  The decrease in probability is so gradual that it’s hard to see with the naked eye in this graph.  Moreover, if 5000 candidates apply for your job, and your hiring process has to decide on the spot about each candidate that you interview, with no opportunity to ever go back and consider someone that you previously passed on, you can still achieve a 36.8% chance of choosing the very best candidate in the entire 5000-person applicant pool.

How were your guesses, both at the start of the previous post and the start of this one?

Let’s revisit the following table again, this time with probabilities filled in through 5000 candidates.  There’s only one guess left to make.  Even though the probability of choosing the best has decreased only slightly as we increase the number of candidates from 50 to 500 to 5000 candidates, there’s still a very long way to go from 5000 to almost 7.8 billion candidates!  Make your guess for the probability of successfully choosing the best if every person in the world applies for this job.

In the next post we will determine what happens as the number of candidates approaches infinity.  This problem provides a wonderful opportunity to apply some ideas and tools from single-variable calculus.  We will also discuss some other applications, including how you can amaze your friends and, much more importantly, find your soulmate in life!

P.S.  Here is a link to a file with the R code for evaluating the probability function:

choosingthebestDownload