Skip to content

#19 Lincoln and Mandela, part 1

Two great leaders will be featured in this post and the next: Abraham Lincoln and Nelson Mandela.  Well, to be honest, featured is too strong, but these men provide the background for in-class activities that help students to understand two very important concepts in statistics: random sampling and random assignment.

When I first mention these two terms in class, I suspect that many students only hear random and don’t pay much attention to sampling versus assignment.  I admit that I did not make a big deal of this distinction myself when I started teaching.  But now I try to emphasize that random sampling and random assignment are very different ideas with very different goals.  In a nutshell:

  • Random sampling concerns how to select observational units for a sample.  Random sampling allows for generalizing the results of a sample to the larger population.
  • Random assignment pertains to how observational units come to be in groups to be compared.  Random assignment allows for the possibility of drawing a cause-and-effect conclusion.

This post will discuss random sampling with reference to Lincoln, and the next will concern random assignment while mentioning Mandela.  Along the way we’ll sneak in a touch of history and also some psychology.  As always, questions for students appear in italics.


I begin this activity by asking students to consider the 268 words in this speech as the population of interest:

The natural first question is: What speech is this, and who wrote it?  I’m glad that most students recognize this as Lincoln’s Gettysburg Address.  Then I give these instructions:

  • Circle ten words as a representative sample from this population.
  • For each word in your sample, record how many letters are in the word.
  • Calculate the average (mean) number of letters per word in your sample.
  • Plot your sample average on a dotplot on the board, along with the sample averages of your classmates.

Those who remember post #11 (here) will not be surprised that I next ask students: Identify the observational units and variable, first in your sample and then for the graph on the board.  For the students’ samples of ten words, the observational units are words, and the variable is the length of the word, as measured by number of letters.  But for the dotplot that students produce on the board, the observational unit are samples of 10 words, and the variable is the average length of a word.

All of this is prelude to the important question: How can we use the dotplot on the board to tell whether this sampling method (my telling students to circle ten words) is any good?  Before a student will respond, I often have to add: What additional information would you like to know to help you decide whether this sampling method was good?  At this point a student usually responds that they would like to know the average word length in the entire population of 268 words.  I reply: Great idea, and before class I calculated this population average to be 4.295 letters per word.  Then I draw a vertical line through the dotplot at this value.  Here are results from a recent class:

At this point I define sampling bias as a systematic tendency for a sampling method to over-represent some observational units and under-represent others.  Then I ask: Would you say that this sampling method (my asking students to circle ten words) is biased?  If so, in which direction?  How can you tell from the dotplot?

Students recognize that a large majority of the sample averages are greater than the population average.  This means that there’s a systematic tendency for this sampling method to over-represent large words and under-represent small words.  In other words, this sampling method is biased toward over-estimating the average length of a word in the Gettysburg Address.

I emphasize to students that sampling bias is a property of the sampling method, not of any one sample generated by the method.  One illustration of this is to ask: Whose idea was it to select a sample by circling ten words based solely on human judgment?  Students reply, somewhat sheepishly, that it was my idea.  I respond that this is absolutely right: The sampling bias here is my fault, not theirs, because the sampling method was my idea.

Then I ask: Suggest some reasons for why this sampling method turned out to be biased in this way.  Students are quick to suggest good explanations for this sampling bias.  They mention that longer words (such as government, battlefield, and consecrate) convey the meaning of the speech better than smaller words (such as a, by, and for).  Students also suggest that longer words are more likely to be selected because they are just more interesting than smaller words.

Next I ask whether sample size is the problem: Would asking people to circle twenty words (rather than ten) eliminate, or at least reduce, the sampling bias?  Most students realize that taking a larger sample of words would not help with this problem, because people would still be prone to select larger words rather than smaller ones.

Before we conclude this discussion of biased sampling, I ask students to give me a chance to redeem myself by proposing a new sampling method: Suppose that I ask you to close your eyes and point at the page ten times in order to select words for your sample.  Would this sampling method be unbiased?  (After all, doesn’t closing your eyes guarantee a lack of bias?)  Explain.  Most students correctly realize that this sampling method is still biased toward longer words.  You would be more likely to select longer words than shorter ones, because longer words take up more space on the page.

Finally, I ask: Suggest a different sampling method that would be unbiased.  Some students immediately respond with a magic word: random!  So I follow up with: What does it mean to select a random sample of words in this situation?  This question is harder, but eventually a student says that random sampling gives every word, whether it is an interesting word such as dedicate or a boring word like of, the same chance of being selected.


We then proceed to examine properties of random sampling.  Sometimes I ask students to generate their own random samples of words from this population.  One option for doing this is to give them a numbered list of the 268 words and then use a random number generator (such as the one at random.org) to select their sample.  They can then calculate their sample mean word length and put a dot on a new dotplot on the board, using the same scale as the original dotplot.

Another option is to move directly to using an applet (available here) to select random samples of words.  This applet starts by showing the distribution of word lengths in the population, which is skewed to the right:

You can select random samples by first clicking on Show Sampling Options.  I ask students to start by selecting one random sample of 5 words, which produces a result such as:

The applet calculates the sample mean word length for this sample and plots that on a graph.  Then asking the applet to select 999 more samples results in a graph of sample means that looks like:

Now we’re ready for the key questions: Does this distribution of sample means indicate sampling bias or unbiasedness of this random sampling method?  What aspect of the distribution leads you to this conclusion?  The shape and variability in this distribution are completely irrelevant to the issue of sampling bias.  To address this issue, we focus on the center of the distribution.  We see that the center of the distribution of sample means is very close to the population mean.  We can quantify this by noting that the mean of the 1000 sample means is 4.336 letters/word, which is quite close to the population mean of 4.295 letters/word. Therefore, this random sampling method appears to be unbiased.

Before moving on, I want to point out how challenging the following statement can be for students:

The mean of the sample means is the population mean.

This sentence contains only ten words, but three of them are the word mean(s)!  We can rewrite this statement mathematically, using common notation, as:

Notice that this equation contains only three symbols (in addition to the equals sign), but all three of them describe a mean!  It takes considerable time and careful thought for students to recognize and understand what these three means are and how they relate to each other:

  • The population mean.  For the population of 268 words in the Gettysburg Address, the value of the population mean is 4.295 letters/word.
  • The sample mean, which varies from sample to sample.  Each student calculated his/her own sample mean and represented it with a dot on the board. The first random sample generated by the applet above had a sample mean of 3.6 letters/word.  The applet then generated 999 more random samples and calculated the sample mean number of letters/word for each one.
  • The mean of the sample means.  We could have calculated this for the students’ sample means in class; we did not bother, but we know from the graph that the mean of the sample means would have been much greater than 4.295.  The applet did calculate the mean of the 1000 sample means that it generated; the mean of these sample means turned out to be 4.336 letters/word.  If we went on to generate all possible random samples, in the long run the mean of the sample means would be 4.295, the same value as the population mean.

My next question for students: Consider taking random samples of size 20 words per sample, rather than 5 words per sample.  How (if at all) would you expect the distribution of sample means to change, in terms of center, variability, and shape?  After students think about this, discuss it among themselves, and record their predictions, we use the applet to make this change, which produces a result such as:

We see that the center of this distribution is still close to the population mean of 4.295 letters/word.  Most students expect this, because this simply shows that random sampling is still unbiased with a larger sample size.  The key finding is that the variability of sample means is smaller with a larger sample size.  How can we tell?  One way is that the sample means now range from about 3 to 6 letters/word, whereas before (with a smaller sample size of 5) they ranged from about 2 to 8 letters/word.  Even better, we can note that the standard deviation of the sample means is now about 0.463, which is much less than its value of 0.945 with the smaller sample size.  The shape of the distribution of sample means is a bit more symmetric and normal-looking with the larger sample size than with the smaller sample size, much less skewed than the distribution of the population.

This last point foreshadows the concept of a sampling distribution of a sample mean and the Central Limit Theorem.  I think this context and applet provide a great opportunity to study those ideas*, but at this point I prefer to keep the focus on the topics of sampling bias and random sampling.

* One feature that I particularly like about this applet is that it displays three distributions at once, which are crucial (and challenging) for students to keep in mind when studying sampling distributions:

  • Population distribution (of word lengths)
  • Sample distribution (of word lengths)
  • Sampling** distribution (of average word lengths in a sample)

** It’s very unfortunate that the modifier words sample and sampling are so similar, yet the distributions they describe are precisely a key distinction to understand.  Perhaps we should avoid using the term sampling distribution and instead say distribution of sample averages.  It’s nice to be able to use shorthand when speaking with colleagues who understand the ideas, but in this case the extra words provide clarity for students who are just beginning to consider the ideas.


Before leaving the topic of sampling bias and random sampling, I ask a few more questions of my students, all in the context of selecting a sample of students at our university to complete a survey:

  • Would it be easy or hard to select a random sample of 50 Cal Poly students?

It takes a while for some students to realize that selecting such a random sample would be very hard to achieve.  It’s unlikely that university administrators would provide a list of all students at the university.  Having access to such a list would enable us to select a random sample of students’ names, but we would still face the challenges of contacting them successfully and then, even more problematic, convincing them to respond to our survey.

  • Suppose that you select a sample of Cal Poly students by standing in front of the library or recreation center and approaching 50 students who pass by.  Would this constitute a random sample of Cal Poly students?  What if you stand in front of the recreation center and approach 50 students who pass by?

Most students realize that this sampling method (standing in one location and recruiting passersby) does not constitute random sampling.  Some students would be more likely to be selected than others, in part because they are out-and-about on campus more often.  It’s also likely that you would be more likely to approach students who appear to be …, well, …, approachable, as opposed to students who look more intimidating or less friendly.  Even though the word random is used in an everyday sense to mean anything that is unplanned or unstructured, random sampling has a technical meaning.

  • Even though the convenience sampling described above is not random, could it nevertheless result in a sample that is representative of the population of Cal Poly students?  Identify a variable for which you would not be willing to consider such a convenience sample (as described above) to be representative of the population of Cal Poly students.  Also identify a variable for which you would be willing to consider such a sample (as described above) to be representative of the population of Cal Poly students.

We should certainly not consider a convenience sample, selected from students who pass by the library or recreation center, to be representative of the population for most variables, such as how often a student uses the recreation center per week, and whether or not a student knows where the library is on campus.  We should also be wary for variables about the student’s major, or how many hours they study per week, or how much sleep they get per night.  But there’s probably no reason to doubt that such a sample is representative of the population for a variable such as blood type.


I have used far more than 268 words to write this post.  Clearly I am much less economical with words than Abraham Lincoln in his Gettysburg Address.  I look forward to name-dropping Nelson Mandela into the next post, which will feature random assignment and discuss how that is quite different from random sampling.

P.S. Beth Chance and I developed the Gettysburg Address activity based the famous “random rectangles” activity developed by Dick Scheaffer and others.  As I told Dick when I interviewed him for the Journal of Statistics Education (here), I suspect that random rectangles is the most widely used activity for teaching statistics of all time, at least among activities that do not involve M&M candies.  You can read more about the genesis of the random rectangles activity in this JSE article (here).

P.P.S. This website (here) provides six different versions of the Gettysburg Address, with minor variations (and slightly different numbers of words) among them.  The one used above is the Hay copy.

#18 What do you expect?

I argued in post #6 (here) that the most dreaded two-word term in statistics is standard deviation.  In this post I discuss the most misleading two-word term in statistics.  There’s no doubt in my mind about which term holds this distinction.  What do you expect me to say?

If you expect me to say expected value, then your expectation is correct.

Below are four examples for helping students to understand the concept of expected value and avoid being misled by its regrettable name.  You’ll notice that I do not even use that misleading name until the end of the second example.  As always, questions that I pose to students appear in italics.


1. Let’s return to the random babies activity from post #17 (here).  I used the applet (here) to generate one million repetitions of distributing four babies to their mothers at random, with the following results:

I ask students: Calculate the average number of matches per repetition.  I usually get some blank stares, so I ask: Remind me how to calculate an average.  A student says to add up the values and then divide by the number of values.  I respond: Yes, that’s all there is to it, so please do that with these one million values.  At this point the blank stares resume, along with mutterings that they can’t possibly be expected* to add a million values on their own.

* There’s that word again.

But of course adding these one million values is not so hard at all: Adding the 375,124 zeroes takes no time, and then adding the 332,938 ones takes barely a moment.  Then you can make use of a wonderful process known as multiplication to calculate the entire sum: 0×(375,124) + 1×(332,938) + 2×(250,014) + 4×(41,924) = 1,000,662.  Dividing by 1,000,000 just involves moving the decimal point six places to the left.  This gives 1.000662 as the average number of matches in the one million simulated repetitions of this random process of distributing four babies to their mothers at random.

Then I ask: What do you think the long-run average (number of matches per repetition) will be if we continue to repeat this random process forever and ever?   Most students predict that the long-run average will be 1.0, and I tell them that this is exactly right.  I also show the applet’s graph of the average number of matches as a function of number of repetitions (for the first 1000 repetitions), which shows considerable variation at first but then gradual convergence toward a long-run value:


At this point we discuss how to calculate the theoretical long-run average based on exact probabilities rather than simulation results.  To derive the formula, let’s rewrite the calculation of the average number of matches in one million repetitions from above:

Notice that this calculation is a weighted average, where each possible value (0, 1, 2, 4) is weighted by the proportion of repetitions that produced the value.  Now recall the exact probabilities that we calculated in post #17 (here) for this random process:

and then replace the proportions in the weighted average calculation with the exact, theoretical probabilities:

This expression works out to be 24/24, which is better known as the value 1.0.  This is the theoretical long-run average number of matches that would result from repeating this random process forever and ever.  In general, a theoretical long-run average is the weighted average of the possible values of the random process, using probabilities as weights.  We can express this in a formula as follows, where LRA represents long-run average, x represents the possible values, and p(x) represents their probabilities:

Back to the random babies context, next I ask:

  • Is this long-run average the most likely value to occur?  Students recognize that the answer is no, because we are slightly more likely to obtain 0 matches than 1 match (because probability 9/24 is greater than 8/24).
  • How likely is the long-run average value to occur?  We would obtain exactly 1 match one-third (about 33.33%) of the time, if we were to repeat the random process over and over.
  • Do you expect the long-run average value to occur if you conduct the random babies process once?  Not really, because it’s twice as likely that we will not obtain 1 match than it is that we will obtain 1 match.

2. Now a very generic example: Consider rolling a fair, ordinary, six-sided die (or number cube), and then observing the number of dots on the side that lands up.  Calculate and interpret the long-run average value from this random process.

Saying that the die is fair means that the six possible outcomes should be equally likely, so the possible values and their probabilities are:

We can calculate the long-run average to be: LRA = 1×(1/6) + 2×(1/6) + 3×(1/6) + 4×(1/6) + 5×(1/6) + 6×(1/6) = 21/6 = 3.5.  This means that if we were to roll the die for a very large number of rolls, the average number of dots appearing on the side that lands up would be very close to 3.5.

Now I ask the same three questions from the end of the previous example:

  • Is this long-run average the most likely value to occur in the die-rolling process?  Of course not, because it’s downright impossible to obtain 3.5 dots when rolling a die. 
  • How likely is the long-run average value to occur?  Duh, like I just said, it’s impossible!  The probability is zero.
  • Do you expect the long-run average value to occur if you roll a die once?  Once more, with feeling: Of course not!

Students naturally wonder why I asked these seemingly pointless questions for the die-rolling example.  Here’s where things get a bit dicey (pun intended).  I sheepishly reveal to students that the common term for this quantity that we have been calculating and interpreting is expected value, abbreviated as EV or E(X).

Let’s ask those questions again about the die-rolling process, but now using standard terminology:

  • Is the expected value the most likely value to occur in the die-rolling process? 
  • How likely is the expected value to occur? 
  • Do you expect the expected value to occur if you conduct the die rolling process once? 

The answers to these questions are the same as before: No, of course not, the expected value (3.5 dots) is certainly not expected, because it’s impossible!

Isn’t this ridiculous?  Can we blame students for getting confused between the expected value and what we expect to happen?  As long as we’re stuck with this horribly misleading term, it’s incumbent on us to help students understand that the expected value of a random process does not in any way, shape, or form mean the value that we expect to occur when we conduct the random process.  How can we do this?  You already know my answer: Ask good questions!


3. Now let’s consider the gambling game of roulette.  When an American roulette wheel (as shown below) is spun, a ball eventually comes to rest in one of its 38 numbered slots.  The slots have colors: 18 red, 18 black, and 2 green.

The simplest version of the game is that you can bet on either a number or a color:

  • If you bet $1 on a color (red or black) and the ball lands in a slot of that color, then you get $2 back for a net profit of $1.  Otherwise, your net profit is -$1.
  • If you bet $1 on a number and the ball lands in that number’s slot, then you get $36 back for a net profit of $35.  Otherwise, your net profit is -$1.

I ask students to work through the following questions in groups, and then we discuss the answers:

  • a) List the possible values of your net profit from a $1 bet on a color, and also report their associated probabilities.  The possible values for net profit are +1 (if the ball lands on your color) and -1 (if it lands on a different color).  The wheel contains 18 slots of your color, so the probability that your net profit is +1 is 18/38, which is about 0.474.  The probability that your net profit is -1 is therefore 20/38, which is about 0.526.  Not surprisingly, it’s a little more likely that you’ll lose than win.
  • b) Determine the expected value of the net profit from betting $1 on a color.  The expected value is $1×(18/38) + (-$1)×(20/38) = -$2/38, which is about -$0.053.
  • c) Interpret what this expected value means.  If you were to bet $1 on a color for a large number of spins of the wheel, then your average net profit would be very close to a loss of $0.053 (about a nickel) per spin.
  • d) Repeat (a)-(c) for betting $1 on a number.  The possible values of net profit are now +35 (if the balls lands on your number) and -1 (otherwise).  The respective probabilities are 1/38 (about 0.026) and 37/38 (about 0.974).  The expected value of net profit is $35×(1/38) + (-$1)×(37/38) = -$2/38, which is about -$0.053.  If you were to bet $1 on a number for a large number of spins of the wheel, then your average net profit would be very close to a loss of $0.053 (about a nickel) per spin.
  • e) How do the expected values of the two types of bets compare?  Explain what this means.  The two expected values are identical.  This means that if you bet for a large number of spins, your average net profit will be to lose about a nickel per spin, regardless of whether you bet on a color or number.
  • f) Are the two types of bets identical?  (Would you get the same experience by betting on a color all evening vs. betting on a number all evening?)  If not, explain their primary difference.  No, the bets are certainly not identical, even though their expected values are the same.  If you bet on a number, you will win much less often than if you bet on a color, but your winning amount will be much larger when you do win.
  • g) The expected value from a $1 bet might seem too small to form the basis for the huge gambling industry.  Explain how casinos can make substantial profits based on this expected value.  Remember that the expected value is the average net profit per dollar bet per spin.  Casinos rely on attracting many customers and keeping them gambling for a large number of spins.  For example, if 1000 gamblers make $1 bets on 1000 spins each, then the expected value* of the casino’s income would 1000×1000×($2/38) ≈ $52,638.58.

* I have resisted the temptation to use a shorthand term such as expected income or expected profit throughout this example.  I believe that saying expected value every time might help students to avoid thinking of “expected” in the everyday sense of the word when we intend its technical meaning.


4. I like to use this question on exams to assess students’ understanding of expected value: At her birthday party, Sofia swings at a piñata repeatedly until she breaks it.  Her mother tells Sofia that she has determined the probabilities associated with the possible number of swings that could be needed for Sofia to break the piñata, and she has calculated the expected value to be 2.4.  Interpret what this expected value means.

A good answer is: If Sofia were to repeat this random process (of swinging until she breaks a piñata) for a very large number of piñatas, then the long-run average number of swings that she would need will be very close to 2.4 swings per piñata.

I look for three components when grading students’ interpretations: 1) long-run, 2) average, and 3) context.  Let’s consider each of these:

  1. The phrase long-run does not need to appear, but the idea of repeating the random process over and over for a large number of repetitions is essential.  I strongly prefer that the interpretation describe what “long run” means by indicating what would be repeated over and over (in this case, the process of swinging at a piñata until it breaks).  
  2. The idea of “average” is absolutely crucial to interpreting expected value, but it’s not uncommon for students to omit this word from their interpretations.   The interpretation makes no sense if it says that Sofia will take 2.4 swings in the long run.
  3. As is so often the case in statistics, context is key.  If a student interprets the expected value as “long-run average” with no other words provided, then the student has not demonstrated an ability to apply the concept to this situation.  In fact, a student could respond “long-run average” without bothering to read a single word about the context.

I also think it’s helpful to ask students, especially those who are studying to become teachers themselves, to critique hypothetical responses to interpreting the expected value, such as:

  • A. The long-run average is 2.4 swings.
  • B. The average number of swings that Sofia needs to break the piñata is 2.4 swings.
  • C. If Sofia were to repeat this random process (of swinging until she breaks a piñata) for a very large number of piñatas, then she would need very close to 2.4 swings in the long run.

I would assign partial credit to all three of these responses. Response A is certainly succinct, and it includes the all-important long-run average.  But the only mention of context in response A is the word “swings,” which I do not consider sufficient for describing the process of Sofia swinging at a piñata until it breaks.  Response B sounds pretty good, as it mentions average and describes the context well, but it is missing the idea of long-run.  Adding “if she were to repeat this process with a large number of piñatas” to response B would make it worthy of full credit.  Response C is so long and generally on-point that it might be hard to see what’s missing.  But response C makes no mention of the word or idea of average.  All that’s needed for response C to deserve full credit is to add “on average” at the end or insert “an average of” before “2.4 swings.”


Can we expect students to understand what expected value means?  Sure, but the unfortunate name makes this more of a challenge than it should be, as it practically begs students to confuse expected value with the value that we expect to occur.  As much as I would like to replace this nettlesome term with long-run average and its abbreviation LRA, I don’t expect* this alternative to catch on in the short term.  But I do hope that this change catches on before the long run arrives.

* Sorry, I can’t stop using this word!

P.S. I borrowed the scenario of Sofia swinging at a piñata from my colleague John Walker, who proposed this context in an exam question with more involved probability calculations.

#17 Random babies

Be forewarned that what you are about to read is highly objectionable. The topic is an introduction to basic ideas of randomness and probability, but that’s not the offensive part.  No, the despicable aspect is the context of the example, which I ask you to accept in the spirit of silliness intended.

One of the classic problems in probability is the matching problem.  When I first studied probability, this was presented in the context of a group of men at a party who throw their hats into the middle of a room and later retrieve their hats at random.  As I prepared to present this problem at the start of my teaching career, I wanted to use a context that would better capture students’ attention.  I described a hospital that returns newborn babies to their mothers at random.  Of course I realized that this context is horrific, but I thought it might be memorable, and I was hoping that it’s so far beyond the pale as to be laughable.  On the end-of-course student evaluations, one question asked what should be changed about the course, and another asked what should be retained.  For the latter question, several of my students wrote: Keep the random babies!  I have followed this advice for thirty years.

If you’d prefer to present this activity with a context that is value-neutral and perhaps even realistic, you could say that a group of people in a crowded elevator drop their cell phones, which then get jostled around so much that the people pick them up at random. That’s a value-neutral and perhaps even realistic setting. It’s also been suggested to me that the context could be a veterinarian who gives cats back to their owners at random*!

* In case you missed post #16 (here), I like cats.


After I describe this scenario to students, for the case with four babies and mothers, I ask: Use your intuition to arrange the following events in order, from least likely to most likely:

  • None of the four mothers gets the correct baby.
  • At least one of the four mothers gets the correct baby.
  • All of the four mothers gets the correct baby.

At this point I don’t care how good the students’ intuitions are, but I do want them to think about these events before we begin to investigate how likely they are.  How will we conduct this investigation?  Simulate!

Before we proceed to use technology, we start with a by-hand simulation using index cards.  I give four index cards to each student and ask them to write a baby’s first name on each card.  Then I ask students to take a sheet of scratch paper and divide it into four sections, writing a mother’s last name in each section*.  You know what comes next: Students shuffle the cards (babies) and randomly distribute them to the sections of the sheet (mothers).  I ask students to keep track of the number of mothers who get the correct baby, which we call the number of matches.  Then I point out that just doing this once does not tell us much of anything. We need to repeat simulating this random process for a large number of repetitions.  I usually ask each student to repeat this three times.

* I used to provide students with names, but I think it’s more fun to let them choose names for themselves.  I emphasize that they must know which baby goes with which mother.  I recommend that they use alliteration, for example with names such as Brian Bahmanyar and Hector Herrera and Jacob Jaffe and Sean Silva**, to help with this.

** These are the names of four graduates from the Statistics program at Cal Poly. Check out their (and others’) alumni updates to our department newsletter (here) to learn about careers that are available to those with a degree in statistics.

Once the students have completed their three repetitions, each goes to the board, where I have written the numbers 0, 1, 2, 3, 4 across the top*, and students put tally marks to indicate their number of matches for each of their repetitions.  Then we count the tallies for each possible value, and finally convert these counts to proportions.  Here are some sample results:

* I make the column for exactly 3 matches very skinny, because students should realize that it’s impossible to obtain this result (because if 3 mothers get the right baby, then the remaining baby must go to the correct mother also).

At this point I tell students that these proportions are approximate probabilities.  I add that the term probability refers to the long-run proportion of times that the event would occur, if the random process were repeated for a very large number of repetitions.  Based on the by-hand simulation with 96 repetitions shown above, our best guesses are that nobody would receive the correct baby in 40.6% of all repetitions and that all four mothers would get the correct baby in 3.1% of all repetitions.


How could we produce better approximations for these probabilities?  Many students realize that more repetitions should produce better approximations.  At this point we turn to an applet (here) to conduct many more repetitions quickly and efficiently.  The screen shots below show how the applet generates the babies (!) and then distributes them at random to waddle to homes, with the colors of diapers and houses indicating which babies belong where.  The sun comes out to shine gloriously at houses with correct matches, while clouds and rain fall drearily on houses that get the wrong baby.

We repeat this for 1 repetition (trial) at a time until we finally tire of seeing the stork and the cute babies, and then we ask the applet to conduct 1000 repetitions.  Here are some sample results:

These are still approximate probabilities, but these are probably closer to the truth (meaning, closer to the theoretical long-run proportions) than our by-hand approximations, because they are based on many more repetitions (1000 instead of 96).  By clicking on the bar in the graph corresponding to 0 matches, we obtain the following graph, which shows the proportion (relative frequency) of occurrences of 0 matches as a function of the number of repetitions (trials):

I point out that this proportion bounces around quite a bit when there are a small number of trials, but the proportion seems to be settling down as the number of repetitions increases.  In fact, it’s not too much of a stretch to believe that the proportion might be approaching some limiting value in the long run.  This limiting value is what the term probability means.

Determine the approximate probability that at least one mother gets the correct baby.  Indicate two different ways to determine this.  Also interpret this (approximate) probability.  One way is to add up the number of repetitions with at least one match: (344 + 241 + 46) / 1000 = 0.631.  Another way is to subtract the estimate for 0 matches from one: 1 – 0.369 = 0.631.  Based on our simulation analysis, we estimate that at least one mother would get the correct baby in 63.1% of all repetitions, if this random process of distributing four babies to mothers at random were repeated a very large number of times.


Can we calculate the exact, theoretical probabilities here?  In other words, can we figure out the long-run limiting values for these proportions?  Yes, we can, and it’s not terribly hard.  But I don’t do this in “Stat 101” courses because I consider this to be a mathematical topic that can distract students’ attention from statistical thinking.  The essential point for statistical thinking is to think of probability as the long-run proportion of times that an event would happen if the random process were repeated a very large number of times, and I think the simulation analysis achieves this goal.

I do present the calculation of exact probabilities in introductory courses for mathematically inclined students and also in a statistical literacy course that includes a unit on randomness and probability.  The first step is to list all possible outcomes of the random process, called a sample space.  In other words, we need to list all ways to distribute four babies to their mothers at random.  This can be quite challenging and time-consuming for students who are not strong mathematically, so I present the sample space to them:

How is this list to be understood?  I demonstrate this for students by analyzing entries in the first column.  The outcome 1234 in the upper left means that all four mothers get the correct baby.  The outcome 2134 below that means that mothers 3 and 4 got the correct baby, but mothers 1 and 2 had their babies swapped.  The outcome 3124 (below the previous one) means that mother 4 got the correct baby, but mother 1 got baby 3 and mother 2 got baby 1 and mother 3 got baby 2.  The outcome 4123 in the bottom left means that all four mothers got the wrong baby: mother 1 got baby 4, and mother 2 got baby 1, and mother 3 got baby 2, and mother 4 got baby 3.

How does this list lead us to probabilities?  We take the phrase “at random” to mean that all 24 of these possible outcomes are equally likely.  Therefore, we can calculate the probability of an event by counting how many outcomes comprise the event and dividing by 24, the total number of outcomes.

Determine the number of matches for each outcome.  Then count how many outcomes produce 0 matches, 1 match, and so on.  Finally, divide by the total number of outcomes to determine the exact probabilities.  Express these probabilities as fractions and also as decimals, with three decimal places of accuracy. I ask students to work together on this and compare their answers with nearby students.  The correct answers are:

Compare these (exact) probabilities to the approximate ones from the by-hand and applet simulations.  Students notice that the simulation analyses, particularly the applet one based on a larger number of repetitions, produced reasonable approximations.

Determine and interpret the probability that at least one mother gets the correct baby.  This probability is (8+6+1)/24 = 15/24 = .625.  We could also calculate this as 1 – 9/24 = 15/24 = .625.  If this random process were repeated a very large number of times, then at least one mother would get the correct baby in about 62.25% of the repetitions.

Determine and interpret the probability that at least half of the four mothers get the correct baby.  This probability is (6+1)/24 = 7/24 ≈ .292.  This means that if this random process were repeated a very large number of times, then at least half of the mothers would get the correct baby in about 29.2% of the repetitions.

Finally, we return to the question of ordering the three events listed above, from least likely to most likely.  The correct ordering is:

  • All four of the mothers get the correct baby (probability .042).
  • None of the four mothers gets the correct baby (probability .375).
  • At least one of the four mothers gets the correct baby (probability .625).

Here are some follow-up questions that I have asked on a quiz or exam:

For parts (a) – (c), suppose that three people (Alisha, Beth, Camille) drop their cell phones in a crowded elevator.  The phones get jostled so much that each person picks up a phone at random.  The six possible outcomes can be listed (using initials) as: ABC, ACB, BAC, BCA, CAB, CBA.

  • a) The probability that all three of them pick up the correct phone can be shown to be 1/6 ≈ .167.  Does this mean that if they repeat this random process (of dropping their three phones and picking them up at random) for a total of 6 repetitions, you can be sure that all three will get the correct phone exactly once?  Answer yes or no; also explain your answer.
  • b) Determine the probability that at least one of them picks up the correct phone.  Express this probability as a fraction and a decimal.  Show your work.
  • c) Interpret what this probability means by finishing this sentence: If the random process (of three people picking up cell phones at random) were repeated a very large number of times, then …

For parts (d) – (f), suppose instead that six people in a crowded elevator drop their cell phones and pick them up at random.

  • d) Would the probability that all of the people pick up the correct phone be smaller, the same, or larger than with three people?
  • e) Which word or phrase – impossible, very unlikely, or somewhat unlikely – best describes the event that exactly five of the six people pick up the correct phone?
  • f) Which word or phrase – impossible, very unlikely, or somewhat unlikely – best describes the event that all six people pick up the correct phone?

Answers: a) No. The 1/6 probability refers to the proportion of times that all three would get the correct phone in the long run, not in a small number (such as six) of repetitions. b) There are four outcomes in which at least one person gets the correct phone (ABC, ACB, BAC, CBA), so this probability is 4/6 = 2/3 ≈ .667. c) … all three people would pick up the correct phone in about 2/3 (or about 66.7%) of the repetitions. d) Smaller e) Impossible f) Very unlikely


I like to think that this memorable context forms the basis for an effective activity that helps students to develop a basic understanding of probability as the long-run proportion of times that an event occurs.

P.S. As I’ve said before, Beth Chance deserves the lion’s share (and then some) of the credit for the applet collection that I refer to often. Carlos Lima, a former student of Beth’s for an introductory statistics course, designed and implemented the animation features in the “random babies” applet.

#16 Questions about cats

I like cats*.  I also notice that it’s simply impossible to spell STATISTICS without the letters C, A, T, and S. These two facts provide more than enough justification for me to ask many questions in class that pertain to cats in one way or another.  I believe that the upcoming questions about felines (and their human friends) can help students to learn important concepts in descriptive statistics, probability, and statistical inference**.

* This is one of the shortest sentences that I’ve ever written, even shorter than: Ask good questions.

** If you are more interested in cats than in these statistical concepts, I invite you to skip down to the P.P.S. at the end of this post to see photos of my cats.


I heard Jay Lehmann present the following question at a conference.  I liked it so much (not only because it mentions cats) that I began using it on my own final exams:

1a) Which would be larger – the average weight of 10 randomly selected people, or the average weight of 1000 randomly selected cats (ordinary domestic housecats)?

Jay mentioned that some of his students struggle with this question, because they don’t think proportionally.  They believe that the weights of 1000 cats must be larger than the weight of 10 people.  This would be true, of course, if we were talking about combined weight, but the question asks about average weight, which requires thinking on a per individual (person or cat) basis. There’s no doubt that people weigh more on average than cats.

I’m pleased to say that my students had no difficulty with this question.  But I decided to ask a second question:

1b) Which would be larger – the standard deviation of the weights of 1000 randomly selected people, or the standard deviation of the weights of 10 randomly selected cats (ordinary domestic housecats)?

The correct answer, of course, is that the standard deviation would be much larger for people than for cats, because weights of people range from just a few pounds for newborns to hundreds and hundreds of pounds for overweight adults.  Cats’ weights range only from a pound or less in kittens to a few dozen pounds for overweight cats.

My students did very poorly on this question.  Why?  I think they believe that a larger sample size produces a smaller standard deviation, period.  I never said that, of course.  What I did say, and what we investigated with simulation, is that the standard deviation of a sample mean decreases as the sample size increases.  We also explored how the standard deviation of a sample proportion decreases as the sample size increases.  We also looked at some formulas that make this more explicit, such as:

I’m afraid that many students came away from these discussions believing that “larger sample sizes produce smaller standard deviations” without paying attention to the crucial of a sample statistic part.  In an effort to curb this misunderstanding, I now try to never say or write standard deviation without adding of what for more clarity.

My students’ performance on this question is especially disheartening because I fear that a higher percentage get this wrong on the final exam than would have at the beginning of the course.  In other words, I worry that my teaching on this topic is violating the fundamental principle of “first do no harm.”

Oh dear, after a light-hearted introduction, this post has taken a discouraging turn!  Let’s move on to happier thoughts about cats (and even dogs) …


The following questions address some basic ideas of working with percentages.  You could use these to introduce, or assess students’ understanding of, probabilities of unions of events.

2. The 2018 General Social Survey (GSS) interviewed a national sample of American adults and found that 47% have a pet dog and 25% have a pet cat.

a) Does it necessarily follow that 72% (which is 47% + 25%) of those surveyed had a pet dog or a pet cat?  If not, is it even possible (in principle anyway) for this to be true?  Under what circumstance (however unrealistic) would this be true?

This conclusion does not follow, because some people have both a pet dog and a pet cat.  In other words, having a dog and having a cat are not mutually exclusive.  It’s theoretically possible that 72% of those surveyed have a pet dog or a pet cat, but this would only be true if absolutely nobody in the survey had both a dog and a cat.

b) The 2018 GSS also found that 14% of survey respondents had both a dog and a cat.  What can you conclude about the percentage who had a dog or a cat?

By adding 47% and 25%, we double-count the people who had both a dog and a cat.  We can compensate for this double-counting by subtracting off the percentage who had both.  The percentage of those surveyed who had a dog or a cat is therefore 47% + 25% – 14% = 58%.

This can be seen by putting the given percentages into the 2×2 table on the left below and then filling in the remaining percentages to produce the table on the right.  The filled-in table shows that you can calculate the percentage who had a dog or a cat by adding the three percentages in red, or else (as I did above) by adding the (marginal) percentages for each pet and then subtracting off the (joint) percentage with both pets in order to compensate for double-counting.

c) If we only knew the percentages in part (a) and not the percentage in part (b), what would be the smallest possible percentage of respondents who owned a pet dog or a pet cat?  Describe the (unrealistic) situation in which this extreme case would occur.

This question is very challenging for many students.  One way to tackle this is to start with the 2×2 table on the left below.  Then realize that to make the percentage with a dog or cat as small as possible, we need to make the percentage in the upper-left cell (with both a dog and a cat) as large as possible.  How large can that percentage be?  No larger than 25%, the percentage with a cat.  The completed table on the right shows that this extreme situation occurs only if none of the respondents had a cat and not a dog.  In other words, the most extreme case is that every person with a cat also had a dog, which gives 47% with a dog or a cat, the same as the percentage with a dog.


The following set of questions is one of my favorites (again, not only because it concerns cats).  I have long used this example to introduce students to two important ideas in statistical inference: the fundamental distinction between statistical significance and practical importance, and the consistency between confidence intervals and hypothesis tests.

3. The 2012 Statistical Abstract of the United States gives information from a national survey of 47,000 U.S. households in 2006, which found that 32.4% of the households sampled had a pet cat.  Consider this as a random sample of American households in 2006.

a) What are the observational units and variable?  What type of variable is this?

The observational units are households, not people and not cats.  The variable is whether or not the household has a cat, which is … (get ready for it) … a CATegorical variable.

b) Conduct a hypothesis test of whether the sample data provide strong evidence that the population proportion of all American households that had a pet cat in 2006 differed from one-third. Summarize your conclusion.

The z-test statistic is calculated as:

With such a large (in absolute value) z-test statistic, the p-value is very small (about 0.00002).  The sample data provide extremely strong evidence that the proportion of all American households that had a pet cat in 2006 was not one-third.

c) Produce and interpret a 99.9% confidence interval for the population proportion of all American households that own a pet cat.

This confidence interval is calculated as:

This becomes .324 ± .007, which is the interval (.317 → .331).  We can be 99.9% confident that the population proportion of American households that had a pet cat in 2006 was between .317 and .331.

Parts (a) – (c) provide fairly routine practice. The following parts introduce students to important ideas.  I encourage students to think through these questions in groups before I lead a discussion about the answers and what they’re supposed to learn from them.  I also caution students to read parts (e) and (f) very carefully to notice the small but important difference in these questions.

d) Are the test decision and confidence interval consistent with each other?  Explain.

Yes, these results are consistent.  The hypothesis test provided extremely strong evidence that the population proportion is not one-third, and the confidence interval does not include the value one-third (roughly .3333).

e) Do the sample data provide very strong evidence that the population proportion who own a pet cat is not one-third?  Explain whether the p-value or confidence interval helps you to decide.

Yes.  The p-value is extremely small (approximately .00002), so the sample data provide very strong evidence that the population proportion is not one-third.  Whatever this population proportion might equal, we have very strong evidence that it’s not one-third.

f) Do the sample data provide strong evidence that the population proportion who own a pet cat is very different from one-third?  Explain whether the p-value or confidence interval helps you to decide.

No.  The confidence interval shows us that we can be very confident that the population proportion who had a cat in 2006 is between about .317 and .331.  In other words, we can be very confident that between 31.7% and 33.1% of all American households had a pet cat in 2006.  In practical terms, this is quite close to one-third, or 33.33%.

g) What aspect of this study is responsible for the somewhat surprising pair of findings that we have very strong evidence that: (1) the population proportion is not one-third, and (2) the population proportion is quite close to one-third?

The driving factor is the very large sample size of 47,000 households. With such a large sample size, even the small difference between the sample percentage (32.4%) and the hypothesized percentage (33.33%) is enough to be statistically significant, meaning that a difference that large would be very unlikely to occur by chance alone.  The large sample size also produces a very narrow confidence interval (even with a very high confidence level), so we can be very confident that the population percentage is very close to 32.4%, which in turn is quite close to one-third in practical terms

The bottom line here is very important for students to understand about statistical inference: With a large sample size, a small difference can be statistically significant but not practically important.


Next comes a series of questions for showing how confidence intervals and hypotheses tests relate when comparing two groups and highlighting the important role of sample size in statistical inference.

4. A national survey of pet owners in the U.S. found that 53% of cat owners and 63% of dog owners said that they would perform CPR on their pets in the event of a medical emergency.

a) Are these numbers parameters or statistics?  Explain.

These numbers are statistics, because they describe the sample of dog and cat owners who were surveyed, not all dog and cat owners in the U.S.

b) State the appropriate null and alternative hypotheses for testing whether the difference between 53% and 63% is statistically significant in this context.

The null hypothesis is that the population proportions who would perform CPR on their pet are the same for dog owners and cat owners.  The alternative hypothesis is that these population proportions are different.  We could represent these hypotheses in symbols as H0: π_dog = π_cat, Ha: π_dog ≠ π_cat.

c) What additional information would you need in order to conduct a test of these hypotheses?

We need to know the sample sizes: how many dog owners and how many cat owners were surveyed?  I have to admit that I am incredibly picky when I grade student responses on this question.  If a student responds with “sample size,” that only gets partial credit. The response needs to use the plural, because learning the combined sample size is not sufficient information for conducting the test.

d) Suppose for now that the sample sizes had been 100 in each group.  Determine the z-score and p-value of the test.  Would you reject the null hypothesis at the .05 significance level?

I ask students to use technology to perform the calculations here, so they can focus on the more important concept to be addressed after part (e).  A free online tool is available here.  The test statistic turns to be z ≈ 1.43, with a two-sided p-value of 0.1520.  This p-value is greater than .05, so the observed difference in sample proportions is not statistically significant at the .05 level.

e) Determine and interpret a 95% confidence interval for the difference in the two population proportions.

Again I ask students to use technology for the calculation, which produces a 95% CI of (-0.036 → 0.236).  We can be 95% confident that the proportion of all dog owners who would perform CPR is anywhere from .036 smaller to .236 larger than the proportion of all cat owners who would perform CPR.

f) Are the test decision and confidence interval consistent with each other?  Explain how you can tell.

Yes, these results are consistent.  We did not conclude that the two groups differ, and the confidence interval (for the difference in population proportions) includes the value zero.

g) Now suppose that the sample sizes had been 500 in each group.  Determine the z-score and p-value and confidence interval.  Summarize your conclusions.

The test statistic becomes z ≈ 3.20, with a two-sided p-value of 0.0014.  The 95% CI becomes (0.039 → 0.161).  Now we do have strong evidence that dog owners and cat owners differ with regard to the population proportion who would perform CPR on their pets.  We can be 95% confident the proportion of all dog owners who would perform CPR is somewhere from .039 to .161 larger than the proportion of all cat owners who would perform CPR.

h) Describe how the p-value and confidence interval changed with the larger sample sizes.

The p-value became much smaller, enough to indicate that the difference in the observed sample proportions was unlikely to have occurred by chance alone.  The confidence interval became much narrower, enough that it contains only positive values, indicating that a higher proportion of dog owners than cat owners would perform CPR on their pet in an emergency.

The point here is to help students recognize once again the substantial role that sample size plays in statistical inference.


I promised back in post #6 (here) that I would devote a future post to nothing but questions about cats.  I am happy to check this off as a promise kept.  I hope that cat-lovers and dog-lovers alike have found something worthwhile in this post. Among their many other benefits to society, cats can help students to learn statistics!


P.S. The percentages from the GSS in question #2 came from a Washington Post article (here).  An earlier Washington Post article (here) summarized discrepancies in pet ownership estimates from different sources. The data in question #3 can be found in Table 1241 of the 2012 Statistical Abstract of the United States (here).  The survey about performing CPR on pets was summarized in a Los Angeles Times article (here).

P.P.S. I dedicate this post to the three cats who have been provided so much happiness to my wife and me.  Our first cat Eponine was a classic scaredy-cat, afraid of her own shadow.  She decided early in life that she would never do anything daring but would try to live as long as possible.  She succeeded quite well, making it to 23 years and 3 months.  On the other hand, Cosette sought adventure and lived every day to the fullest.  As a self-respecting calico cat, she became the undisputed, benevolent head of our household from the moment she joined it.  Our current cat Puti is a very good-natured boy who loves to purr, sit on laps, and complain that his 6am breakfast is served much too late in the day.

My three cats: Eponine (top left), Cosette (top right), Puti (bottom)

#15 How confident are you? part 2

How confident are you that your students can interpret a 95% confidence interval (CI) correctly?  This post continues the previous one (here) by considering numerical data and highlighting a common misconception about interpreting a CI for a population mean.

Here is the formula for a one-sample t-interval for a population mean μ, using conventional notation:

It’s worth making sure that students understand this notation.  Two quiz questions that I often ask are: 1.Remind me: what’s the difference between μ and x-bar?  2. Remind me of what the symbol s stands for, and be sure to use three words in your response.  Of course,I want students to say that μ is the symbol for a population mean and x-bar for a sample mean.  I also hope they’ll say that s stands for a sample standard deviation.  If they respond only with standard deviation, I tell them that this response is too vague and does not earn full credit.


Let’s dive in to an example that we’ll use throughout this post: I’d like to estimate the average runtime of a feature film in the thriller genre.  I selected a simple random sample of 50 thriller films from the population of 28,369* thrillers listed at IMDb (here).

* There are actually 41,774 feature films in the thriller genre listed at IMDb on October 13, 2019, but runtimes are provided for only 28,369 of them.

Consider the following (Minitab) output of the sample data:

My questions for students are:

  • (a) What are the observational units and variable?  What type of variable is this?
  • (b) Describe the relevant population and parameter.  Also indicate an appropriate symbol for this parameter.
  • (c) Identify the appropriate confidence interval procedure.
  • (d) Are the technical conditions for this procedure satisfied?  Explain.
  • (e) Calculate a 95% confidence interval for the population mean.
  • (f) Interpret this interval.
  • (g) What percentage of the films in the sample have times that fall within this interval?
  • (h) Is this percentage close to 95%?  Should it be?  Explain what went wrong, or explain that nothing went wrong.

Here are my answers:

  • (a) The observational units are the films.  The variable is the runtime of the film, measured in minutes, which is a numerical variable.
  • (b) The population is all feature films in the thriller genre listed at IMDb for which runtimes are provided.  The parameter is the mean (average) runtime among these flims, denoted by μ.
  • (c) We will use a one-sample t-interval procedure to estimate the population mean μ.
  • (d) The dotplot of the sample data reveals that the distribution of runtimes is skewed to the right.  But the skewness is not extreme, so the sample size of 50 films should be large enough for the t-interval procedure to be valid.
  • (e) The 95% CI for μ is calculated as: 101.70 ± 2.010×25.30/sqrt(50), which is 101.70 ± 7.19, which is the interval (94.51 → 108.89) minutes.
  • (f) We are 95% confident that the population mean runtime of a feature film in the thriller genre in IMDb is between 94.51 and 108.89 minutes.
  • (g) Only 7 of the 50 films (14%) run for more than 94.51 minutes and less than 108.89 minutes, as shown in red in this dotplot:
  • (h) This percentage (14%) is nowhere close to 95%.  Moreover, there’s no reason to expect this percentage to be close to 95%.  Nothing went wrong here.  Remember that the CI is estimating the population mean (average), not individual values.  We do not expect 95% of the individual films’ runtimes to be within this CI.  Rather, we are 95% confident that the population mean of the runtimes is within this CI.

Question (h) indicates a very common and troublesome student misconception.  Many students mistakenly believe that a 95% CI for a population mean is supposed to contain 95% of the data values.  These students are confusing confidence about a parameter with prediction about an individual.  How can we help them to see the mistake here?  I hope that questions (g) and (h) help with this, as students should see for themselves that only 7 of the 50 films (14%) in this sample fall within the CI.  You might also point out that as the sample size increases, the CI for μ will continue to get narrower, so the interval will include fewer and fewer data values.  We can also be sure to ask students to identify parameters in words as often as possible, because I think this misconception goes back to not paying enough attention to what a parameter is in the first place.

Something else we could consider doing* to help students to distinguish between confidence and prediction is to teach them about prediction intervals, which estimate individual values rather than the population mean.  In many situations the relevant question is one of prediction.  For example, you might be much more interested in predicting how long the next thriller film that you watch will take, as opposed to wanting to estimate how long a thriller film lasts on average.

* I confess that I do not typically do this, except in courses for mathematically inclined students such as those majoring in statistics, mathematics, or economics.

Here is the formula for a prediction interval:

Comparing this to the confidence interval formula above, we see that the prediction interval formula has an extra s (sample standard deviation) term. This accounts for variability from individual to individual, which makes the prediction interval much wider than the confidence interval.  For the sample data on runtimes of thriller films, the 95% prediction interval is: 101.70 ± 2.010×25.30×sqrt(1+1/50), which is 101.70 ± 51.36, which is the interval (50.34 → 153.06) minutes.  Notice how wide this interval is: Its half-width is 51.36 minutes (nearly an hour), compared to a half-width of just 7.19 minutes for the confidence interval above.  This prediction interval captures 45 of the 50 runtimes in this sample (90%).

An important caveat is that unlike the t-confidence interval procedure for a population mean, this prediction interval procedure relies heavily on the assumption of a normally distributed population, regardless of sample size.  The runtime distribution is skewed to the right, so this t-prediction interval procedure is probably not valid.  A simpler alternative is to produce a prediction interval by using the (approximate) 2.5th and 97.5th percentiles of the sample data.  For this sample, we could use the second-smallest and second-largest runtime values, which gives a prediction interval of (60 → 163) minutes.  This interval contains 48/50 (96%) of the runtimes in the sample.


Now let’s re-consider question (f), which asked for an interpretation of the confidence interval.  Below are four possible student answers.  As you read these, please think about whether or not you would award full credit for that interpretation:

  • 1. We are 95% confident that μ is between 94.5 and 108.9.
  • 2. We are 95% confident that the population mean is between 94.5 and 108.9 minutes.
  • 3. We are 95% confident that the population mean runtime of a thriller film in the IMDb list is between 94.5 and 108.9 minutes.
  • 4. We are 95% confident that the population mean runtime of a thriller film in the IMDb list is between 94.5 and 108.9 minutes.  This confidence stems from knowing that 95% of all confidence intervals generated by this procedure would succeed in capturing the actual value of the population mean.

I hope we agree that none of these interpretations is flat-out wrong, and they get progressively better as we progress from #1 through #4.  Where would you draw the line about deserving full credit?  I would regard #3 as good enough.  I think #1 and #2 fall short by not providing context.  I view #4 as going beyond what’s needed because the question asked only for an interpretation of the interval, not for the meaning of the 95% confidence level.  I suggest asking a separate question specifically about interpreting confidence level*, in order to assess students’ understanding of that concept.

* I have asked: Explain what the phrase “95% confidence” means in this interpretation. This is a challenging question for most students.


Continuing this deep dive into into interpreting a confidence interval for a population mean, please consider the following incorrect answers.  Think about which you consider to be more or less serious than others, and also reflect on which interpretations deserve full credit, partial credit, or no credit.

  • A. We are 95% confident that a thriller film in the IMDb list runs for between 94.5 and 108.9 minutes.
  • B. There’s a 95% chance that a thriller film in the IMDb list runs for between 94.5 and 108.9 minutes.
  • C. About 95% of all thriller films in the IMDb list run for between 94.5 and 108.9 minutes.
  • D. We are 95% confident that the mean runlength of a thriller film in this sample from the IMDb list was between 94.5 and 108.9 minutes.
  • E. We are 95% confident that the mean runlength of a thriller film in a new random sample from the IMDb list would be between 94.5 and 108.9 minutes.
  • F. There’s a 95% chance (or a 0.95 probability) that the population mean runlength of a thriller film in the IMDb list is between 94.5 and 108.9 minutes.

I contend that A, B, and C are all egregiously wrong.  They all make the same mistake of thinking that the interval predicts the runtime of individual films rather than estimating a mean.  I suppose you could say that A is better than B and C because it uses the word “confident.” In fact, simply inserting “on average” at the end of the sentence would be sufficient to fix A.  But the idea of “on average” is a crucial one to have omitted!

I believe that D and E are slightly less wrong than A, B, and C, because they do include the idea of mean.  But they refer to a sample mean instead of the population mean.  This is also a serious error and so would receive no credit in my class.  I might say that D is worse than E, because we know for sure that the mean runtime in this sample is the midpoint of the confidence interval.

What about F?  It’s not quite correct, because it uses the language of chance and probability rather than confidence.  The population mean μ is a fixed value, so it’s not technically correct* to refer to the probability or chance that μ falls in a particular interval.  What’s random is the confidence interval itself, because the interval obtained from this procedure would vary from sample to sample if we were to take repeated random samples from the population**.  But I consider this distinction between confidence and probability to be fairly minor, especially compared to the much more substantive distinction between confidence and prediction.  I would nudge a student who produced F toward more appropriate language but would award full credit for this interpretation.

* Unless we take a Bayesian approach, which I will discuss in a future post.

** As we saw in the previous post (here) by using the Simulating Confidence Intervals applet (here).


I ask a version of the “do you expect 95% of the data to fall within the CI” question almost every time I ask about interpreting a confidence interval.  I remember one student from many years ago who seemed to be either tickled or annoyed by my repeating this question so often.  In response to such a question on the final exam, he wrote something like: “Boy, some students must get this wrong a lot because you keep asking about it.  Okay, once again, my answer is …”  You might be expecting me to conclude this post on an ironic note by saying that the student then proceeded to give a wrong answer.  But no, he nailed it.  He knew that we do not expect anywhere near 95% of the data values to fall within a 95% confidence interval for the population mean.  I hope that this student would be tickled, and not annoyed, to see that I have now devoted most of a blog post to this misconception.

P.S. The sample data on runtimes can be found in the file below.

#14 How confident are you? part 1

How confident are you that your students understand what “95% confidence” means?  Or that they realize why we don’t always use 99.99% confidence?  That they can explain the sense in which larger samples produce “better” confidence intervals than smaller samples?  For that matter, how confident are you that your students know what a confidence interval is trying to estimate in the first place?  This blog post, and the next one as well, will focus on helping students to understand basic concepts of confidence intervals. (As always, my questions to students appear in italics below.)


I introduce confidence intervals (CIs) to my students with a CI for a population proportion, using the conventional method given by:

Let’s apply this to a surveyed that we encountered in post #8 (here) about whether the global rate of extreme poverty has doubled, halved, or remained about the same over the past twenty years.  The correct answer is that the rate has halved, but 59% of a random sample of 1005 adult Americans gave the (very) wrong answer that they thought the rate had doubled (here). 

Use this sample result to calculate a 95% confidence interval.  This interval turns out to be:

This calculation becomes .59 ± .03, which is the interval (.56, .62)*.  Interpret what this confidence interval means.  Most students are comfortable with concluding that we are 95% confident that something is between .56 and .62.  The tricky part is articulating what that something is.  Some students mistakenly say that we’re 95% confident that this interval includes the sample proportion who believe that the global poverty rate has doubled.  This is wrong, in part because we know that the sample proportion is the exact midpoint of this interval.  Other students mistakenly say that if researchers were to select a new sample of 1005 adult Americans, then we’re 95% confident that between 56% and 62% of those people would answer “doubled” to this question.  This is incorrect because it is again trying to interpret the confidence interval in terms of a sample proportion.  The correct interpretation needs to make clear what the population and parameter are: We can be 95% confident that between 56% and 62% of all adult Americans would answer “doubled” to the question about how the global rate of extreme poverty has changed over the past twenty years.

* How are students supposed to know that this (.56, .62) notation represents an interval?  I wonder if we should use notation such as (.56 → .62) instead?

Now comes a much harder question: What do we mean by the phrase “95% confident” in this interpretation?  Understanding this concept requires thinking about how well the confidence interval procedure would perform if it were applied for a very large number of samples.  I think the best way to explore this is with … (recall from the previous post here that I hope for students to complete this sentence with a joyful chorus of a single word) … simulation!


To conduct this simulation, we use one of my favorite applets*.  The Simulating Confidence Intervals applet (here) does what its name suggests:

  • simulates selecting random samples from a probability distribution,
  • generates a confidence interval (CI) for the parameter from each simulated sample,
  • keeps track of whether or not the CI successfully captures the value of the population parameter, and
  • calculates a running count of how many (and what percentage of) intervals succeed.

* Even though this applet is one of my favorites, it only helps students to learn if you … (wait for it) … ask good questions!

The first step in using the applet is to specify that we are dealing with a proportion, sampling from a binomial model, and using the conventional z-interval, also known as the Wald method:

The next step is to specify the value of the population proportion.  The applet needs this information in order to produce simulated samples, but it’s crucial to emphasize to students that you would not know the value of the population proportion in a real study.  Indeed, the whole point of selecting a random sample and calculating a sample proportion is to learn something about the unknown value of the population proportion.  But in order to study properties of the CI procedure, we need to specify the value of the population proportion.  Let’s use the value 0.40; in other words we’ll assume that 40% of the population has the characteristic of interest.  Let’s make this somewhat more concrete and less boring: Suppose that we are sampling college students and that 40% of college students have a tattoo.  We also need to enter the sample size; let’s start with samples of n = 75 students.  Let’s generate just 1 interval at first, and let’s use 95% confidence:

Here’s what we might observe* when we click the “Sample” button in the applet:

* Your results will vary, of course, because that’s the nature of randomness and simulation.

The vertical line above the value 0.4 indicates that the parameter value is fixed.  The black dot is the value of the simulated sample proportion, which is also the midpoint of the interval (0.413* in this case).  The confidence interval is shown in green, and the endpoint values (0.302 → 0.525) appear when you click on the interval.  You might ask students to use the sample proportion and sample size to confirm the calculation of the interval’s endpoints.  You might also ask students to suggest why the interval was colored green, or you might ask more directly: Does this interval succeed in capturing the value of the population proportion (which, you will recall, we stipulated to be 0.4)?  Yes, the interval from 0.302 to 0.525 does include the value 0.4, which is why the interval was colored green.

* This simulated sample of 75 students must have included 31 successes (with a tattoo) and 44 failures, producing a sample proportion of 31/75 ≈ 0.413).

At this point I click on “Sample” several times and ask students: Does the value of the population proportion change as the applet generates new samples?  The answer is no, the population proportion is still fixed at 0.4, where we told the applet to put it.  What does vary from sample to sample?  This a key question.  The answer is that the intervals vary from sample to sample.  Why do the intervals vary from sample to sample?  Because the sample proportion, which is the midpoint of the interval, varies from sample to sample.  That’s what the concept of sampling variability is all about.

I continue to click on “Sample” until the applet produces an interval that appears in red, such as:

Why is this interval red?  Because it fails to capture the value of the population proportion.  Why does this interval fail when most succeed?  Because random chance produced an unusually small value of the sample proportion (0.253), which led to a confidence interval (0.155 → 0.352) that falls entirely below the value of the population proportion 0.40.

Now comes the fun part and a pretty picture.  Instead of generating one random sample at a time, let’s use the applet to generate 100 samples/intervals all at once.  We obtain something like:

This picture captures what the phrase “95% confidence” means.  But it still takes some time and thought for students to understand what this shows.  Let’s review:

  • The applet has generated 100 random samples from a population with a proportion value of 0.4.
  • For each of the 100 samples, the applet has used the usual method to calculate a 95% confidence interval.
  • These 100 intervals are displayed with horizontal line segments.
  • The 100 sample proportions are represented by the black dots at the midpoints of the intervals.
  • The population proportion remains fixed at 0.4, as shown by the vertical line. 
  • The confidence intervals that are colored green succeed in capturing the value 0.4.
  • The red confidence intervals fail to include the value 0.4.

Now, here’s the key question: What percentage of the 100 confidence intervals succeed in capturing the value of the population proportion?  It’s a lot easier to count the red ones that fail: 5 out of 100.  Lo and behold, 95% of the confidence intervals succeed in capturing the value of the population proportion.  That is what “95% confidence” means.

The applet also has an option to sort the intervals, which produces:

This picture illustrates why some confidence intervals fail: The red intervals were the unlucky ones with an unusually small or large value of the sample proportion, which leads to a confidence interval that falls entirely below or above the population proportion value of 0.4.

A picture like this appears in many statistics textbooks, but the applet makes this process interactive and dynamic.  Next I keep pressing the “Sample” button in order to generate many thousands of samples and intervals.  The running total across thousands of samples should reveal that close to 95% of confidence intervals succeed in capturing the value of the population parameter.

An important question to ask next brings this idea back to statistical practice: Survey researchers typically select only one random sample from a population, and then they produce a confidence interval based on that sample.How do we know whether the resulting confidence interval is successful in capturing the unknown value of the population parameter?  The answer is that we do not know.  This answer is deeply unsatisfying to many students, who are uncomfortable with this lack of certainty.  But that’s the unavoidable nature of the discipline of statistics.  Some are comforted by this follow-up question: If we can’t know for sure whether the confidence interval contains the value of the population parameter, on what grounds can we be confident about this?  Our 95% confidence stems from knowing that the procedure produces confidence intervals that succeed 95% of the time in the long run.  That’s what the large abundance of green intervals over red ones tells us.  In practice we don’t know where the vertical line for the population value is, so we don’t know whether our one confidence interval deserves to be colored green or red, but we do know that 95% of all intervals would be green, so we can be 95% confident that our interval deserves to be green.


Whew, that’s a lot to take in!  But I must confess that I’m not sure that this long-run interpretation of confidence level is quite as important as we instructors often make it out to be.  I think it’s far more important that students be able to describe what they are 95% confident of: that the interval captures the unknown value of the population parameter.  Both of those words are important – population parameter – and students should be able to describe both clearly in the context of the study.

I can think of at least three other aspects of confidence intervals that I think are more important (than the long-run interpretation of confidence level) for students to understand well.


1. Effect of confidence level – why don’t we always use 99.99% confidence?

Let’s go back to the applet, again with a sample size of 75.  Let’s consider changing the confidence level from 95% to 99% and then to 80%.  I strongly encourage asking students to think about this and make a prediction in advance: How do you expect the intervals to change with a larger confidence level?  Be sure to cite two things that will change about the intervals.  Once students have made their predictions, we use the applet to explore what happens:

99% confidence on the left, 80% confidence on the right

The results for 99% confidence are on the left, with 80% confidence on the right.  A larger confidence level produces wider intervals and a larger percentage of intervals that succeed in capturing the parameter value.  Why do we not always use 99.99% confidence?  Because those intervals would typically be so wide as to provide very little useful information*.

* Granted, there might be some contexts for which this level of confidence is necessary.  A very large sample size could prevent the confidence interval from becoming too wide, as the next point shows.


2. Effect of sample size – in what sense do larger samples produce better confidence intervals than smaller samples? Let’s return to the applet with a confidence level of 95%.  Now I ask: Predict what will change about the intervals if we change the sample size from 75 to 300.  Comment on both the intervals’ widths and the percentage of intervals that are successful.  Most students correctly predict that the larger sample size will produce intervals that are more narrow.  But many students mistakenly predict that the larger ample size will result in a higher percentage of successful intervals.  Results such as the following (n = 75 on the left, n = 300 on the right) convince them that they are correct about narrower intervals, but the percentage of successful ones remains close to 95%, because that is controlled by the confidence level:

n = 75 on the left, n = 300 on the right

This graph (and remember that students using the applet would see many such graphs dynamically, rather than simply seeing this static image) confirms students’ intuition that a larger sample size produces narrower intervals.  That’s the sense in which larger sample sizes produce better confidence intervals, because narrower intervals indicate a more precise (i.e., better) estimate of the population parameter for a given confidence level.

Many students are surprised, though, to see that the larger sample size does not affect the green/red breakdown.  We should still expect about 95% of confidence intervals to succeed in capturing the population proportion, for any sample size, because we kept the confidence level at 95%.


3. Limitations of confidence intervals – when should we refuse to calculate a confidence interval?

Suppose that an alien lands on earth and wants to estimate the proportion of human beings who are female*.  Fortunately, the alien took a good statistics course on its home planet, so it knows to take a sample of human beings and produce a confidence interval for this proportion.  Unfortunately, the alien happens upon the 2019 U.S. Senate as its sample of human beings.  The U.S. Senate has 25 women senators (its most ever!) among its 100 members in 2019.

* I realize that this context is ridiculous, but it’s one of my favorites.  In my defense, the example does make use of real data.

a) Calculate the alien’s 95% confidence interval.  This interval is:

This calculation becomes .25 ± .085, which is the interval (.165 → .335).

b) Interpret the interval.  The alien would be 95% confident that the proportion of all humans on earth who are female is between .165 and .335.

c) Is this consistent with your experience living on this planet?  No, the actual proportion of humans who are female is much larger than this interval, close to 0.5.

d) What went wrong?  The alien did not select a random sample of humans.  In fact, the alien’s sampling method was very biased toward under-representing females.

e) As we saw with the applet, about 5% of all 95% confidence intervals fail to capture the actual value of the population parameter.  Is that the explanation for what went wrong here?  No!  Many students are tempted to answer yes, but this explanation about 5% of all intervals failing is only relevant when you have selected random samples over and over again.  The lack of random sampling is the problem here.

f) Would it be reasonable for the alien to conclude, with 95% confidence, that between 16.5% and 33.5% of U.S. senators in the year 2019 are female?  No.  We know (for sure, with 100% confidence) that exactly 25% of U.S. senators in 2019 are female.  If that’s the entire population of interest, there’s no reason to calculate a confidence interval.  This question is a very challenging one, for which most students need a nudge in the right direction.

The lessons of this example are:

  • Confidence intervals are not appropriate when the data were collected with a biased sampling method.  A confidence interval calculated from such a sample can provide very dubious and misleading information.
  • Confidence intervals are not appropriate when you have access to the entire population of interest.  In this unusual and happy circumstance, you should simply describe the population.

I feel a bit conflicted as I conclude this post.  I have tried to convince you that the Simulating Confidence Intervals applet provides a great tool for leading students to explore and understand what the challenging concept of “95% confidence” really means.  But I also have also aimed to persuade you that many instructors over-emphasize this concept at the expense of more important things for students to learn about confidence intervals.

I will continue this discussion of confidence intervals in the next post, moving on to numerical variables and estimating a population mean.

#13 A question of trust

Which do you trust more: a simulation-based* or normal-based analysis of an inference question?  In other words, if a simulation analysis and normal approximation give noticeably different p-values, which would you believe to be closer to the correct p-value?  Please think about this question in the abstract for a moment.  Soon we’ll come back to it in specific example.

* If you’re not familiar with simulation-based inference, I recommend reading post #12 (here) first.


Here’s the example that we’ll consider throughout this post: Stemming from concern over childhood obesity, researchers investigated whether children might be as tempted by toys as by candy for Halloween treats (see abstract of article here).  Test households in five Connecticut neighborhoods offered two bowls to trick-or-treating children: one with candy and one with small toys.  For each child, researchers kept track of whether the child selected the candy or the toy.  The research question was whether trick-or-treaters are equally likely to select the candy or toy.  More specifically, we will investigate whether the sample data provide strong evidence that trick-or-treaters have a tendency to select either the candy or toy more than the other.

In my previous post (here) I argued against using terminology and formalism when first introducing the reasoning process of statistical inference.  In this post I’ll assume that students have now been introduced to the structure of hypothesis tests, so we’ll start with a series of background questions before we analyze the data (my questions to students appear in italics):

  • What are the observational units?  The trick-or-treaters are the observational units.
  • What is the variable, and what type of variable is it?  The variable is the kind of treat selected by the child: candy or toy.  This is a binary, categorical variable.
  • What is the population of interest?  The population is all* trick-or-treaters in the U.S.  Or perhaps we should restrict the population to all trick-or-treaters in Connecticut, or in this particular community.
  • What is the sample?  The sample is the trick-or-treaters in these Connecticut neighborhoods whose selections were recorded by the researchers.
  • Was the sample selected randomly from the population?  No, it would be very difficult to obtain a list of trick-or-treaters from which one could select a random sample.  Instead this is a convenience sample of trick-or-treaters who came to the homes that agreed to participate in the study.  We can hope that these trick-or-treaters are nevertheless representative of a larger population, but they were not randomly selected from a population.
  • What is the parameter of interest?  The parameter is the population proportion of all* trick-or-treaters who would select the candy if presented with this choice between candy and toy.  Alternatively, we could define the parameter to be the population proportion who would select the toy.  It really doesn’t matter which of the two options we designate as the “success,” but we do need to be consistent throughout our analysis.  Let’s stick with candy as success.
  • What is the null hypothesis, in words?  The null hypothesis is that trick-or-treaters are equally likely to select the candy or toy.  In other words, the null hypothesis is that 50% of all trick-or-treaters would select the candy.
  • What is the alternative hypothesis, in words?  The alternative hypothesis is that trick-or-treaters are not equally likely to select the candy or toy.  In other words, the alternative hypothesis is that the proportion of all trick-or-treaters who would select the candy is not 0.5.  Notice that this is a two-sided hypothesis.
  • What is the null hypothesis, in symbols?  First we have to decide what symbol to use for a population proportion.  Most teachers and textbooks use p, but I prefer to use π.  I like the convention of using Greek letters for parameters (such as μ for a population mean and σ for a population standard deviation), and I see no reason to abandon that convention for a population proportion.  Some teachers worry that students will immediately think of the mathematical constant 3.14159265… when they see the symbol π, but I have not found this to be a problem.  The null hypothesis is H0: π = 0.5.
  • What is the alternative hypothesis, in symbols?  The two-sided alternative hypothesis is Ha: π ≠ 0.5.

* I advise students that it’s always a nice touch to insert the word “all” when describing a population and parameter.

Whew, that was a lot of background questions!  Notice that I have not yet told you how the sample data turned out.   I think it’s worth showing students that the issues above can and should be considered before looking at the data.  So, how did the data turn out?  The researchers found that 148 children selected the candy and 135 selected the toy.  The value of the sample proportion who selected the candy is therefore 148/283 ≈ 0.523.

Let’s not lose sight of the research question here: Do the sample data provide strong evidence that trick-or-treaters have a tendency to select either the candy or toy more than the other?  To pursue this I ask: How can we investigate whether the observed value of the sample statistic (.523 who selected the candy) would be very surprising under the null hypothesis that trick-or-treaters are equally likely to select the candy or toy?  I hope that my students will erupt in a chorus of, “Simulate!”*

* I tell my students that if they ever drift off to sleep in class and are startled awake to find that I have called on them with a question, they should immediately respond with: Simulate!  So many of my questions are about simulation that there’s a reasonable chance that this will be the correct answer.  Even if it’s not correct, I’ll be impressed.


Here is a graph of the distribution of sample proportions resulting from 10,000 repetitions of 283 coin flips (using the One Proportion applet here):

I ask students: Describe the shape, center, and variability of the distribution of these simulated sample proportions.  The shape is very symmetric and normal-looking.  The center appears to be near 0.5, which makes sense because our simulation assumed that 50% of all children would choose the candy.  Almost all of the sample proportions fall between 0.4 and 0.6, and it looks like about 90% of them fall between 0.45 and 0.55.

But asking about shape, center, and variability ignores the key issue.  Next I ask this series of questions:

  • What do we look for in the graph, in order to assess the strength of evidence about the research question?  We need to see whether the observed value of the sample statistic (0.523) is very unusual.
  • Well, does it appear that 0.523 is unusual?  Not unusual at all.  The simulation produced sample proportions as far from 0.5 as 0.523 fairly frequently.
  • So, what do we conclude about the research question, and why?  The sample data (0.523 selecting the candy) would not be surprising if children were equally likely to choose the candy or toy, so the data do not provide enough evidence to reject the (null) hypothesis that children are equally likely to choose the candy or toy.

We could stop there, absolutely.  We don’t need to calculate a p-value or anything else in order to draw this conclusion.  We can see all we need from the graph of simulation results.  But let’s go ahead and calculate the (approximate) p-value from the simulation.  Because we have a two-sided alternative, a sample proportion will be considered as “extreme” as the observed one if it’s at least as far from 0.5 as 0.523 is.  In other words, the p-value is the probability of obtaining a sample proportion of 0.477 or less, or 0.523 or more, if the null hypothesis were true.  The applet reveals that 4775 of the 10,000 simulated sample proportions are that extreme, as shown in red below:

The approximate p-value from the simulation analysis is therefore 0.4775.  This p-value is nowhere near being less than 0.05 or 0.10 or any reasonable significance level, so we conclude that the sample data do not provide sufficient evidence to reject the null hypothesis that children are equally likely to choose the candy or toy.


When I first asked about how to investigate the research question, you might have been thinking that we could use a normal approximation, also known as a one-proportion z-test.  Let’s do that now: Apply a one-proportion z-test to these data, after checking the sample size condition.  The condition is certainly satisfied: 283(.5) = 141.5 is far larger than 10.  The z-test statistic can be calculated as:

This z-score tells us that the observed sample proportion who selected candy (0.523) is less than one standard deviation away from the hypothesized value of 0.5.  The two-sided p-value from the normal distribution to be ≈ 2×0.2198 = 0.4396.  Again, of course, the p-value is not small and so we conclude that the sample data do not provide sufficient evidence to reject the null hypothesis of equal likeliness.


But look at the two p-values we have generated: 0.4775 and 0.4396.  Sure, they’re in the same ballpark, but they’re noticeably different.  On a percentage basis, they differ by 8-9%, which is non-trivial.  Which p-value is correct?  This one is easy: Neither is correct!  These are both approximations.

Finally, we are back to the key question of the day, alluded to the title of this post and posed in the first paragraph: Which do you trust more: the (approximate) p-value based on simulation, or the (approximate) p-value based on the normal distribution?  Now that we have a specific example with two competing p-values to compare, please think some more about your answer before you read on.


Many students (and instructors) place more trust in the normal approximation.  One reason for this is that the normal distribution is based on a complicated formula and sophisticated mathematics.  Take a look at the probability density function* of a normal distribution:

* Oh dear, I must admit that in this expression the symbol π does represent the mathematical constant 3.14159265….

How could such a fancy-looking formula possibly go wrong?  More to the point, how could this sophisticated mathematical expression possibly do worse than simulation, which amounts to just flipping a coin a whole bunch of times?

An even more persuasive argument for trusting the normal approximation, in many students’ minds, is that everyone gets the same answer if they perform the normal-based method correctly.  But different people get different answers from a simulation analysis.  Even a single person gets different answers if they conduct a simulation analysis a second time.  This lack of exact replicability feels untrustworthy, doesn’t it?


So, how can we figure out which approximation is better?  Well, what does “better” mean here?  It means closer to the actual, exact, correct p-value.  Can we calculate that exact, correct p-value for this Halloween example? If so, how? Yes, by using the binomial distribution.

If we let X represent a binomial distribution with parameters n = 283 and π = 0.5, the exact p-value is calculated as Pr(X ≤ 135) + Pr(X ≥ 148)*.  This probability turns out (to four decimal places) to be 0.4757.  This is the exact p-value, to which we can compare the approximate p-values.

* Notice that the values 135 and 148 are simply the observed number who selected toy and candy, respectively, in the sample.

So, which approximation method does better?  Simulation-based wins in a landslide over normal-based:

This is not a fluke.  With 10,000 repetitions, it’s not surprising that the simulation-based p-value* came so close to the exact binomial p-value.  The real question is why the normal approximation did so poorly, especially in this example where the validity conditions were easily satisfied, thanks to a large sample size of 283 and a population proportion of 0.5.

* I promise that I only ran the simulation analysis once; I did not go searching for a p-value close to the exact one. We could also calculate a rough margin-of-error for the simulation-based p-value to be about 1/sqrt(10,000) ≈ .01.


The problem with the normal approximation, and a method for improving it, go beyond the scope of a typical Stat 101 course, but I do present this in courses for mathematically inclined students.  First think about it: Why did the normal approximation do somewhat poorly here, and how might you improve the normal approximation?

The problem lies in approximating a discrete probability distribution (binomial) with a continuous one (normal).  The exact binomial probability is the sum of the heights of the red segments in the graph below, whereas the normal approximation calculates the area under the normal curve to the left of 135 and the right of 148:

The normal approximation can be improved with a continuity correction, which means using 135.5 and 147.5, rather than 135 and 148, as the endpoints for the area under the curve.  This small adjustment leads to including a bit more of the area under the normal curve.  The continuity-corrected z-score becomes 0.713 (compared to 0.773 without the correction) and the two-sided normal-based p-value (to four decimal places) becomes 0.4756, which differs from the exact binomial p-value by only 0.0001.  This seemingly minor continuity correction greatly improves the normal approximation to the binomial distribution.


My take-away message is not that normal-based methods are bad, and also not that we should teach the continuity correction to introductory students.  My point is that simulation-based inference is good!  I think many teachers regard simulation as an effective tool for studying concepts such as sampling distributions and for justifying the use of normal approximations.  I agree with this use of simulation wholeheartedly, as far as it goes.  But we can help our students to go further, recognizing that simulation-based inference is very valuable (and trustworthy!) in its own right.

#12 Simulation-based inference, part 1

We have had tastes of simulation-based inference (abbreviated SBI) in earlier posts.  Post #2 (here), about my all-time favorite question, presented simulation results for the statistic (mean/median) as a measure of skewness.  Post #9 (here), about the 1970 draft lottery, presented a simulation analysis of the correlation coefficient as a measure of lottery fairness.  Now let’s take a step back and consider how one might first introduce students to the concept of statistical inference, more specifically the concept of strength of evidence, through simulation.  You could do this near the very beginning of a course as an introduction to statistical thinking, or you could present this as an introduction to a unit on statistical inference.

Let’s start with real data from a genuine research study in brain science* (described here).  A patient suffered brain damage that caused a loss of vision on the left side of her visual field.  A researcher showed two cards to this patient.  Each card showed a simple line drawing of a house.  However, one of the drawings showed flames coming out of the left side of the house.  The researcher shuffled the two cards, placed them down on a table with one card above the other, and asked the patient which house she would rather live in.  The patient replied that this was a silly question because “they are the same.”  The researcher asked her to choose anyway. The cards were shuffled and placed on the table a total of 17 times. The patient chose the non-burning house in 14 of those 17 showings.  The researcher investigated whether these data provide strong evidence that this patient has a condition known as “blindsight,” meaning that she responds to information from the blind part of her visual field even though she cannot “see” those stimuli.

* I learned about this study from a presentation by Rob Kass at the 2017 U.S. Conference on Teaching Statistics, and I saw Doug Tyson give a workshop presentation about using this context to introduce simulation-based inference.

After I present the background of this study, I first ask students*: Identify the observational units and variable in this study.  The observational units are the 17 showings of pairs of cards, and the variable is which house the patient chose, a binary categorical variable.  Then I ask an obvious question: Did the patient identify the non-burning house for more than half of the showings?  Of course the answer is yes, but I think this question helps to prepare students for the challenging question that comes next: Identify two possible explanations for this result.

* Those of you who read post #11 (here) will not be surprised by this.

I often have to nudge students in the direction I’m looking for.  I have in mind that one explanation is that this patient truly has blindsight, so she really is more likely to choose the non-burning house.  The other explanation, which is surprisingly difficult for students to consider without prompting, is that the patient’s selections are simply the result of random chance. In other words, this second explanation asserts that the patient is equally likely to choose either version of the house on each showing.

Then I ask: Which of these two explanations is easier to investigate, and how might we investigate it with a common device?  Most students realize that the “random chance” explanation is fairly easy to investigate by tossing a coin.  How many coin tosses do we need?  Seventeen, one for each showing of a pair of houses to the patient.  What will heads and tails represent?  Heads will represent choosing the non-burning house, tails will represent choosing the burning house.  (Or you could swap these, it doesn’t matter which is which.)

At this point I ask each student in the class to toss a coin* 17 times and count the number of heads.  As the students finish their tosses, they go to the board and put a dot on a dotplot to indicate how many heads they obtained in their 17 tosses.  In this manner a class of 35 students produces a graph** such as:

* I recommend taking coins to class with you, because carrying coins is not very common for today’s students!

** You might ask students about the observational units and variable in this graph.  The variable is fairly clear and should appear in the axis label: number of heads in 17 coin tosses.  But the observational units are trickier to think about: 35 sets of 17 coin tosses.  I often wait until the end of the activity to ask students about this, because I don’t want to distract attention from the focus on understanding strength of evidence.

What can we learn from this graph, about whether the study’s result provides strong evidence that this patient has blindsight?  The important aspect of the graph for addressing this question is not the symmetric shape or the center near 8.5 (half of 17), although those are worth pointing out as what we expect in this situation.  Our goal is to assess whether the observed result for this patient (14 selections of the non-burning house in 17 showings) would be surprising, if in fact the subject’s selections were random.  What’s important in the graph is that none of these 35 repetitions of the study produced 14 or more heads in 17 simulated coin tosses.  This suggests that it would be pretty surprising to obtain a result as extreme as the one in this study, if the subject was making selections at random.  So, this suggests that the patient’s selections were not random, that she was actually more likely to select the non-burning house.  In other words, our simulation analysis appears to provide fairly strong evidence that this subject truly has blindsight.

Now I hope that a student will ask: Wait a minute, is 35 repetitions enough to be very informative?  Good question! We really should conduct this simulation analysis with thousands of repetitions, not just 35, in order to get a better sense for what would happen if the subject’s selections are random.  I jokingly ask students whether they would like to spend the next several hours tossing coins, but we agree that using software would be much quicker.

We turn to an applet from the RossmanChance* collection to perform the simulation (link; click on One Proportion).  First we need to provide three inputs for the simulation analysis:

* As I have mentioned before, Beth Chance deserves virtually all of the credit for these applets.

One of my favorite aspects of this applet is that it mimics the tactile simulation.  The applet shows coins being tossed, just as students have already done with their own coins:

Here are the results of 10,000 repetitions:

With so many repetitions, we now have a very good sense for what would happen with 17 selections made at random, if this study were repeated over and over.  We see a symmetric distribution centered around 8.5 heads.  We also notice that getting 14 heads in 17 tosses is not impossible with a random coin.  But we see (and this is the key) that it’s very unlikely to obtain 14 or more heads in 17 tosses of a random coin.  We can take this one step further by counting how many of the 10,000 repetitions produced 14 or more heads:

In 10,000 simulated repetitions of this study, under the assumption that only random chance controlled the patient’s selections, we find that only 59 of those repetitions (less than one percent) resulted in 14 or more selections of the non-burning house.  What can we conclude from this, and why?  Well, this particular patient really did select the non-burning house 14 times.  That would be a very surprising result if she were making selections randomly.  Therefore, we have very strong evidence that the patient was not making selections randomly, in other words that she does have this ability known as blindsight.

There you have it: the reasoning process of statistical inference as it relates to strength of evidence, presented in the context of real data from a genuine research study.  I think students can begin to grasp that reasoning process after a half-hour activity such as this.  I think it’s important not to clutter up the presentation with unnecessary terminology and formalism.  Some of the most important decisions a teacher makes concern what to leave out.  We have left out a lot here: We have not used the terms null and alternative hypothesis, we have not identified a parameter, we have not calculated a test statistic, we have not used the term p-value, we have not spoken of a test decision or significance level or rejecting a null hypothesis.  All of that can wait for future classes; keep the focus for now on the underlying concepts and reasoning process.

Before the end of this class period, I like to introduce students to a new study to see whether they can reproduce such a simulation analysis, draw the appropriate conclusion about strength of evidence, and explain the reasoning process behind their conclusion.  Here’s a fun in-class data collection, based again on a genuine research study: A phenomenon called facial prototyping suggests that people tend to associate certain facial characteristics with certain names.  I present students with two faces from the article here), tell them that the names are Bob and Tim, and ask who (Bob or Tim) is on the left:

In a recent class, 36 of 46 students associated the name Tim with the face on the left.  I asked my students: Conduct a simulation analysis to investigate whether this provides strong evidence that college students have a tendency to associate Tim with the face on the left.  Summarize your conclusion, and explain your reasoning.

First students need to think about what values to enter for the applet inputs:

Just as with the blindsight study, we again need 0.5 for the probability of heads, because if people attach names to faces at random, they would put Tim on the left 50% of the time.  We need 46 tosses, one for each student in the sample.  Any large number will suffice for the number of repetitions; I like to use 10,000. Here are the results:

We see that it’s incredibly unlikely to obtain 36 or more heads in 46 tosses of a random coin.  So, it would be extremely surprising for 36 or more of 46 students to attach Tim to the face on the left, if in fact students make selections purely at random.  Therefore, our class data provide very strong evidence that college students do have a genuine tendency to associate the name Tim with the face on the left.

I like starting with the blindsight example before facial prototyping, because I find it comforting to know in advance that the data are 14 successes in 17 trials for the first example.  I also like that the p-value* turns out to be less than .01 in the blindsight example.  Collecting data from students in class is fun and worthwhile, but you never know in advance how the data will turn out.  The Bob/Tim example is quite dependable; I have used it with many classes and have found consistent results that roughly 65-85% put Tim on the left.

* I’m very glad to be able to use this term with you, even though I hold off on using it with my students.  Having a common language that your readers understand can save a lot of time!

Simulation-based inference (SBI) has become a prominent aspect of my teaching, so it will be a common theme throughout this blog.  Part 2 of this post will introduce SBI for comparing two groups, but I will hold off on that post for a while.  Next week’s post will continue the SBI theme by asking which you put more trust in: simulation-based inference or normal-based inference?

P.S. Simulation-based inference has become much more common in introductory statistics textbooks over the past decade.  One of the first textbooks to put SBI front-and-center was Statistics: Learning in the Presence of Variation, by Bob Wardrop.  I consider Wardrop’s book, published in 1993, to have been ahead of its time.  Beth Chance and I focused on SBI in Investigating Statistical Concepts, Applications, and Methods (ISCAM), which is an introductory textbook aimed at mathematically inclined students (link).  Intended for a more general audience, Introduction to Statistical Investigations presents SBI beginning in chapter 1.  I contributed to this textbook, written by an author team led by Nathan Tintle (link) and making use of Beth’s applets (link). The Lock family has written a textbook called Statistics: Unlocking the Power of Data in which SBI figures prominently (link), using StatKey software (link).  Josh Tabor and Chris Franklin have written Statistical Reasoning in Sports, which uses SBI extensively (link) and has an accompanying collection of applets (link). Andy Zieffler and the Catalysts for Change group at the University of Minnesota have also developed a course and textbook for teaching statistical thinking with SBI (link), making use of TinkerPlots software (link). This list is by no means exhaustive, and instructors can certainly use other software tools, such as R, to implement SBI.  Dozens of instructors have contributed advice to a blog about teaching SBI (link).

#11 Repeat after me

I often repeat myself in class to emphasize a particular point.  A prominent example of this is that I ask the same series of questions at the outset of almost every example throughout the entire course, from the first day of class to the last.  No doubt some of my students roll their eyes as I ask these same questions over and over and over again.  I join in the fun by poking fun at myself as I ask these questions, time after time after time.

What are these questions that I ask so repeatedly as I introduce every example?  The answer is so boring that it’s bound to be a big letdown after this long lead-in.  I’m almost too embarrassed to tell you.  Okay, here goes: What are the observational units and variables in this study?  I also ask students to classify the type of variable (categorical or numerical*).  If there is more than one variable, I also ask about the role of each variable (explanatory or response).  Like I said, very boring.  But I ask these questions in class every single day.

* Until very recently, I always said quantitative rather than numerical.  But now I have decided that just as many of us have retired qualitative in favor of categorical, we can keep things simpler and more consistent by abandoning quantitative for numerical.

Why do I make such a big deal of repeating these questions for every example?  Because students often struggle with knowing what kind of analysis to perform on a given dataset, and the first step toward answering this question is to identify what the observational units and variables are.  These questions are fundamental to knowing how to analyze the data: what kind of graph to produce, which statistic(s) to calculate, and what inference procedure to use.

Very early in the course, I ask my students: Consider yourselves as the observational units in a statistical study; classify the following variables as categorical or numerical:

  • Whether or not you were born in California
  • The day of the week on which you were born
  • How many miles you are from where you were born
  • How many of the original seven Harry Potter books you have read
  • The hand you use to write
  • How many minutes of sleep you have gotten in the past 24 hours
  • Whether or not you have gotten at least 7 hours of sleep in the past 24 hours

Most students find classifying these variables to be straight-forward, but then I ask: Explain why the following are not variables (still considering yourselves to be observational units):

  • Average amount of sleep in the past 24 hours among students in our class
  • Proportion of students in our class who are left-handed

Most students find this question to be difficult.  I explain that these are summaries that describe our class as a whole, not something that can vary from student to student.  If we were to consider classes at our school as the observational units, then we could legitimately consider these to be variables, because these quantities would vary from class to class.

Then I ask: Explain why this question is not a variable:

  • Have left-handers read more Harry Potter books, on average, compared to right-handers?

My point here is that this is a research question, not a variable that can be recorded for each student in the class. This research question involves two variables: handedness (categorical) and number of Harry Potter books read (numerical).

I proceed to give students a series of research questions and ask: What are the observational units and variable(s) in a study to address these questions?  Here are five examples:

1. How long do singers take to sing the national anthem at the start of the Super Bowl?  The observational units here are Super Bowl games.  The variable is the time taken for singing the national anthem at the game, which is numerical.  Here’s a graph of the data from 1991 through 2019:

2. What percentage of kissing couples lean their heads to the right?  The observational units are kissing couples, not individual people.  The variable is the direction in which the couple leans their heads while kissing, which is categorical and binary.  A study of this phenomenon published in Nature in 2003 found that 80 of 124 kissing couples leaned their heads to the right, as shown in this graph:

3. Can a cat’s percent body fat be used to predict its takeoff velocity when jumping?  The observational units are cats, the explanatory variable is percent body fat, and the response variable is takeoff velocity.  Both variables are numerical.  Some students get tripped up by percent body fat being numerical, because they mistakenly think that percents are only associated with categorical variables.  Researchers investigated this question by collecting data on a sample of domestic housecats, producing the following graph:

4. Do people display different amounts of creativity depending on whether they experience intrinsic or extrinsic motivation?  People with extensive experience with creative writing were randomly assigned to one of two groups: 24 people answered a survey about intrinsic motivations for writing (such as the pleasure of self-expression) and the other 23 people answered a survey about extrinsic motivations (such as public recognition).  Then all 47 people were instructed to write a Haiku poem, and these poems were evaluated for creativity on a numerical scale of 0-30 by a panel of judges.  The observational units here are the writers.  The explanatory variable is the motivation type – intrinsic or extrinsic, which is categorical and binary.  The response variable is the creativity score of their Haiku poem, which is numerical.  The resulting data are displayed in the following graph*:

* Links to data sources can be found in a P.S. at the end of this post.  For now I want to say that I came across these data in The Statistical Sleuth by Ramsey and Schafer.  In addition to having the best title of any statistics textbook, the Sleuth also includes this wonderful sentence: Statistics is like grout – the word feels decidedly unpleasant in the mouth, but it describes something essential for holding a mosaic in place.

5. Were eight-hour hospital shifts on which Kristen Gilbert worked as a nurse more likely to have a patient death than shifts on which Gilbert did not work?  Data on this question were presented in the murder case of Kristen Gilbert, a nurse accused of being a serial killer of patients.  Many students are tempted to say that the observational units are patients, but the shifts are the observational units here.  The explanatory variable is whether or not Gilbert was working on the shift, which is categorical and binary.  The response variable is whether or not a patient died on the shift, which is also categorical and binary. The data are summarized in the table and displayed in the graph below:

Notice that these research questions involve five different scenarios: one numerical variable, one categorical variable, two numerical variables, one variable of each type, and two categorical variables.  I draw students’ attention to how the type of graph is different for each scenario.  You might also notice that one of these studies (#4) is a randomized experiment, but the others are observational.  Another question that I ask repeatedly at the outset of most examples is whether the study involved random sampling, random assignment, both, or neither.  I will return to this theme in a future post.

I also like to show Hans Rosling’s video about human progress across 200 countries in 200 years in 4 minutes, and then I present the following “bubble” graph from Rosling’s gapminder software:

Before we get to interesting questions about this graph, I start with these (boring, repetitive) questions: a) What are the observational units in this graph? b) What variable does Rosling use to represent health?  What type of variable is this?  Is this the explanatory or response variable in the graph? c) What variable does Rosling use to represent wealth?  What type of variable is this?  ?  Is this the explanatory or response variable in the graph? d) What variable is represented by the color of the dots?  What type of variable is this e) What variable is represented by the size of the dots?  What type of variable is this?

Everything I’ve described here happens very early in the course, but these questions about observational units and variables keep coming and coming throughout the entire term.  When we study five-number summaries and boxplots, first I ask about the observational units and variables in the dataset.  When I am ready to introducing scatterplots and correlation and regression, first I ask about the observational units and variables in the dataset.  When it’s time to study chi-square tests, first I ask about the observational units and variables in the dataset.  You get the idea.

Observational units and variables are especially important when studying sampling distributions.  Consider these two graphs, from an activity about sampling words from the Gettysburg Address:

The graph on the left shows the distribution of word length, as measured by number of letters, in a random sample of 10 words.  The observational units are words, and the variable (that varies from word to word) is word length.  On the other hand, the graph on the right displays the distribution of sample mean word lengths in 1000 random samples of size 10.  The observational units now are not individual words but samples of 10 words each, and the variable (that varies from sample to sample) is the sample mean word length.  This distinction can be challenging for students to follow, but it’s crucial for understanding what a sampling distribution is.

To assess how well students understand observational units and variables, I ask questions such as the following on assignments, quizzes, and exams:

A1. Suppose that the observational units in a study are patients who entered the emergency room at French Hospital in the previous week.  For each of the following, indicate whether it is a categorical variable, a numerical variable, or not a variable with regard to these observational units. a) How long the patient waits to be seen by a medical professional b) Whether or not the patient has health insurance c) Day of the week on which the patient arrives d) Average wait time before the patient is seen by a medical professional e) Whether or not wait times tend to be longer on weekends than weekdays f) Total cost of the emergency room visit

These are fairly straightforward for most students, but some struggle with the ones that are not variables at all (d, e).

A2. Select either all Super Bowl games that have been played or all movies that have won the Academy Award for Best Picture as the observational units in a study.  Identify one categorical variable and one numerical variable that could be recorded for these observational units.

This can be a bit tricky for students, in part because the observational units are not people.  It’s also naturally harder for students to think up variables for themselves rather than answer questions about variables provided to them.

A3. Researchers studied whether metal bands used for tagging penguins are harmful to their survival.  Researchers tagged 100 penguins with RFID chips, and then they randomly assigned half of the penguins to also receive a metal band.  Researchers then kept track of which penguins survived throughout the study and which did not. a) Identify the observational units. b) Identify and classify the explanatory variable. c) Identify and classify the response variable.

This question is not especially challenging, but some students have trouble with providing a clear description of the variables.  I prefer language such as “whether or not the penguin received a metal band” and “whether or not the penguin survived.”  If a student writes “metal band” and “survival,” it’s not clear whether they are describing the variables or one of the outcomes for each variable.

A4. Consider transactions at the on-campus snack bar to be the observational units in a statistical study.  State a research question that involves a categorical variable and a numerical variable for these observational units.  Also clearly identify and classify the two variables.

I have found that this question is very challenging for students.  I now realize that they need lots of practice with coming up with their own research questions.  I have in mind answers such as: Do people who pay with cash take longer to serve, on average, compared to people who pay with a card?  The explanatory variable is whether the customer pays with cash or card, which is categorical and binary. The response variable is how long the transaction takes to complete, which is numerical.

Let me wrap this up: I know these are boring questions.  I frequently say to my students: Like always, let’s answer the boring questions before we get to the interesting parts.  I’m truly reluctant to publish this blog post about such boring questions!  But I do think these are important questions to ask, and I am convinced that it’s helpful to ask them over and over and over again.  I have also come to believe that answering these questions is not as straightforward for students as I used to think.  In addition, I hope that students appreciate the interesting research questions and datasets and contexts, which we revisit later in the course, in which I pose these questions.

I forget: Did I mention that I often repeat myself in class to emphasize a particular point?

P.S. The data on Super Bowl national anthem singing times came from here and here. The article about kissing couples can be found here. The article about cat jumping is here. The abstract for the article about motivation and creativity is here. The data about the Kristen Gilbert case came from an article written for Statistics: A Guide to the Unknown (described here) by George Cobb and Steven Gelbach, who were statistical expert witnesses on opposite sides of the case. The Rosling video is available here, and the gapminder software is here. The study about penguin survival can be found here.

#10 My favorite theorem

This blog does not do suspense*, so I’ll come right out with it: Bayes’ Theorem is my favorite theorem.  But even though it is my unabashed favorite, I introduce Bayes’ Theorem to students in a stealth manner.  I don’t present the theorem itself, or even its name, until after students have answered an important question by essentially deriving the result for themselves.  The key is to use a hypothetical table of counts, as the following examples illustrate.  As always, questions that I pose to students appear in italics.

* See question #8 in post #1 here.

1. The ELISA test for HIV was developed in the mid-1980s for screening blood donations.  An article from 1987 (here) gave the following estimates about the ELISA test’s effectiveness in the early stages of its development:

  • The test gives a (correct) positive result for 97.7% of blood samples that are infected with HIV.
  • The test gives a (correct) negative result for 92.6% of blood samples that are not infected with HIV.
  • About 0.5% of the American public was infected with HIV.

First I ask students: Make a prediction for the percentage of blood samples with positive test results that are actually infected with HIV.  Very few people make a good prediction here, but I think this prediction step is crucial for creating cognitive dissonance that leads students to take a closer look at what’s going on.  Lately I have rephrased this question as multiple choice, asking students to select whether their prediction is closest to 10%, 30%, 50%, 70%, or 90%.  Most students respond with 70% or 90%.

Then I propose the following solution strategy: Assume that the given percentages hold exactly for a hypothetical population of 1,000,000 people, and use the percentages fill in the following table of counts:

The numbers in parentheses indicate the order in which we can use the given percentages to complete the table of counts.  I insist that all of my students get out their calculators, or use their phone as a calculator, as we fill in the table together, as follows:

  1. 005 × 1,000,000 = 5,000
  2. 1,000,000 – 5,000 = 995,000
  3. 0.977 × 5,000 = 4,885
  4. 5,000 – 4,885 = 115
  5. 0.926 × 995,000 = 921,370
  6. 995,000 – 921,370 = 73,630
  7. 4,885 + 73,630 = 78,515
  8. 115 + 921,370 = 921,485

These calculations produce the following table:

Then I say to my students: That was fun, and it filled 10 minutes of class time, but what was the point?  What do we do now with this table to answer the original question?  Many students are quick to point out that we can determine the percentage of positive results that are actually HIV-infected by starting with 78,515 (the total number of positive results) as the denominator and using 4,885 (the number of these positive results that are actually HIV-infected) as the numerator.  This produces: 4,885 / 78,515 ≈ 0.062, or 6.2%.

At this point I act perplexed* and say: Can this really be right?  Why would this percentage be so small when the accuracy percentages for the test are both greater than 90%?  This question is much harder for students, but I encourage them to examine the table and see what’s going on.  A student eventually points out that there are a lot more false positives (people who test positive but do not have the disease) than there are true positives (people who test positive and do have the disease).  Exactly! And why is that?  I often need to direct students’ attention to the base rate: Only half of one percent have the disease, so a very large percentage of them are outnumbered by a fairly small percentage of the 99.5% who don’t have the disease.  In other words, 7.4% of 995,000 people greatly outnumbers 97.7% of 5,000 people.

* I am often truly perplexed, so I have no trouble with acting perplexed to emphasize a point.

I like to think that most students understand this explanation, but there’s no denying that this is a difficult concept.  Simply understanding the question, which requires recognizing the difference between the two conditional percentages (percentage of people with disease who test positive versus percentage of people with positive test result who have disease), can be a hurdle.  To help with this I like to ask: What percentage of U.S. Senators are American males?  What percentage of American males are U.S. Senators?  Are these two percentages the same, fairly close, or very different?  The answer to the first question is a very large percentage: 80/100 = 80% in 2019, but the answer to the second question is an extremely small percentage: 80 / about 160 million ≈ 0.00005%.  These percentages are very different, so it shouldn’t be so surprising that the two conditional percentages* with the ELISA test are also quite different.  At any rate I am convinced that the table of counts makes this more understandable than plugging values into a formula for Bayes’ Theorem would.

* I have avoided using the term conditional probability here, because I think the term conditional percentage is less intimidating to students, suggesting something that can be figured out from a table of counts rather than requiring a mathematical formula.

Some students think this fairly small percentage of 6.2% means that the test result is not very informative, so I ask: How many times more likely is a person to be HIV-infected if they have tested positive, as compared to a person who has not been tested?  This requires some thought, but students recognize that they need to compare 6.2% with 0.5%.  The wording how many times can trip some students up, but many realize that they must take the ratio of the two percentages: 6.2% / 0.5% = 12.4. Then I challenge students with: Write a sentence, using this context, to interpret this value.  A person with a positive test result is 12.4 times more likely to be HIV-infected than someone who has not yet been tested.

I also ask students: Can a person who tests negative feel very confident that they are free of the disease?  Among the blood samples that test negative, what percentage are truly not HIV-infected?  Most students realize that this answer can be determined from the table above: Among the 921,485 who test negative, 921,370 do not have the disease, which is a proportion of 0.999875, or 99.9875%.  A person who tests negative can be quite confident that they do not have the disease.  Such a very high percentage is very important for screening blood donations.  It’s less problematic that only 6.2% of the blood samples that are rejected (due to a positive test result) are actually HIV-infected.

You might want to introduce students to some terminology before moving on.  The 97.7% value is called the sensitivity of the test, and the 92.6% value is called the specificity.  You could also tell students that they have essentially derived a result called Bayes’ Theorem as they produced and analyzed the table of counts.  You could give them a formula or two for Bayes’ Theorem.  The one on the left, presented in terms of H for hypothesis and E for evidence, has a two-event partition (such as disease, not).  A more general of Bayes’ Theorem appears on the right.

I present these versions of Bayes’ Theorem in probability courses and in courses for mathematically inclined students, but I do not show any formulas in my statistical literacy course.  For a standard “Stat 101” introductory course, I do not present this topic at all, as the focus is exclusively on statistical concepts and not probability.

Before we leave this example, I remind students that these percentages were from early versions of the ELISA test in the 1980s, when the HIV/AIDS crisis was first beginning.  Improvements in testing procedures have produced much higher sensitivity and specificity (link).  Running more sophisticated tests on those who test positive initially also greatly decreases the rate of false positives.

I have debated with myself whether to change this HIV testing context for students’ first introduction to these ideas.  One argument against using this context is that the information about sensitivity and specificity is more than three decades old.  Another argument is that 97.7% and 92.6% are not convenient values to work with; perhaps students would be more comfortable with “rounder” values like 90% and 80%.  But I continue to use this context, partly to remind students of how serious the HIV/AIDS crisis was, and because I think the example is compelling.  An alternative that I found recently is to present these ideas in terms of a 2014 study of diagnostic accuracy of breathalyzers sold to the public (link).

Where to next?  With my statistical literacy course, I give students more practice with constructing and analyzing tables of counts to calculate reverse conditional percentages, as in the following example.

A national survey conducted by the Pew Research Center in late 2018 produced the following estimates about educational attainment and Twitter use among U.S. adults:

  • 10% have less than a high school diploma; 8% of these adults use Twitter
  • 59% have a high school diploma but no college degree; 20% of these adults use Twitter
  • 31% have a college degree; 30% of these adults use Twitter

What percentage of U.S. adults who use Twitter have less than a high school diploma?  What percentage have a high school degree but no college degree?  What percentage have a college degree?Which age groups are more likely than they were initially?  Which are less likely?

Again we can answer these questions (about reverse conditional percentages from what was given) by constructing a table of counts for a hypothetical population.  This time we need three rows rather than two, in order to account for the three education levels. I recommend providing students with the outline of the table, but without indicating the order in which to fill it in this time:

With numbers in parentheses again indicating the order in which the cells can be calculated, the completed table turns out to be:

From this table we can calculate that 8/219 ≈ .037, or 3.7% of Twitter users have less than a high school degree, 118/219 ≈ .539, or 53.9% of Twitter users have a high school but not college degree, and 93/219 ≈ .425, or 42.5% of Twitter users have a college degree.  These percentages have increased from the base rate only for the college degree holders, as 31% of the public has a college degree but 42.5% of Twitter users do.

3. A third application that I like to present concerns the famous Monty Hall Problem.  Suppose that a new car is hidden behind one door on a game show, while goats are hidden behind two other doors.  A contestant picks a door, and then (to heighten the suspense!) the host reveals what’s behind a different door that he knows to have a goat.  Then the host asks whether the contestant prefers to stay with the original door or switch to the remaining door.  The question for students is: Does it matter whether the contestant stays or switches?  If so, which strategy is better, and why?

Most people believe that staying or switching does not matter.  I recommend that students play a simulated version of the game many times, with both strategies, to get a sense for how the strategies compare.  An applet that allows students to play simulated games appears here.  The following graph shows the results of 1000 simulated games with each strategy:

It appears that switching wins more often than staying!  We can determine the theoretical probabilities of winning with each strategy by using Bayes’ Theorem.  More to the point, we can use our strategy of constructing a table of hypothetical counts.  Let’s suppose that the contestant initially selects door #1, so the host will show a goat behind door #2 or door #3.  Let’s use 300 for the number of games in our table, just so we’ll have a number that’s divisible by 3.  Here’s the outline of the table:

How do we fill in this table? Let’s proceed as follows:

  1. Row totals: If the car is equally likely to be placed behind any of the three doors, then the car should be behind each door for 100 of the 300 games.
  2. Bottom (not total) row: Remember that the contestant selected door #1, so when the car is actually behind door #3, the host has no choice but to reveal door #2 all 100 times.
  3. Middle row: Just as with the bottom row, now the host has no choice but to reveal door #3 all 100 times.
  4. Top row: When the car is actually behind the same door that the contestant selected, the host can reveal either of the other doors, so let’s assume that he reveals each 50% of the time, or 50 times in 100 games.

The completed table is therefore:

We can see from the table that for the 150 games where the host reveals door #2, the car is actually behind door #3 for 100 of those 150 games, which is 2/3 of the games.  In other words, if the contestant stays with door #1, they will win 50/150 times, but by switching to door #3, they win 100/150 times. Equivalently, for the 150 games where the host reveals door #3, the car is actually behind door #2 for 100 of those games, which is again 2/3 of the games.  Bottom line: Switching gives the contestant a 2/3 chance of winning the car, whereas staying only gives a 1/3 chance of winning the car.  The easiest way to understand this, I think, is that by switching, the contestant only loses if they picked the correct door to begin with, which happens one-third of the time.

This post is already quite long, but I can’t resist suggesting a follow-up question for students: Now suppose that the game show producers place the car behind door #1 50% of the time, door #2 40% of the time, and door #3 10% of the time.  What strategy should you use?  In other words, which door should you pick to begin, and then should you stay or switch?  What is your probability of winning the car with the optimal strategy in this case?  Explain.

Encourage students to remember the bottom line from above: By switching, you only lose if you were right to begin with.  So, the optimal strategy here is to select door #3, the least likely door, and then switch after the host reveals a door with a goat.  Then you only lose if you were right to begin with, so you only lose 10% of the time.  This optimal strategy gives you a 90% chance of winning the car.  Students who can think this through and describe the correct optimal strategy have truly understood the resolution of the famous Monty Hall Problem.

One final question for this post: Why is Bayes’ Theorem my favorite?  It provides the mechanism for updating uncertainty in light of partial information, which enables us to answer important questions, such as the reliability of medical diagnostic tests, and also fun recreational ones, such as the Monty Hall Problem.  More than that, Bayes’ Theorem provides the foundation for an entire school of thought about how to conduct statistical inference.  I’ll discuss that in a future post.

P.S. Tom Short and I wrote a JSE article (link) about this approach to teaching Bayes’ Theorem in 1995, but the idea is certainly not original with us.  Gerd Gigerenzer and his colleagues introduced the term “natural frequencies” for this approach; they have demonstrated its effectiveness for improving people’s Bayesian reasoning (link).  The Monty Hall Problem is discussed in many places, including by Jason Rosenhouse in his book (link) titled The Monty Hall Problem.  While I’m mentioning books, I will also point out Sharon Bertsch McGrayne’s wonderful book about Bayesian statistics (link), titled The Theory That Would Not Die.