Small Schools Rank Higher: It’s Built-In; Do the Math (& a twist)

From time to time you will see institutions ranked according to various criteria. Generally this is done with the intention of demonstrating how well each is performing. It’s not unusual to see this done with schools and here’s a claim that is often made, and substantiated by the numbers:

Small Schools Tend to Lead the Ranks

This is consistent. For years I saw it in my own region, reflected in the annual report card issued by the Halifax-Based Atlantic Institute for Market Studies (AIMS). Year after year, small schools led the provincial rankings. As a professional whose entire career was devoted to the betterment of small rural schools I wanted to be able to brag about this, to puff out my chest and say, “look, I told you that small schools were better for our children. It’s obvious that the extra care and attention they get on an individual basis, as well as the better socialization caused by the fact that everyone knows everyone else, is making a positive difference.”

I never did that, of course. It’s not because I don’t believe in small schools–I truly do. My silence on the matter was, to some degree due to the fact that at the time the studies were published I was a non-executive member of the Department of Education. As such I was not authorized to speak on its behalf. That was not the real reason though.

No, I knew that a far more powerful force was afoot; something that affected the results much more than did either good teaching, a supportive (small) ecosystem and the presence of many brilliant bay woman and men.

Although all of those are positive factors.

No the most powerful effect was something else, something related to straightforward mathematical behaviour, and if you’ll spare a few minutes of your attention I will explain.

Simply Put: Small schools have an advantage in these rankings that is due only to the fact that they are small.

And there is an unexpected twist too, one not often mentioned in the discussion of the reports.

Here goes!

Let’s simplify the situation and assume that the rankings are based on the outcome of one test only. Furthermore, let’s say that the result of that test, for any given student, is completely random; that is, any student who writes it will get a random grade between 0 and 100. In other words let’s act like there’s really no difference between the students at all of the schools. The small ones will still come out on top.

Let’s see what this would mean for ten small schools (we’ll arbitrarily name them sml01 to sml10), each one having only fifteen students in grade twelve and writing the test on which our report is based. The results for all of the students are tabulated below. In reality the table was produced using a random number generator in Microsoft Excel. You don’t need to read the table in detail. It’s just here so you know I’m not making the whole thing up!

School sml01 sml02 sml03 sml04 sml05 sml06 sml07 sml08 sml09 sml10
Score 15.0 14.0 45.0 55.0 53.0 70.0 55.0 100.0 79.0 56.0
Score 6.0 34.0 94.0 75.0 64.0 75.0 59.0 75.0 73.0 52.0
Score 15.0 83.0 65.0 30.0 84.0 46.0 64.0 10.0 35.0 20.0
Score 64.0 16.0 80.0 77.0 10.0 55.0 85.0 32.0 91.0 97.0
Score 29.0 28.0 96.0 98.0 6.0 67.0 51.0 74.0 69.0 9.0
Score 43.0 29.0 49.0 79.0 17.0 64.0 54.0 11.0 32.0 91.0
Score 31.0 49.0 62.0 33.0 0.0 92.0 35.0 59.0 91.0 45.0
Score 51.0 21.0 98.0 75.0 47.0 57.0 32.0 32.0 25.0 58.0
Score 45.0 27.0 18.0 6.0 24.0 31.0 84.0 5.0 89.0 2.0
Score 62.0 62.0 38.0 84.0 16.0 23.0 39.0 84.0 36.0 17.0
Score 9.0 0.0 6.0 67.0 53.0 99.0 54.0 23.0 97.0 15.0
Score 38.0 34.0 21.0 70.0 58.0 40.0 37.0 21.0 56.0 50.0
Score 21.0 55.0 51.0 97.0 92.0 40.0 48.0 100.0 76.0 4.0
Score 17.0 50.0 43.0 38.0 6.0 1.0 72.0 77.0 16.0 25.0
Score 35.0 69.0 83.0 34.0 75.0 16.0 46.0 51.0 77.0 33.0
Average 32.1 38.1 56.6 61.2 40.3 51.7 54.3 50.3 62.8 38.3

Table 1: School results for ten small schools.

That’s a huge pile of numbers and we are only looking at the results for the schools so lets just redo that table showing only the schools and the averages.

School Average
sml01 32.1
sml02 38.1
sml03 56.6
sml04 61.2
sml05 40.3
sml06 51.7
sml07 54.3
sml08 50.3
sml09 62.8
sml10 38.3
Table 2: Small School Results

Notice that the results show a fair bit of variability. They cluster around an average of 50  but some schools had averages in the thirties while others were up around 60.

Now let’s do it all over again, but this time let’s see what would happen in larger schools (named big01 to big10). For the small schools we assumed there were only 15 students per grade level but for the larger ones let’s assume that there are in fact 120 students in grade 12 writing the test.

The rather long table is below just so you know I’m not pulling the numbers out of my head. As was the case with the small school simulation it was done using a random number generator in Microsoft Excel and just pasted directly into WordPress. Scroll to the bottom of the table 🙂

School big01 big02 big03 big04 big05 big06 big07 big08 big09 big10
Score 67.0 100.0 82.0 25.0 27.0 100.0 10.0 69.0 22.0 96.0
Score 42.0 92.0 63.0 16.0 42.0 12.0 69.0 94.0 66.0 60.0
Score 100.0 27.0 42.0 59.0 88.0 79.0 83.0 49.0 27.0 96.0
Score 44.0 29.0 46.0 63.0 28.0 69.0 31.0 19.0 18.0 59.0
Score 16.0 33.0 66.0 81.0 11.0 21.0 76.0 67.0 70.0 85.0
Score 95.0 31.0 11.0 2.0 80.0 15.0 21.0 78.0 91.0 33.0
Score 32.0 32.0 38.0 34.0 44.0 61.0 55.0 0.0 89.0 64.0
Score 82.0 6.0 96.0 57.0 8.0 52.0 64.0 55.0 62.0 72.0
Score 100.0 48.0 45.0 10.0 49.0 93.0 89.0 72.0 87.0 72.0
Score 88.0 91.0 33.0 0.0 36.0 11.0 76.0 10.0 78.0 55.0
Score 54.0 95.0 41.0 68.0 92.0 75.0 54.0 75.0 20.0 52.0
Score 44.0 79.0 88.0 69.0 82.0 9.0 31.0 74.0 1.0 78.0
Score 71.0 36.0 2.0 51.0 58.0 2.0 17.0 68.0 29.0 36.0
Score 93.0 47.0 89.0 91.0 25.0 47.0 85.0 96.0 63.0 23.0
Score 9.0 79.0 33.0 41.0 68.0 19.0 74.0 81.0 57.0 47.0
Score 77.0 84.0 28.0 44.0 2.0 54.0 37.0 48.0 25.0 54.0
Score 20.0 74.0 33.0 57.0 15.0 65.0 85.0 59.0 21.0 80.0
Score 56.0 98.0 27.0 68.0 45.0 75.0 58.0 71.0 92.0 58.0
Score 7.0 70.0 83.0 74.0 26.0 52.0 71.0 40.0 75.0 87.0
Score 80.0 32.0 65.0 7.0 54.0 62.0 68.0 7.0 87.0 88.0
Score 65.0 12.0 68.0 22.0 5.0 26.0 36.0 92.0 79.0 40.0
Score 87.0 89.0 51.0 70.0 96.0 98.0 56.0 13.0 10.0 51.0
Score 52.0 71.0 13.0 86.0 88.0 54.0 11.0 20.0 26.0 18.0
Score 69.0 57.0 11.0 36.0 39.0 5.0 38.0 56.0 82.0 40.0
Score 95.0 54.0 54.0 77.0 52.0 74.0 100.0 82.0 35.0 7.0
Score 49.0 80.0 24.0 42.0 11.0 82.0 70.0 18.0 30.0 19.0
Score 46.0 26.0 3.0 56.0 54.0 50.0 2.0 9.0 26.0 47.0
Score 58.0 57.0 98.0 62.0 65.0 50.0 7.0 94.0 9.0 43.0
Score 86.0 86.0 32.0 81.0 63.0 49.0 60.0 61.0 93.0 5.0
Score 9.0 54.0 74.0 65.0 27.0 38.0 42.0 30.0 42.0 99.0
Score 41.0 37.0 30.0 70.0 77.0 86.0 58.0 48.0 53.0 99.0
Score 23.0 82.0 9.0 73.0 9.0 9.0 86.0 27.0 57.0 50.0
Score 52.0 97.0 91.0 90.0 58.0 11.0 56.0 16.0 53.0 89.0
Score 54.0 84.0 46.0 0.0 26.0 55.0 36.0 94.0 89.0 46.0
Score 94.0 75.0 32.0 16.0 77.0 9.0 87.0 21.0 58.0 59.0
Score 77.0 27.0 93.0 65.0 61.0 23.0 53.0 60.0 29.0 23.0
Score 41.0 26.0 34.0 21.0 24.0 57.0 34.0 78.0 99.0 90.0
Score 73.0 67.0 83.0 54.0 99.0 63.0 24.0 65.0 75.0 37.0
Score 55.0 76.0 30.0 85.0 92.0 57.0 31.0 69.0 82.0 43.0
Score 12.0 38.0 53.0 56.0 40.0 67.0 3.0 50.0 86.0 90.0
Score 48.0 89.0 86.0 77.0 80.0 83.0 92.0 38.0 67.0 0.0
Score 59.0 81.0 65.0 0.0 47.0 24.0 57.0 18.0 27.0 90.0
Score 32.0 72.0 46.0 54.0 92.0 54.0 41.0 99.0 0.0 87.0
Score 5.0 55.0 0.0 78.0 13.0 83.0 60.0 68.0 68.0 86.0
Score 0.0 0.0 91.0 66.0 38.0 22.0 2.0 82.0 32.0 12.0
Score 19.0 74.0 40.0 54.0 93.0 37.0 68.0 75.0 57.0 35.0
Score 13.0 81.0 36.0 39.0 50.0 3.0 44.0 19.0 100.0 16.0
Score 36.0 95.0 4.0 100.0 60.0 89.0 47.0 99.0 70.0 43.0
Score 29.0 46.0 12.0 92.0 35.0 28.0 17.0 74.0 38.0 85.0
Score 49.0 84.0 35.0 70.0 36.0 12.0 32.0 43.0 81.0 39.0
Score 87.0 32.0 89.0 71.0 11.0 0.0 93.0 51.0 10.0 39.0
Score 43.0 27.0 12.0 9.0 81.0 78.0 52.0 99.0 82.0 86.0
Score 51.0 41.0 50.0 73.0 83.0 65.0 51.0 44.0 89.0 5.0
Score 21.0 56.0 89.0 6.0 47.0 41.0 57.0 17.0 72.0 53.0
Score 12.0 39.0 51.0 18.0 96.0 75.0 23.0 39.0 75.0 39.0
Score 0.0 48.0 11.0 51.0 61.0 22.0 39.0 35.0 88.0 75.0
Score 33.0 53.0 23.0 68.0 88.0 69.0 48.0 40.0 19.0 100.0
Score 31.0 30.0 82.0 31.0 13.0 55.0 89.0 94.0 40.0 60.0
Score 90.0 5.0 19.0 26.0 68.0 60.0 77.0 63.0 51.0 6.0
Score 41.0 65.0 72.0 76.0 91.0 11.0 71.0 37.0 68.0 53.0
Score 24.0 80.0 70.0 73.0 61.0 4.0 79.0 59.0 37.0 73.0
Score 11.0 24.0 72.0 48.0 64.0 28.0 38.0 79.0 66.0 22.0
Score 22.0 13.0 14.0 83.0 2.0 21.0 95.0 100.0 55.0 55.0
Score 50.0 97.0 59.0 85.0 15.0 82.0 77.0 31.0 21.0 92.0
Score 81.0 9.0 45.0 56.0 16.0 55.0 66.0 69.0 79.0 78.0
Score 36.0 74.0 68.0 7.0 36.0 42.0 5.0 76.0 41.0 76.0
Score 30.0 35.0 68.0 59.0 92.0 50.0 9.0 50.0 98.0 97.0
Score 30.0 31.0 2.0 1.0 62.0 64.0 82.0 88.0 84.0 53.0
Score 4.0 46.0 55.0 54.0 61.0 42.0 81.0 77.0 25.0 27.0
Score 32.0 51.0 79.0 58.0 2.0 33.0 66.0 92.0 20.0 68.0
Score 70.0 76.0 52.0 24.0 2.0 21.0 6.0 98.0 63.0 37.0
Score 54.0 68.0 91.0 56.0 58.0 32.0 41.0 74.0 64.0 45.0
Score 37.0 48.0 29.0 42.0 4.0 93.0 10.0 29.0 97.0 40.0
Score 14.0 47.0 46.0 83.0 80.0 52.0 42.0 54.0 33.0 29.0
Score 15.0 2.0 100.0 12.0 9.0 84.0 52.0 53.0 53.0 6.0
Score 8.0 23.0 35.0 63.0 78.0 34.0 30.0 75.0 14.0 54.0
Score 16.0 90.0 13.0 80.0 32.0 29.0 99.0 21.0 34.0 80.0
Score 99.0 48.0 47.0 5.0 71.0 88.0 77.0 68.0 50.0 2.0
Score 45.0 15.0 18.0 38.0 49.0 8.0 90.0 13.0 71.0 33.0
Score 42.0 50.0 86.0 80.0 79.0 53.0 21.0 81.0 53.0 36.0
Score 16.0 14.0 51.0 14.0 19.0 97.0 50.0 49.0 8.0 2.0
Score 34.0 85.0 55.0 54.0 49.0 63.0 1.0 58.0 73.0 13.0
Score 8.0 98.0 9.0 7.0 70.0 78.0 41.0 18.0 94.0 74.0
Score 59.0 43.0 31.0 30.0 97.0 85.0 64.0 94.0 3.0 91.0
Score 29.0 34.0 6.0 17.0 43.0 78.0 67.0 17.0 50.0 34.0
Score 80.0 12.0 98.0 24.0 84.0 25.0 96.0 76.0 16.0 67.0
Score 87.0 89.0 11.0 86.0 5.0 39.0 83.0 98.0 27.0 13.0
Score 62.0 73.0 69.0 91.0 47.0 52.0 91.0 57.0 87.0 39.0
Score 22.0 64.0 86.0 64.0 10.0 88.0 6.0 62.0 91.0 26.0
Score 28.0 74.0 88.0 19.0 45.0 97.0 94.0 3.0 75.0 30.0
Score 27.0 11.0 11.0 55.0 39.0 30.0 39.0 54.0 99.0 86.0
Score 94.0 85.0 60.0 1.0 42.0 23.0 57.0 97.0 58.0 24.0
Score 78.0 7.0 30.0 94.0 26.0 75.0 100.0 11.0 99.0 11.0
Score 94.0 12.0 81.0 50.0 49.0 36.0 68.0 95.0 67.0 33.0
Score 5.0 9.0 39.0 23.0 31.0 29.0 23.0 22.0 57.0 46.0
Score 67.0 55.0 98.0 81.0 80.0 72.0 31.0 53.0 80.0 95.0
Score 68.0 57.0 17.0 34.0 26.0 38.0 46.0 55.0 74.0 24.0
Score 8.0 39.0 82.0 34.0 65.0 74.0 34.0 39.0 62.0 19.0
Score 83.0 97.0 14.0 84.0 71.0 66.0 62.0 13.0 8.0 82.0
Score 83.0 78.0 39.0 45.0 15.0 70.0 63.0 65.0 75.0 68.0
Score 8.0 32.0 75.0 8.0 53.0 67.0 22.0 4.0 34.0 46.0
Score 91.0 38.0 48.0 85.0 11.0 93.0 96.0 17.0 80.0 13.0
Score 46.0 90.0 14.0 41.0 0.0 40.0 97.0 74.0 0.0 25.0
Score 26.0 14.0 85.0 92.0 29.0 63.0 77.0 94.0 80.0 8.0
Score 99.0 60.0 48.0 94.0 23.0 37.0 74.0 57.0 2.0 96.0
Score 51.0 99.0 89.0 67.0 69.0 5.0 91.0 6.0 97.0 97.0
Score 41.0 21.0 55.0 63.0 68.0 55.0 1.0 60.0 11.0 54.0
Score 35.0 18.0 65.0 78.0 96.0 79.0 3.0 22.0 80.0 44.0
Score 64.0 62.0 37.0 12.0 81.0 71.0 50.0 29.0 33.0 82.0
Score 27.0 24.0 4.0 29.0 86.0 36.0 11.0 47.0 77.0 5.0
Score 40.0 14.0 44.0 95.0 78.0 90.0 42.0 41.0 99.0 83.0
Score 10.0 93.0 37.0 48.0 87.0 27.0 79.0 15.0 94.0 57.0
Score 92.0 100.0 24.0 81.0 61.0 93.0 68.0 8.0 54.0 46.0
Score 55.0 79.0 17.0 88.0 96.0 83.0 88.0 99.0 49.0 21.0
Score 88.0 46.0 20.0 17.0 74.0 76.0 12.0 53.0 89.0 22.0
Score 85.0 42.0 26.0 26.0 87.0 69.0 49.0 68.0 57.0 49.0
Score 59.0 31.0 21.0 74.0 56.0 3.0 94.0 95.0 26.0 18.0
Score 100.0 75.0 59.0 39.0 64.0 54.0 7.0 30.0 34.0 38.0
Score 58.0 100.0 60.0 96.0 100.0 70.0 71.0 19.0 58.0 7.0
Score 13.0 66.0 94.0 82.0 92.0 87.0 51.0 51.0 80.0 13.0
Score 36.0 3.0 32.0 29.0 83.0 95.0 20.0 42.0 53.0 95.0
Average 48.7 53.9 48.2 51.7 52.1 51.8 52.8 53.9 56.0 50.4
Table 3: School results for ten big schools.

As before let’s just look at the averages for each school.

School Average
big01 48.7
big02 53.9
big03 48.2
big04 51.7
big05 52.1
big06 51.8
big07 52.8
big08 53.9
big09 56.0
big10 50.4
Table 4: Big School Results

Notice that, like table 2 the results are clustered about an average of around 50. Notice, though, that the numbers are not spread nearly as much.

Let’s put the two tables side-by-side for a better look

Scool Average School Average
sml01 32.1 big01 48.7
sml02 38.1 big02 53.9
sml03 56.6 big03 48.2
sml04 61.2 big04 51.7
sml05 40.3 big05 52.1
sml06 51.7 big06 51.8
sml07 54.3 big07 52.8
sml08 50.3 big08 53.9
sml09 62.8 big09 56.0
sml10 38.3 big10 50.4
Table 5: Averages for both small and big schools

The thing to notice is that the big schools show much less variability. In small schools, individual students who do very well or very poorly (we call them outliers) tend to have a large effect on the average. In larger schools, the increased number of results tends to “smooth out” the results; to make them less variable.

This is something that is well-known in mathematics. It  even has a name: The Law of Large Numbers. Simply put, in larger populations repeated experiments tend to cluster better about the expected result.

Now, this is where things get interesting. Recall that this is all about the fact that small schools get a built-in advantage due only to the fact that they are small. Let’s see what it looks like when all twenty schools are ranked from highest to lowest.

School Average
sml09 62.8
sml04 61.2
sml03 56.6
big09 56.0
sml07 54.3
big08 53.9
big02 53.9
big07 52.8
big05 52.1
big06 51.8
big04 51.7
sml06 51.7
big10 50.4
sml08 50.3
big01 48.7
big03 48.2
sml05 40.3
sml10 38.3
sml02 38.1
sml01 32.1
Table 6: All twenty schools ranked from highest to lowest.

Did you see what happened? The top schools were all small schools. They reached the top due to nothing other than the law of mall numbers working in their favour. Random variability–two or three bright students or an absence of  two or three weaker students had a profoundly positive effect on the school average.

Recall also I mentioned there would be a twist. Notice that while the highest ranking institutions were drawn from the pool of small schools, so, too were the lowest ranking ones, and for the same reason–namely the presence of a few weaker students or he absence of a few strong ones.

So, based on this little experiment it’s plain to see that when ranked this way, small schools will tend to come out on top simply because they are small and the fact that the law of large numbers is better able to work in their favour.

As for the small schools at the bottom, it happens too and it’s at best likely that these are rarely mentioned due to the presence of selection bias on behalf of whoever wishes to weave the numbers into a narrative that suits their own political ends. One wonders, though, how many small schools have been closed or otherwise penalized for nothing other than being the unfortunate victims of chance.

Closing Note: this is in no way intended to cast AIMS in any negative light. To the best of my knowledge neither they, nor the various Departments of Education nor the various school districts ever tried to spin the reports into any grandiose claims regarding big and small schools. The false claims I have heard have generally be made by private individuals, each with their own axes to grind.

As for my own conclusion: Ranking systems, regardless of the context, whether it be health care, law enforcement, customer care or, as is the case here, school-based student achievement, serve a useful purpose but be wary of the law of large numbers before making any sweeping generalizations.

Advertisements