The Armour Goes in Unexpected Places

It would have been in most respects a normal day for an online distance education teacher in the early nineties. I settled in to my spot in the studio and made sure everything was working. First the mikes—all OK. Next the Telewriter: I picked up the pen and wrote on the screen and then remotely loaded the first ‘slide’ for the day’s lesson. Again everything was fine. As always the first thing to do would be to greet the students by name and just chat for a few minutes. Besides ensuring that the audio and graphics capabilities were working it had the much more important function of getting the students to open up, to come out of their schools, defined as they were by the walls of the classroom, and now enter into the online one defined only by who was present that day.

Today I had a new student. I was a bit surprised as it was several months into the school year. I asked her name but she did not reply. Eventually another student at that school answered for her, telling me her name and letting me know she was shy.

Over the next few weeks I did my best to get my new student—let’s call her Angela—involved, but all to no avail. She would not respond when asked a question and would not ever write on the electronic whiteboard when asked to contribute to the day’s work. Her first written work assignment was comprised of mostly blank sheets and so, I decided it was time to contact the school. I called the principal and then learned the awful truth.

———-

In a previous job, around 14 years ago, my designation was Program Implementation Specialist and one of my initial tasks was to put together a team of online teachers who would lead the changeover from the distance education system used in my province since 1988—the one described in part above, and may be described in more detail here if you are interested. Together, the Program Development Specialist and I devised a recruitment strategy that involved an online application system that would be used to provide a short-list of candidates. Those candidates would then be interviewed by a panel of three and would be subject to a reference check. All components were scored and the scores were used to rank the potential candidates, who would then be seconded.

This system was used by me and my colleagues for seven years and provided me with a significant experience in selecting those would be well suited to online learning. Through constant use I came to anticipate the response to one particular question as it tended to give an almost instant measure of whether the interviewee was or was not a suitable candidate. The question? “What would be your response if you noticed that a particular student was not doing well in the course? That is, if you noticed that a student was not engaged, not submitting work on time or doing work that was of sub-par quality?” Typical answers included: putting on extra classes, creating tutorials, providing “worksheets” and maybe even involving disciplinary measures. None of those, however, were the one I sought. I wanted something else.

———-

Oftentimes the truth or the best course of action is not the one that seems obvious. Take my own academic discipline—physics—for example. There’s nothing commonsensical about the majority of what is typically found in the high school physics curriculum despite the protestations of inexperienced (or just plain ignorant) instructors who claim they can “make it easy.” Newton’s first law (objects tend to remain at rest or in constant motion unless acted on by an unbalanced force) is about as counter-intuitive as it gets. Objects remain at rest—no they don’t! Just YOU try sliding a book across a floor; it comes to a stop in no time! No! Newton’s first law is the product of sheer genius; a fantastic off-the-charts insight made by a most unusual individual. Seeing or maybe creating ‘friction’ as a new construct but one that merely presents itself as a new unbalanced force—pure brilliance!

Physics is not something that is not easily absorbed; something that is only understood after a skillfully-constructed instructional framework that involves bringing students right up against their existing world understanding, clearly pointing out the deficiencies and ensuring that the student acknowledges those deficiencies and then carefully rebuilding the worldview in a different way. Not simple at all and certainly not something that happens in a day.

And so it goes with everything. To do better work you have to work hard to get beyond the obvious and, as just pointed out, this involves going up against your “comfort zone” then breaking through it with a whole new worldview. This involves breaking common sense.

———-

Allied Bomber Command faced just such a situation in World War II.

Let me digress for a moment here. I am not one given to glorifying war. While I acknowledge that it is a reality and something that often cannot  be avoided I also want to point out that there is generally no “right” and “wrong” side but instead two opposing groups who have found themselves with no alternative but to act with extreme aggression. It is a reality. Ordinary people like you and I never wish to find ourselves in it but, alas, from time to time it happens and we are faced with no choice but to do what we must.  Under the extreme conditions faced by the various sides oftentimes comes the need to dig down deep and to utilize every and any opportunity that affect the balance of power. Frequently, then, wartime becomes a time of extreme innovation borne of necessity. I wish to consider one case here as it is illustrative of a point I wish to make and not for any other reason.

Bombers, with their heavy deadly loads, are slow lumbering beasts and, as such, are easy targets for fighters who desperately seek to prevent them from achieving their missions. In WW2 many that set out did not return but were instead shot down by the fighter planes they encountered along the way. Those that returned were typically bullet riddled but still able to limp back to base for repair and refitting.

One of the responses to this loss of planes was to install armour that would protect the aircraft from the projectiles from the fighters. Armour, though, is heavy and reduces the load capacity and thus the military effectiveness of the aircraft. The solution, therefore, is to place the armour only where it is absolutely necessary. Bomber command subsequently engaged in a constant, careful study of its in-service aircraft. Each time an aircraft would return from a mission it would be inspected and the location of bullet holes obtained in that flight would be recorded. Typical returning aircraft resembled the drawing below. Notice where the bullet holes are; namely on the wings, tail and fuselage. Based on that it would make sense to place the armour there since, after all, that’s where the hits were occurring, right?

A Lancaster Bomber after a run. The red dots indicate the position of bullet holes.
A Lancaster Bomber after a run. The red dots indicate the position of bullet holes.

Wrong. The reasoning is unsound; fundamentally flawed, in fact.

Fortunately so, too, thought the Allied Bomber Command, thanks to the insight of mathematician Abraham Wald. He assumed that the bullets were not specifically aimed at any one part of the aircraft. Aerial firefighting was much too chaotic an activity to allow for precision aiming. Fighter pilots instead aimed in the general direction of the aircraft and hoped that the bullets/cannon shells would have some negative effect. One would expect, therefore that in an ideal situation, the placement of bullet holes would be more-or-less uniform.

The placement wasn’t uniform, of course as you already noticed from the image. Wald, however went one step further by reasoning—correctly—that hits to vulnerable areas would result in downed aircraft, ones that would not make it back. Since the sample used in the study consisted of aircraft that made it back it would be logical to conclude that they tended NOT to have hits to the vulnerable areas.

Take another look at the diagram. Where are there very few bullet holes? The engine and forward cockpit. Of course! A relatively small number of hits to the engines would render them inoperable. Likewise, hits to the cockpit could result in casualties to the flight crew. In either case the plane would be lost.

Simply put, instead of looking for where the bullets were you should look for where they were not. Those are the parts that need armour, and not the bullet-riddled parts.

———-

So what does this have to do with eLearning? It turns out that in my previous career a significant part of my efforts were dedicated to the improvement of the quality of our instructional efforts. I approached this is various ways: reading about things done differently elsewhere, researching new devices and attendant methods, conferring with teachers and interviewing successful students. These tended, at first, to be my main starting points. Over time, though, I slowly moved away from all of these somewhat.

It started in a somewhat unexpected fashion. Each year I would address all of the intermediate-secondary student teachers at Memorial University in order to explain to them how the province’s distance education program worked. As part of the presentation I would those in the audience who has received part of their high school program from the program to identify themselves and would ask them to offer up their perspectives on the experience.

Of course, in all honesty, I was, in part, “selling” the program. I was part of that same system and certainly took great pride in it and in my contribution to it. While I was making it look like I was seeking an unbiased assessment I know—now—that in the initial stages I was really seeking affirmation; an ‘independent’ external source that validated the program as being worthwhile.

To my great surprise that’s not exactly what I got. Yes, many of the students were quite positive about the experience they’d had in the distance education program, but not all of them were. Numerous students indicated that they’d not found it great or that they much preferred the more traditional face-to-face approach.

The first few times this happened I responded by downplaying the responses, merely assuming that they were just the voices of the disgruntled few who had not enjoyed success probably through their own efforts or, more accurately, lack thereof. In time, though, I came around. Rather than dismissing those voices or, worse, glossing over what they’d said I began showing active interest in their points of view. I would not just let their comments sit unacknowledged; unchallenged. Instead, I slowly came around to a practice whereby I would probe deeper whenever I got the somewhat negative responses, attempting to determine just exactly had led to what I’d found.

It was enlightening, to say the least. Space does not permit a detailed exposition of what I found but, in general, here were a couple of items that were frequently encountered:

  • The choice to enrol in a particular course, which also happened to be a distance education offering, was not made by the student but, rather, by the parents or, even more frequently, the school administrator or the school district office.
  • The instructor had not made a concerted effort to reach out to the student but seemed, rather to either just teach to nobody in particular, seldom involving anyone in the class or, instead, appeared to play favourites.
  • Technical issues had resulted in significant ‘down time.

Now, lest you get the impression that this post is a mean-spirited barb at my former employer, let me assure you that nothing could be further from the truth. The pride I felt, and continue to feel in that program, is built on more than just emotion. It is, rather, something that is rooted in significant evidence that indicates its overall efficacy. The numbers don’t lie and they indicate that the students tend to do well. Just not all of them.

My point, rather, is to point out that in the later part of my career I found much more use in finding out why students did not find success than I did in identifying those factors that were associated with success.

Like Wald, I found it useful to consider the planes that did not return.

———-

As for that telling response to the question, “What would you do if a student is not having success in your course?”

The desired response: “I would find out what was wrong.” That’s a lesson I earned through long and often painful experience.

Never mind the extra classes, the tutorials and the varied approaches, just figure out why the student is not doing well and do what can be done.

———-

But there’s still ‘Angela,’ the student I found in my class, the one who unexpectedly dropped in and who was not finding any success. Yes, I did seek to get to the bottom of it all.

And I did.

I learned that she had just returned to her home community, after living away for several years. Her mom was a single parent but had found a new boyfriend so she’d moved away to be with him, taking her daughter with her. It became an abusive relationship and one night, in a drunken rage, the boyfriend had murdered Angela’s mom while she was present there in the apartment. She’d returned to her home community and was placed in foster care and that’s why she’d been dropped unexpectedly in my grade eleven physics class.

I tried as best I could to make things work for Angela. Unfortunately I did not succeed. I did not end up giving her a passing grade and she was not in my online physics class the following year. I do not know how she fared in life after that but do think of her often, especially when I need a good dose of humility. Sometimes, even with hard work, skill and insight you still cannot get the success you hope for. Yes, you generally do, with effort and teamwork, but not always.

Angela did not have a good experience in my Physics class. It continues to be a humbling truth.

Advertisements

Small Schools Rank Higher: It’s Built-In; Do the Math (& a twist)

From time to time you will see institutions ranked according to various criteria. Generally this is done with the intention of demonstrating how well each is performing. It’s not unusual to see this done with schools and here’s a claim that is often made, and substantiated by the numbers:

Small Schools Tend to Lead the Ranks

This is consistent. For years I saw it in my own region, reflected in the annual report card issued by the Halifax-Based Atlantic Institute for Market Studies (AIMS). Year after year, small schools led the provincial rankings. As a professional whose entire career was devoted to the betterment of small rural schools I wanted to be able to brag about this, to puff out my chest and say, “look, I told you that small schools were better for our children. It’s obvious that the extra care and attention they get on an individual basis, as well as the better socialization caused by the fact that everyone knows everyone else, is making a positive difference.”

I never did that, of course. It’s not because I don’t believe in small schools–I truly do. My silence on the matter was, to some degree due to the fact that at the time the studies were published I was a non-executive member of the Department of Education. As such I was not authorized to speak on its behalf. That was not the real reason though.

No, I knew that a far more powerful force was afoot; something that affected the results much more than did either good teaching, a supportive (small) ecosystem and the presence of many brilliant bay woman and men.

Although all of those are positive factors.

No the most powerful effect was something else, something related to straightforward mathematical behaviour, and if you’ll spare a few minutes of your attention I will explain.

Simply Put: Small schools have an advantage in these rankings that is due only to the fact that they are small.

And there is an unexpected twist too, one not often mentioned in the discussion of the reports.

Here goes!

Let’s simplify the situation and assume that the rankings are based on the outcome of one test only. Furthermore, let’s say that the result of that test, for any given student, is completely random; that is, any student who writes it will get a random grade between 0 and 100. In other words let’s act like there’s really no difference between the students at all of the schools. The small ones will still come out on top.

Let’s see what this would mean for ten small schools (we’ll arbitrarily name them sml01 to sml10), each one having only fifteen students in grade twelve and writing the test on which our report is based. The results for all of the students are tabulated below. In reality the table was produced using a random number generator in Microsoft Excel. You don’t need to read the table in detail. It’s just here so you know I’m not making the whole thing up!

School sml01 sml02 sml03 sml04 sml05 sml06 sml07 sml08 sml09 sml10
Score 15.0 14.0 45.0 55.0 53.0 70.0 55.0 100.0 79.0 56.0
Score 6.0 34.0 94.0 75.0 64.0 75.0 59.0 75.0 73.0 52.0
Score 15.0 83.0 65.0 30.0 84.0 46.0 64.0 10.0 35.0 20.0
Score 64.0 16.0 80.0 77.0 10.0 55.0 85.0 32.0 91.0 97.0
Score 29.0 28.0 96.0 98.0 6.0 67.0 51.0 74.0 69.0 9.0
Score 43.0 29.0 49.0 79.0 17.0 64.0 54.0 11.0 32.0 91.0
Score 31.0 49.0 62.0 33.0 0.0 92.0 35.0 59.0 91.0 45.0
Score 51.0 21.0 98.0 75.0 47.0 57.0 32.0 32.0 25.0 58.0
Score 45.0 27.0 18.0 6.0 24.0 31.0 84.0 5.0 89.0 2.0
Score 62.0 62.0 38.0 84.0 16.0 23.0 39.0 84.0 36.0 17.0
Score 9.0 0.0 6.0 67.0 53.0 99.0 54.0 23.0 97.0 15.0
Score 38.0 34.0 21.0 70.0 58.0 40.0 37.0 21.0 56.0 50.0
Score 21.0 55.0 51.0 97.0 92.0 40.0 48.0 100.0 76.0 4.0
Score 17.0 50.0 43.0 38.0 6.0 1.0 72.0 77.0 16.0 25.0
Score 35.0 69.0 83.0 34.0 75.0 16.0 46.0 51.0 77.0 33.0
Average 32.1 38.1 56.6 61.2 40.3 51.7 54.3 50.3 62.8 38.3

Table 1: School results for ten small schools.

That’s a huge pile of numbers and we are only looking at the results for the schools so lets just redo that table showing only the schools and the averages.

School Average
sml01 32.1
sml02 38.1
sml03 56.6
sml04 61.2
sml05 40.3
sml06 51.7
sml07 54.3
sml08 50.3
sml09 62.8
sml10 38.3
Table 2: Small School Results

Notice that the results show a fair bit of variability. They cluster around an average of 50  but some schools had averages in the thirties while others were up around 60.

Now let’s do it all over again, but this time let’s see what would happen in larger schools (named big01 to big10). For the small schools we assumed there were only 15 students per grade level but for the larger ones let’s assume that there are in fact 120 students in grade 12 writing the test.

The rather long table is below just so you know I’m not pulling the numbers out of my head. As was the case with the small school simulation it was done using a random number generator in Microsoft Excel and just pasted directly into WordPress. Scroll to the bottom of the table 🙂

School big01 big02 big03 big04 big05 big06 big07 big08 big09 big10
Score 67.0 100.0 82.0 25.0 27.0 100.0 10.0 69.0 22.0 96.0
Score 42.0 92.0 63.0 16.0 42.0 12.0 69.0 94.0 66.0 60.0
Score 100.0 27.0 42.0 59.0 88.0 79.0 83.0 49.0 27.0 96.0
Score 44.0 29.0 46.0 63.0 28.0 69.0 31.0 19.0 18.0 59.0
Score 16.0 33.0 66.0 81.0 11.0 21.0 76.0 67.0 70.0 85.0
Score 95.0 31.0 11.0 2.0 80.0 15.0 21.0 78.0 91.0 33.0
Score 32.0 32.0 38.0 34.0 44.0 61.0 55.0 0.0 89.0 64.0
Score 82.0 6.0 96.0 57.0 8.0 52.0 64.0 55.0 62.0 72.0
Score 100.0 48.0 45.0 10.0 49.0 93.0 89.0 72.0 87.0 72.0
Score 88.0 91.0 33.0 0.0 36.0 11.0 76.0 10.0 78.0 55.0
Score 54.0 95.0 41.0 68.0 92.0 75.0 54.0 75.0 20.0 52.0
Score 44.0 79.0 88.0 69.0 82.0 9.0 31.0 74.0 1.0 78.0
Score 71.0 36.0 2.0 51.0 58.0 2.0 17.0 68.0 29.0 36.0
Score 93.0 47.0 89.0 91.0 25.0 47.0 85.0 96.0 63.0 23.0
Score 9.0 79.0 33.0 41.0 68.0 19.0 74.0 81.0 57.0 47.0
Score 77.0 84.0 28.0 44.0 2.0 54.0 37.0 48.0 25.0 54.0
Score 20.0 74.0 33.0 57.0 15.0 65.0 85.0 59.0 21.0 80.0
Score 56.0 98.0 27.0 68.0 45.0 75.0 58.0 71.0 92.0 58.0
Score 7.0 70.0 83.0 74.0 26.0 52.0 71.0 40.0 75.0 87.0
Score 80.0 32.0 65.0 7.0 54.0 62.0 68.0 7.0 87.0 88.0
Score 65.0 12.0 68.0 22.0 5.0 26.0 36.0 92.0 79.0 40.0
Score 87.0 89.0 51.0 70.0 96.0 98.0 56.0 13.0 10.0 51.0
Score 52.0 71.0 13.0 86.0 88.0 54.0 11.0 20.0 26.0 18.0
Score 69.0 57.0 11.0 36.0 39.0 5.0 38.0 56.0 82.0 40.0
Score 95.0 54.0 54.0 77.0 52.0 74.0 100.0 82.0 35.0 7.0
Score 49.0 80.0 24.0 42.0 11.0 82.0 70.0 18.0 30.0 19.0
Score 46.0 26.0 3.0 56.0 54.0 50.0 2.0 9.0 26.0 47.0
Score 58.0 57.0 98.0 62.0 65.0 50.0 7.0 94.0 9.0 43.0
Score 86.0 86.0 32.0 81.0 63.0 49.0 60.0 61.0 93.0 5.0
Score 9.0 54.0 74.0 65.0 27.0 38.0 42.0 30.0 42.0 99.0
Score 41.0 37.0 30.0 70.0 77.0 86.0 58.0 48.0 53.0 99.0
Score 23.0 82.0 9.0 73.0 9.0 9.0 86.0 27.0 57.0 50.0
Score 52.0 97.0 91.0 90.0 58.0 11.0 56.0 16.0 53.0 89.0
Score 54.0 84.0 46.0 0.0 26.0 55.0 36.0 94.0 89.0 46.0
Score 94.0 75.0 32.0 16.0 77.0 9.0 87.0 21.0 58.0 59.0
Score 77.0 27.0 93.0 65.0 61.0 23.0 53.0 60.0 29.0 23.0
Score 41.0 26.0 34.0 21.0 24.0 57.0 34.0 78.0 99.0 90.0
Score 73.0 67.0 83.0 54.0 99.0 63.0 24.0 65.0 75.0 37.0
Score 55.0 76.0 30.0 85.0 92.0 57.0 31.0 69.0 82.0 43.0
Score 12.0 38.0 53.0 56.0 40.0 67.0 3.0 50.0 86.0 90.0
Score 48.0 89.0 86.0 77.0 80.0 83.0 92.0 38.0 67.0 0.0
Score 59.0 81.0 65.0 0.0 47.0 24.0 57.0 18.0 27.0 90.0
Score 32.0 72.0 46.0 54.0 92.0 54.0 41.0 99.0 0.0 87.0
Score 5.0 55.0 0.0 78.0 13.0 83.0 60.0 68.0 68.0 86.0
Score 0.0 0.0 91.0 66.0 38.0 22.0 2.0 82.0 32.0 12.0
Score 19.0 74.0 40.0 54.0 93.0 37.0 68.0 75.0 57.0 35.0
Score 13.0 81.0 36.0 39.0 50.0 3.0 44.0 19.0 100.0 16.0
Score 36.0 95.0 4.0 100.0 60.0 89.0 47.0 99.0 70.0 43.0
Score 29.0 46.0 12.0 92.0 35.0 28.0 17.0 74.0 38.0 85.0
Score 49.0 84.0 35.0 70.0 36.0 12.0 32.0 43.0 81.0 39.0
Score 87.0 32.0 89.0 71.0 11.0 0.0 93.0 51.0 10.0 39.0
Score 43.0 27.0 12.0 9.0 81.0 78.0 52.0 99.0 82.0 86.0
Score 51.0 41.0 50.0 73.0 83.0 65.0 51.0 44.0 89.0 5.0
Score 21.0 56.0 89.0 6.0 47.0 41.0 57.0 17.0 72.0 53.0
Score 12.0 39.0 51.0 18.0 96.0 75.0 23.0 39.0 75.0 39.0
Score 0.0 48.0 11.0 51.0 61.0 22.0 39.0 35.0 88.0 75.0
Score 33.0 53.0 23.0 68.0 88.0 69.0 48.0 40.0 19.0 100.0
Score 31.0 30.0 82.0 31.0 13.0 55.0 89.0 94.0 40.0 60.0
Score 90.0 5.0 19.0 26.0 68.0 60.0 77.0 63.0 51.0 6.0
Score 41.0 65.0 72.0 76.0 91.0 11.0 71.0 37.0 68.0 53.0
Score 24.0 80.0 70.0 73.0 61.0 4.0 79.0 59.0 37.0 73.0
Score 11.0 24.0 72.0 48.0 64.0 28.0 38.0 79.0 66.0 22.0
Score 22.0 13.0 14.0 83.0 2.0 21.0 95.0 100.0 55.0 55.0
Score 50.0 97.0 59.0 85.0 15.0 82.0 77.0 31.0 21.0 92.0
Score 81.0 9.0 45.0 56.0 16.0 55.0 66.0 69.0 79.0 78.0
Score 36.0 74.0 68.0 7.0 36.0 42.0 5.0 76.0 41.0 76.0
Score 30.0 35.0 68.0 59.0 92.0 50.0 9.0 50.0 98.0 97.0
Score 30.0 31.0 2.0 1.0 62.0 64.0 82.0 88.0 84.0 53.0
Score 4.0 46.0 55.0 54.0 61.0 42.0 81.0 77.0 25.0 27.0
Score 32.0 51.0 79.0 58.0 2.0 33.0 66.0 92.0 20.0 68.0
Score 70.0 76.0 52.0 24.0 2.0 21.0 6.0 98.0 63.0 37.0
Score 54.0 68.0 91.0 56.0 58.0 32.0 41.0 74.0 64.0 45.0
Score 37.0 48.0 29.0 42.0 4.0 93.0 10.0 29.0 97.0 40.0
Score 14.0 47.0 46.0 83.0 80.0 52.0 42.0 54.0 33.0 29.0
Score 15.0 2.0 100.0 12.0 9.0 84.0 52.0 53.0 53.0 6.0
Score 8.0 23.0 35.0 63.0 78.0 34.0 30.0 75.0 14.0 54.0
Score 16.0 90.0 13.0 80.0 32.0 29.0 99.0 21.0 34.0 80.0
Score 99.0 48.0 47.0 5.0 71.0 88.0 77.0 68.0 50.0 2.0
Score 45.0 15.0 18.0 38.0 49.0 8.0 90.0 13.0 71.0 33.0
Score 42.0 50.0 86.0 80.0 79.0 53.0 21.0 81.0 53.0 36.0
Score 16.0 14.0 51.0 14.0 19.0 97.0 50.0 49.0 8.0 2.0
Score 34.0 85.0 55.0 54.0 49.0 63.0 1.0 58.0 73.0 13.0
Score 8.0 98.0 9.0 7.0 70.0 78.0 41.0 18.0 94.0 74.0
Score 59.0 43.0 31.0 30.0 97.0 85.0 64.0 94.0 3.0 91.0
Score 29.0 34.0 6.0 17.0 43.0 78.0 67.0 17.0 50.0 34.0
Score 80.0 12.0 98.0 24.0 84.0 25.0 96.0 76.0 16.0 67.0
Score 87.0 89.0 11.0 86.0 5.0 39.0 83.0 98.0 27.0 13.0
Score 62.0 73.0 69.0 91.0 47.0 52.0 91.0 57.0 87.0 39.0
Score 22.0 64.0 86.0 64.0 10.0 88.0 6.0 62.0 91.0 26.0
Score 28.0 74.0 88.0 19.0 45.0 97.0 94.0 3.0 75.0 30.0
Score 27.0 11.0 11.0 55.0 39.0 30.0 39.0 54.0 99.0 86.0
Score 94.0 85.0 60.0 1.0 42.0 23.0 57.0 97.0 58.0 24.0
Score 78.0 7.0 30.0 94.0 26.0 75.0 100.0 11.0 99.0 11.0
Score 94.0 12.0 81.0 50.0 49.0 36.0 68.0 95.0 67.0 33.0
Score 5.0 9.0 39.0 23.0 31.0 29.0 23.0 22.0 57.0 46.0
Score 67.0 55.0 98.0 81.0 80.0 72.0 31.0 53.0 80.0 95.0
Score 68.0 57.0 17.0 34.0 26.0 38.0 46.0 55.0 74.0 24.0
Score 8.0 39.0 82.0 34.0 65.0 74.0 34.0 39.0 62.0 19.0
Score 83.0 97.0 14.0 84.0 71.0 66.0 62.0 13.0 8.0 82.0
Score 83.0 78.0 39.0 45.0 15.0 70.0 63.0 65.0 75.0 68.0
Score 8.0 32.0 75.0 8.0 53.0 67.0 22.0 4.0 34.0 46.0
Score 91.0 38.0 48.0 85.0 11.0 93.0 96.0 17.0 80.0 13.0
Score 46.0 90.0 14.0 41.0 0.0 40.0 97.0 74.0 0.0 25.0
Score 26.0 14.0 85.0 92.0 29.0 63.0 77.0 94.0 80.0 8.0
Score 99.0 60.0 48.0 94.0 23.0 37.0 74.0 57.0 2.0 96.0
Score 51.0 99.0 89.0 67.0 69.0 5.0 91.0 6.0 97.0 97.0
Score 41.0 21.0 55.0 63.0 68.0 55.0 1.0 60.0 11.0 54.0
Score 35.0 18.0 65.0 78.0 96.0 79.0 3.0 22.0 80.0 44.0
Score 64.0 62.0 37.0 12.0 81.0 71.0 50.0 29.0 33.0 82.0
Score 27.0 24.0 4.0 29.0 86.0 36.0 11.0 47.0 77.0 5.0
Score 40.0 14.0 44.0 95.0 78.0 90.0 42.0 41.0 99.0 83.0
Score 10.0 93.0 37.0 48.0 87.0 27.0 79.0 15.0 94.0 57.0
Score 92.0 100.0 24.0 81.0 61.0 93.0 68.0 8.0 54.0 46.0
Score 55.0 79.0 17.0 88.0 96.0 83.0 88.0 99.0 49.0 21.0
Score 88.0 46.0 20.0 17.0 74.0 76.0 12.0 53.0 89.0 22.0
Score 85.0 42.0 26.0 26.0 87.0 69.0 49.0 68.0 57.0 49.0
Score 59.0 31.0 21.0 74.0 56.0 3.0 94.0 95.0 26.0 18.0
Score 100.0 75.0 59.0 39.0 64.0 54.0 7.0 30.0 34.0 38.0
Score 58.0 100.0 60.0 96.0 100.0 70.0 71.0 19.0 58.0 7.0
Score 13.0 66.0 94.0 82.0 92.0 87.0 51.0 51.0 80.0 13.0
Score 36.0 3.0 32.0 29.0 83.0 95.0 20.0 42.0 53.0 95.0
Average 48.7 53.9 48.2 51.7 52.1 51.8 52.8 53.9 56.0 50.4
Table 3: School results for ten big schools.

As before let’s just look at the averages for each school.

School Average
big01 48.7
big02 53.9
big03 48.2
big04 51.7
big05 52.1
big06 51.8
big07 52.8
big08 53.9
big09 56.0
big10 50.4
Table 4: Big School Results

Notice that, like table 2 the results are clustered about an average of around 50. Notice, though, that the numbers are not spread nearly as much.

Let’s put the two tables side-by-side for a better look

Scool Average School Average
sml01 32.1 big01 48.7
sml02 38.1 big02 53.9
sml03 56.6 big03 48.2
sml04 61.2 big04 51.7
sml05 40.3 big05 52.1
sml06 51.7 big06 51.8
sml07 54.3 big07 52.8
sml08 50.3 big08 53.9
sml09 62.8 big09 56.0
sml10 38.3 big10 50.4
Table 5: Averages for both small and big schools

The thing to notice is that the big schools show much less variability. In small schools, individual students who do very well or very poorly (we call them outliers) tend to have a large effect on the average. In larger schools, the increased number of results tends to “smooth out” the results; to make them less variable.

This is something that is well-known in mathematics. It  even has a name: The Law of Large Numbers. Simply put, in larger populations repeated experiments tend to cluster better about the expected result.

Now, this is where things get interesting. Recall that this is all about the fact that small schools get a built-in advantage due only to the fact that they are small. Let’s see what it looks like when all twenty schools are ranked from highest to lowest.

School Average
sml09 62.8
sml04 61.2
sml03 56.6
big09 56.0
sml07 54.3
big08 53.9
big02 53.9
big07 52.8
big05 52.1
big06 51.8
big04 51.7
sml06 51.7
big10 50.4
sml08 50.3
big01 48.7
big03 48.2
sml05 40.3
sml10 38.3
sml02 38.1
sml01 32.1
Table 6: All twenty schools ranked from highest to lowest.

Did you see what happened? The top schools were all small schools. They reached the top due to nothing other than the law of mall numbers working in their favour. Random variability–two or three bright students or an absence of  two or three weaker students had a profoundly positive effect on the school average.

Recall also I mentioned there would be a twist. Notice that while the highest ranking institutions were drawn from the pool of small schools, so, too were the lowest ranking ones, and for the same reason–namely the presence of a few weaker students or he absence of a few strong ones.

So, based on this little experiment it’s plain to see that when ranked this way, small schools will tend to come out on top simply because they are small and the fact that the law of large numbers is better able to work in their favour.

As for the small schools at the bottom, it happens too and it’s at best likely that these are rarely mentioned due to the presence of selection bias on behalf of whoever wishes to weave the numbers into a narrative that suits their own political ends. One wonders, though, how many small schools have been closed or otherwise penalized for nothing other than being the unfortunate victims of chance.

Closing Note: this is in no way intended to cast AIMS in any negative light. To the best of my knowledge neither they, nor the various Departments of Education nor the various school districts ever tried to spin the reports into any grandiose claims regarding big and small schools. The false claims I have heard have generally be made by private individuals, each with their own axes to grind.

As for my own conclusion: Ranking systems, regardless of the context, whether it be health care, law enforcement, customer care or, as is the case here, school-based student achievement, serve a useful purpose but be wary of the law of large numbers before making any sweeping generalizations.

Structured Integration vs. Cost Savings

How many times do you see “cost saving” being touted as a reason for increased use of educational technology, and most especially distance education? Time and again you will see the adoption of new technology being explained away as cost savings. All you can really do, most of the time, is roll your eyes as you know, beyond doubt, that one of two things will happen. Either (1-not bad) the new technology will wind up costing somewhat more than budgeted—owing to the training costs and other unanticipated costs associated with the adoption and integration process or (2-BAD) it will eventually be abandoned and left to lie, mostly unused, right next to all of the other money wasters that have been purchased through the years.

This does not need to be the case. Properly done, new technologies can be more effective and cheaper; just not that much cheaper. Look around at the cellphones, fuel-injected engines, “green” heating systems and such that have made our lives that much better. The same can happen in our classrooms too but we need to take a much longer view of what comprises cost saving and just plain get over the fool’s quest for that elusive magic bullet.

Cost saving should not be NOT the slashing of departmental budgets and subsequent placement of course notes online just so deficits can be handled in the short term. (Although, admittedly, here in the real world that does have to happen from time to time regardless of how high-minded we would like to be.)That helps nobody as the result will only be a degradation of services, followed by corresponding loss in enrolment. Cost savings might be better framed as the deliberate employment of suitable technologies so that, over time, better outcomes can be achieved at lower cost.

Examples include:

  • Joining classes at separate campuses or schools using videoconference or, even better, a combination of videoconferencing and web conferencing such that smaller student cohorts can be aggregated. In those instances, though, care must be taken such that the host site or the instructor site does not become the “main” site with the remote ones getting the scraps from the educational table.
  • The replacement of non-interactive lectures with series of multimedia-based presentations, preferably with interactive components, such as embedded quizzes or simulations.
  • The gradual replacement of some media types with others but only after a piloting process which (a) shows the worth of the new technology and (b) refines the methodology before full deployment. For example, it may be feasible to replace the printed materials used in a course with online versions, perhaps multimedia or eBooks.

How often has it happened—a new device and its associated procedures shows up unannounced? Perhaps it’s a new set of chromebooks, maybe its clickers, a handheld computer algebra system or a new, shiny, computer numerical control (CNC) machine for the shop class. Whatever. In it comes and with it comes a feeling that you are expected, all of a sudden, to just change everything.

Before proceeding too far it needs to be said that the expectation that you need to change right away if often imagined. It’s been my experience that those responsible for high level decisions do tend to also have a healthy sense of what everyone is up against. After all, the funding that permits that sort of upgrade, itself takes years to put together. The problem is that the expectations that led to the upgrade are often not well understood by those who are expected to implement the change; there’s often a disconnect. Nonetheless, those on the front lines tend to be confronted by a somewhat intimidating set of equipment and feel a corresponding sense of stress on account of what they know needs to happen.

Of course that is just a bit silly. Change does not happen that way. Yes, we are all intelligent and capable of change but none of us is foolish enough to react to every new thing that comes our way, whether invited or not. The change and integration process happens in stages. Assuming that the technology is not another blind alley (and they do happen) it usually plays out something like this:

  1. Familiarization: You have to learn how the equipment works at the most basic level. What’s it for? What do the controls/menus do? What options do you have? In situations like this it’s good to have access to an expert. A demo followed by hands-on activities can be quite useful at this stage.
  2. Utilization: You have to become comfortable with using it. It’s not enough to know what each component does but you have to become adept in its use. Nobody wants to make clumsy or false moves in front of an audience so you need time to practice. If, for example, the device in question is a handheld computer algebra device then use it for your own purposes for a semester or so before even attempting to build lessons around it. If it’s an IWB then you need to take some time to engage in unstructured use—play—with the device in a non-threatening environment. Just close the classroom door and fly solo or, better still, gather a small posse of like-minded colleagues and have a collaborative session.
  3. Integration: Bit by bit you make the use of the technology a part of the natural routine. While you can bring it in all at once it’s much less stressful to layer its use in here and there. If, for example the device is an IWB, instead of ditching your existing lesson plans, try instead to catch the low hanging fruit; that is to redo some of the lessons than lend themselves best to an IWB approach. If it works well, try another and so on.
  4. Reorientation: In time you may find that the “new” equipment and associated methodology becomes your standard approach. That set of chromebooks that you used to despise may, in time, become treasured additions to your classroom; perhaps even indispensable. This will not happen overnight and the stages are likely measured best in semesters, maybe even years.
  5.   Evolution: With new standards come new horizons. You may find unexpected applications of the once-unfamiliar technology. Perhaps you even spot yet another—and for now unfamiliar—set of methodologies that bears promise.

Of course equipment will still arrive unexpectedly and instructors will, to some extent, have to sort it out as best they can. The best advice is to realize that regardless of what else happens the integration process will come in stages, so act accordingly.

ELTM14: Corporate History or Digital Amnesia?

Are you a fan of Pixar movies? I am (except for “Cars”). One of my favourites was “Finding Nemo” a story of a journey home. Along the way, the main character is accompanied by his friend “Dory” a friendly Paracanthurus Hepatus, whose primary character trait, it seems, is that of an extremely short memory.

Last August I retired from the k-12 public school system in my province. Along with personal belongings I left, not only with skills learned through long practice but, more importantly, with the only complete set of memories of the entire k-12 distance education program. Of course that’s not unusual. After all everyone retires at some point, and with all of them goes a piece of the overall history of the various organizations they have belonged to.

While that’s not necessarily a big problem, it can be, especially when you consider all of the decisions that have been made along the way. Each one received the proper amount of diligence and that has meant that, over time, a reasonably cogent set of guidelines and theory has been built up. In short, “oldsters” have a good idea of what to do and how to do it. They also have a good idea of what not to do. But, now it’s to no avail as they’re no longer around to lend a hand where they can.

So what? People move in, out of, and through organizations all the time and, on balance (a) the ability to intelligently match skills with jobs and (b) the spread of new and innovative ideas that results from this far outweighs the small losses that occur with the departure of a colleague. That said, this concern, which we can term “digital amnesia” still is something to be considered. Valuable employees possess not only the skills needed in the moment, but, more importantly, a clearer sense of purpose. This broader vision tends to keep the organization on the best track and, more importantly away from the small pitfalls and dead ends because, in all likelihood, they’ve experienced them before.

Perhaps, with that in mind, it is a useful suggestion for eLearning institutions to follow the lead of others and maybe establish a volunteer panel of advisers of all ages who can meet—virtually, of course—periodically and offer whatever wisdom and advice that may be needed at the time.

Next: technology–the rapture.