Never Mind The Data, Feel The Noise (or, seek the signal, young Jedi)

Everyone in education loves data.

This is the only time it is correct to use the word "Data" in the singular...

Or at least claims to. One sometimes wonders what would happen to the UK education system if a computer virus disabled every Excel spreadsheet overnight — h’mmm, perhaps someone should get in touch with those nice hacker people at Anonymous . . .

However, I digress. I wanted to share a recent epiphany that I’d had about data, particularly educational data. Perhaps it’s not much of an epiphany, but I’ve started so I’ll finish.

It came when I was listening to an interview on the evergreen The Jodcast (a podcast produced by the Jodrell Bank Radio Observatory). Dr Alan Duffy was talking about some of the new technologies that need to be invented in order to run the new massive Square Kilometre Array radio telescope (due to begin observing in 2018):

And then we have to deal with some of the data rates . . . essentially we recreate all of the information that exists on the internet today, and we do that every year without fail, it just keeps pouring off the instrument. And what you’re looking for is the proverbial needle in the haystack . . . how do you pick out the signal that you’re interested in from that amount of data?
The Jodcast, October 2014, 18:00 – 21:00 min approximately [emphasis added]

The realisation that hit me was: it isn’t the data that should be centre stage — it’s the signal that’s contained within that data. And that signal can be as hard to find as the proverbial needle in a haystack, even without data volumes that are multiples of the 2014 Internet.

A simple example from the history of science: Edwin Hubble’s famous graph from 1929 that was one of the first pieces of evidence that we exist in an expanding universe. The data are the difficult and painstaking measurements made by Hubble and his colleague Vesto Slipher that are plotted as small circles on the graph.


The signal is the line of best fit that makes sense of the data by suggesting a possible relationship between the variables. Now, as you can see, not all the points lie on, or even close, to the line of best fit. This is because of noise — random fluctuations that affect any measurement process. Because Hubble and Slipher were pushing the envelope of available technology at the time, their measurements were unavoidably ‘noisy’, but they were still able to extract a signal, and that signal has been both confirmed and honed over the years.

In my experience, when the dread phrase “let’s look at the data” is uttered in education, the “search for a signal” barely extends beyond simplistic numerical comparisons: increase=doubleplus good, decrease=doubledoubleplus ungood.

The way we use currently use data in schools reminds me of SF author William Gibson’s coining of the term cyberspace (way back in the pre-internet 1980s) as the

consensual hallucination experienced daily by billions of legimate operators . . . a graphic representation of data abstracted from the banks of every computer in the human system
— William Gibson, Neuromancer (1984)

In my opinion, almost the whole statistical shebang associated with UK education, from the precipitous data-mountains of the likes of RAISEOnline (TM) to the humblest tracking spreadsheet for a department of one, is actually nothing more than a ‘consensual hallucination’.

The numbers, levels and grades mean something because we say they mean something. And sometimes, it is true, they can tell a story.

Let’s say a student has variable test scores in one subject over a few months: does this tell us something about the child’s actual learning, or about possible inconsistencies in the department’s assessment regime, or about the child’s teachers?

My point is that WE DON’T KNOW without cross referencing other sources of information and using — wait for it — professional judgement.

I believe that the search for a signal should be central to any examination of data, and that this is best done with a human brain through the lens of professional experience. And, given the inevitability of noise and uncertainty in any measurement process, with a generous number of grains of statistical salt.

Weasel Words in Education Part 5: Rigour

A crack team of DfE boffins test the proposed new system for the management and oversight of the United Kingdom’s increasingly fissiparous school system.

Rigour, n.

1. The quality of being extremely thorough and careful.

2. severity or strictness.

3. (when pluralized) harsh and demanding conditions

In education (as in other walks of life) the word rigour is usually meant in sense (1) when applied to one’s own thinking or the thinking of one’s friends or allies: “I am being rigorous. However, you, sir, are merely pedantic.”

These days, sense (2) seems to require the insertion of a prefix, as in “The moderation of our controlled assessments was over-rigorous.”

Rigour is therefore a good thing, right?

However, in my opinion it seems to be used more and more as a talisman rather than as a genuine description.

Mr Gove told the Commons: “The new specifications are more challenging, more ambitious and more rigorous. That means more extended writing in subjects like English and history, more testing of advanced problem-solving skills in mathematics and science.”

The Independent, July 2013

I am not sure if Michael Gove* is using the word in sense (1) or sense (2) here. If he meant it in sense (2) then it is a rhetorical flourish to emphasise the idea that GCSEs will be more challenging. If he meant it in sense (1) then the promise of “extended writing [and] more testing” doesn’t tell me how the new exams will be more thorough and careful. This is not saying that the examination system does not need to be more thorough and careful, merely that “extended writing [and] more testing” won’t necessarily make it so.

Let me emphasise that I am not opposed to rigour. I like rigour and being rigorous, at least in sense (1). I would perhaps favour the words consistent and fair rather than use rigour in sense (2) in an educational context, but that’s a personal preference.

In short, I wish people would be more rigorous in their use of the word rigorous. You shouldn’t just use it because you think it sounds good. A is rigorous while B is not should mean more than I like A and dislike B.

And as a final thought, I strongly suspect that many of the people who are most keen to bemoan the lack of rigour in education would have to step out of the kitchen when push came to shove, as in this little vignette:

[I listened] to magazine columnist Fred Barnes . . . whine on and on about the sorry state of American education, blaming the teachers and their evil union for why students are doing so poorly. “These kids don’t even know what The Iliad and The Odyssey are!” he bellowed, as the other panellists nodded in admiration at Fred’s noble lament.

The next morning I called Fred Barnes at his Washington office. “Fred,” I said, “tell me what The Iliad and The Odyssey are.”

He started hemming and hawing. “Well, they’re … uh … you know … uh … okay, fine, you got me—I don’t know what they’re about. Happy now?”

No, not really. You’re one of the top TV pundits in America, seen every week on your own show and plenty of others. You gladly hawk your “wisdom” to hundreds of thousands of unsuspecting citizens, gleefully scorning others for their ignorance.

— Michael Moore, Stupid White Men (2001), p.58


* His successor Nicky Morgan look set to continue Gove’s use of the term.

Postscript: For the those (including myself) who are classically undereducated: The Iliad is an ancient Greek epic poem by Homer about the Trojan War. The Odyssey is another epic poem by Homer recounting the ten-year journey home from the Trojan War made by Odysseus, the king of Ithaca.

The Joy of Quotation Marks

A colleague of experimental psychologist Steven Pinker once joked that verbs were ‘his little friends’ as Pinker believed that the way they are used can give genuine insight into the hidden machinery of cognitive processes.

You know who my ‘little friends’ are? Punctuation marks. I think that they can often give the game away. Take this doozy:

The best secondary schools trusted the incoming ‘levels’ achieved by pupils in primary school as a starting point . . .
–OFSTED, Maintaining Curiosity in Science, November 2013, p.42

The writer asks schools to trust things called “‘levels'”, which the writer has deliberately placed in quotation marks. H’mmm, interesting. Now why would they choose to do that?

By my count, there are five reasons to use quotation marks:

1. Reported speech — this instance doesn’t seem to fit that usage.

2. When coining a new word or phrase — again, this usage is unlikely in this instance.

3. When referring to a word as a word — again, it doesn’t seem to be the intention here.

4. To indicate the title of a book or article — this is definitely not the case here.

By a process of elimination, this seems to leave only one plausible reason for the writer to choose to use quotation marks:

       5.   To imply that the quoted word or phrase is dubious.

So let’s be clear here: the writer is asking schools to trust things called “‘levels'” that he or she apparently considers dubious enough to wrap in ironic quotation marks.

In this paragraph, Ofsted are urging schools to trust what Ofsted themselves (going by their use of punctuation, at least) consider untrustworthy. What are they going to ask us to do next? Square the circle? Cut down the largest tree in the forest with a herring?

Now, where else have I seen ‘levels’ in quotation marks recently? Oh yes . . .

As part of our reforms to the national curriculum , the current system of ‘levels’ used to report children’s attainment and progress will be removed.  It will not be replaced.
–DfE, June 2013

Let me summarise: in June 2013, the DfE tells us that ‘levels’ are gone, but then in November 2013, Ofsted admonishes us for not taking ‘levels’ seriously enough.

Sigh. Education: does thy right hand know what thy left hand doeth? Ever?

As a teacher, my way forward is crystal clear: it’s time to get busy cutting down the largest tree in the forest. Now, where did I put that herring . . .

The Twelve Physics Pracs of Gove (Part Two)

A true-devoted pilgrim is not weary
To measure kingdoms with his feeble steps

–William Shakespeare, The Two Gentlemen of Verona


A picture [of reality]  . . .  is laid against reality like a measure  . . .   Only the end-points of the graduating lines actually touch the object that is to be measured  . . .   These correlations are, as it were, the feelers of the picture’s elements, with which the picture touches reality.

–Ludwig Wittgenstein, Tractatus-Logico-Philosophicus 2.141-2.1515


What they say of disc jockeys is also true of teachers: that someone, somewhere will remember some of your words forever; or, at least, for the duration of their lifetime. The downside is, of course, that you never know which of your words are going to be remembered. The wittily-crafted, near-Wildean aphorism pregnant with socratic wisdom — probably not. The unintentionally hilarious malapropism that makes you sound like a complete plonker — almost certainly.

To this day, I still remember Dr Prys’ sharp and appropriate response to a flippant comment (possibly from the callow 6th form me) about whether the scientific constants listed in the data book were truly trustworthy: “Look,” he said, “people have dedicated their whole lives to measuring just one of these numbers to one extra decimal place!” True devoted pilgrims indeed, mapping out the Universe step by tiny step, measurement by measurement.

I have written before on what I consider to be the huge importance of practical work in Physics education. Without hands-on experience of the hard work involved in the process of precise measurement, I do not believe that students can fully appreciate the magnificent achievement of the scientific enterprise: in essence, measurement is how scientific theories “touch” reality.

I am encouraged that parts of this view seem to be shared by the writers of the Subject Content guidance. (All hail our Govean apparatchik overlords!)

Of course, this has to be balanced with the acknowledgement that (as I understand it at least) teacher-assessed practical work will no longer count towards a student’s final exam grade. Many are concerned that this is actually a downgrading of the importance of practicals in Science and thus a backward step.

Sadly, they may turn out to be right: “We have to have this equipment for the practical/controlled assessment!” will no longer be a password for unlocking extra funding from recalcitrant SLTs (and from the exam budget too — double win!)

And, undoubtedly, some “teach-to-the-test” schools will quietly mothball their lab equipment (except for the showy stuff — like the telescope that no-one knows how to use — that they bring out for prospective pupil tours).

That would be sad, and although the DfE have, to be fair, nailed their pro-practical colours to the mast, we all know that the dreaded Law of Unintended Consequences may have the last laugh.

I would say it all depends on how the new A levels are actually put together. I will be attending some “launch events” in the near future. I will blog on whether I think we can expect an Apollo 11 or an Apollo 13 at that time.

In the meantime, I will be setting practicals galore as usual, as I’m old-fashioned enough to think that they give a lovely baroque feel to a scheme of work…

Look at me, I design coastlines, I got an award for Norway. Where’s the sense in that? None that I’ve been able to make out. I’ve been doing fiords all my life, for a fleeting moment they become fashionable and I get a major award. In this replacement Earth we’re building they’ve given me Africa to do, and of course, I’m doing it will all fjords again, because I happen to like them. And I’m old fashioned enough to think that they give a lovely baroque feel to a continent. And they tell me it’s not equatorial enough…
–Slartibartfast, from The Hitch-Hikers Guide to the Galaxy by Douglas Adams


Assessing Without Levels: the Lewis Carroll Perspective

An alternative look at assessment without levels…

As part of our reforms to the national curriculum , the current system of ‘levels’ used to report children’s attainment and progress will be removed.  It will not be replaced.

DfE, 2013


Bellman Map


He had bought a large map representing the sea,
Without the least vestige of land:
And the crew were much pleased when they found it to be
A map they could all understand.


“What’s the good of Mercator’s North Poles and Equators,
Tropics, Zones, and Meridian Lines?”
So the Bellman would cry: and the crew would reply
“They are merely conventional signs!


“Other maps are such shapes, with their islands and capes!
But we’ve got our brave Captain to thank:
(So the crew would protest) “that he’s bought us the best–
A perfect and absolute blank!”


— Lewis Carroll, The Hunting of the Snark

The Twelve Physics Pracs of Gove (Part One)

It’s not often that a DfE publication makes me feel like Kent Brockman, the newsreader from The Simpsons.

I’d like to remind them that as a trusted TV personality, I can be helpful in rounding up others to toil in their underground sugar caves.
Kent Brockman: “I, for one, welcome our new insect overlords.”

This feeling stems from reading the “Use of apparatus and techniques – physics” section from the DfE’s April 2014 Subject Content for AS and A level Biology, Chemistry, Physics and Psychology publication (p.23).

I had the rather novel feeling that it’s actually a sound list: and I, for one, welcome this intervention from our Govean-apparatchik overlords.

Why do I welcome this? Well, I feel that all too often we lose sight of the fact that, at its heart, Physics is, and must remain, a practical subject, the foundation of so much of the modern world.

Miroslav Holub’s poem “A Brief Reflection on Accuracy” paints a haunting and disturbing picture of what could be described as an entirely postmodernist, deconstructed and relativist (rather than relativistic) universe:

A certain soldier

    had to fire a cannon at six o’clock sharp every evening.

    Being a soldier he did so. When his accuracy was

    investigated he explained:

I go by

    the absolutely accurate chronometer in the window

    of the clockmaker down in the city.

   [ . . . ]

Oh, said the clockmaker,

    this is one of the most accurate instruments ever. Just imagine,

    for many years now a cannon has been fired at six o’clock sharp.

    And every day I look at this chronometer

    and always it shows exactly six.

[ . . . ]

So much for accuracy.
And fish move in the water, and from the skies
comes a rushing of wings while

Chronometers tick and cannons boom.

Without the grounding supplied by the art and science of measurement, I believe that we would all inhabit a castle-in-the-air universe as outlined above by Holub (whose experiences as an immunological research scientist are said to have influenced much of his poetry).

Is Holub’s nightmarish scenario even a remote possibility? Would we ever be in a world where “chronometers tick and cannons boom” but no-one actually checks the actual time by, say, looking out of the window to see if it’s daylight or not?

As with most nightmares, it’s probably closer than you think: “The sleep of reason brings forth monsters” as Goya suggested, and the steps that produce the monsters are often small, seemingly-harmless compromises of apparently little consequence.

One of my Y13 students, who has been attending a number of interviews for Physics courses, reports that some university departments have told him that “We spend a lot of the first year teaching students how to write formal laboratory reports as we find many of them have not learned how to do this during their A level courses.

Whaaa-aat? I nearly fell off my lab stool when Sam* told me this. In my opinion, that is unconscionable. “Oh, yeah,” Sam went on, “some of the students there said things like ‘Oh, our A level course content makes it unsuitable for practical teaching’.”

Opinions like that, if they genuinely reflect the views of the schoolteachers involved, are steps on the road to bringing forth monsters. Of course, it may not seem like a big deal to either the students or the teachers who are probably following what they see as a reasonable path of little resistance. But it is a big deal, it really is.

“And what did you say, Sam?” I asked.

“I said that we do a formal write up with a full analysis of experimental uncertainties every lesson.”

“Do we, Sam? Every lesson? Really?”
“Yeah, well,” said Sam with a smile, “I lied about that, didn’t I?”

“Exaggerated, Sam. I think you mean exaggerated.”

“Whatever you say, sir,” said Sam.

More on the 12 pracs of Gove in a later post..

* not his real name

National Curriculum Levels: worth keeping?

There is a tide in the affairs of men, or so opined Brutus in Julius Caesar.

Likewise, there is something like a tide in the edu-blogosphere, or at least a prevailing wind. And the prevailing wind right now seems to blowing against the idea of National Curriculum levels (try Joe, Daisy or Keven for their wiser, more coherent thoughts on this issue.)

But here’s the thing: I’ve always quite liked the idea of levels.

There. I’ve said it. Now I feel like Captain Rum in Blackadder:

Aaaaaar! All them other scurvy-bloggers be sayin’ be rid of NC levels! But I says…

Edmund: Look, there’s no need to panic. Someone in the crew will know how to steer this thing.

Rum: The crew, milord?

Edmund: Yes, the crew.

Rum: What crew?

Edmund: I was under the impression that it was common maritime practice for a ship to have a crew.

Rum: Opinion is divided on the subject.

Edmund: Oh, really?

Rum: Yahs. All the other captains say it is; I say it isn’t.

Blackadder II, Episode 3: Potato

Now this is not to say that some schools did some mighty strange things with levels and sub-levels. Like insisting that Key Stage 3 students should progress by two sub-levels per year. And woe betide any teacher that did not achieve this minimal standard of progression, or — horror of horrors! — reported that a student had made negative progress. How dare one cause even the minutest blip on our glorious straight lines on our graphs (drawn in Excel! with colour coding!) of student progression!

And so, for a quiet life, some rascally teachers may have looked at last year’s level, added two sub-levels to it, and entered that.

And, lo, it came to pass that everybody was happy: “Yea, we have numbers, and numbers are scientific. Gosh, some of us even use numbers and letters, which is beyond scientific: I mean, it’s more like advanced cognitive calculus of your learning soul, right? And Ofsted want to see progress over time. Which is shown by our graphs. In Excel. With colour coding. A glorious and undeviating straight line. For every single student. God, we are so good, aren’t we? Outstanding, even.”

That said, I am still in favour of keeping a form of assessment level. No, not the hyperformal “Oh-they’ve-got-to-sit-both-SATS-papers-in-order-to-get-a-reliable-level-and-sublevel” type of level.

What I have got in mind is an approach that was introduced to me many, many moons ago. It was called the CONTROL WORD approach to levels (ring any bells for anyone else yet?)

Level 3: DESCRIBES cause and effect using everyday language (e.g. “The wind blew the door shut”)

Level 4: Uses scientific TERMINOLOGY (e.g. “A force is a push or a pull.”)

Level 5: EXPLAINS cause and effect using scientific terminology (e.g. “The boat slowed because of the drag force of the water.”)

Level 6: Explain cause and effect using an ABSTRACT concept (e.g. “The bulb became dimmer because the resistance of the circuit increased.”)

Level 7: Uses a scientific MODEL to explain a phenomenon (e.g. “The wire has resistance because the freely moving electrons collide with the atoms of the wire and lose energy.”)

Level 8: Links PHENOMENA using a sophisticated model [or models] (e.g. “The atoms vibrate with greater amplitude at higher temperatures. This means that the freely moving electrons will collide more frequently with them. Thus the resistance of the wire increases with temperature.”)

The sublevels were allocated as follows:

(c) can do this with coaching or with highly structured prompts

(b) can usually do this with some prompting or coaching

(a) can do this relied on to do this independently

I’ve always secretly applied this assessment schema when asked for NC levels, and my rule-of-thumb-pulled-out-of-thin-air level has usually been at least comparable with “two-sodding-SATS-papers-to-bloody-well-mark-just-to-generate-one-number-and-one-stupid-letter approach”, or the T.S.S.P.T.B.W.M.J.T.G.O.N.A.O.S.L Approach, as an educational consultant might call it.

Anyhow, now my secret is out. Please feel free to pile on and criticise.

I shall sign off with what I think is an appropriate quotation from Wittgenstein:

My propositions are elucidatory in this way: he who understands me finally recognizes them as senseless, when he has climbed out through them, on them, over them. (He must so to speak throw away the ladder, after he has climbed up on it.)

He must surmount these propositions; then he sees the world rightly.

Tractatus Logico-Philosophicus, 6.54

Coe, Wilshaw and Ofsted

[T]he community suffers nothing very terrible if its cobblers are bad and become degenerate and pretentious; but if the Guardians of the laws and state, who alone have the opportunity to bring it good government and prosperity, become a mere sham, then clearly it is completely ruined.

— Plato. The Republic 421a (Penguin Classics) (Kindle Locations 2815-2817). Penguin Books Ltd. Kindle Edition.

I am not sure if I agree with Plato about cobblers and the community. As Benjamin Franklin once pointed out: “For want of a shoe…the kingdom was lost.”

However, I think his statement about the Guardians of the state stands. Equally so with the state-appointed guardians of education: Ofsted.

I think Professor Coe ( has done the education world a huge service by pointing out that the Ofsted king has no clothes. However, perhaps in the manner of scrupulous academics everywhere, Professor Coe might prefer a more nuanced “the king probably doesn’t have any clothes”.

Coe pointed out that there is no — repeat, no — valid research supporting the “Ofsted model” of classroom observation being either: (a) a reliable tool for assessing teaching quality or effectiveness when cross-referenced with other measures such as student learning gains; or (b) the observation-feedback process leading to an improvement in teaching quality. (See for the section of his talk on classroom observations and for a link to his slides.)

I don’t know about anyone else but I am staggered by this. As a working teacher who is just about maintaining a precarious foothold on the treacherous scree of middle management, I always thought my seniors and betters had reams of evidence supporting the stuff they were asking us to do. And if they didn’t, well probably their seniors and betters did.

To hear a respected academic say that classroom observation might be “the next Brain Gym” was shocking.

And the Ofsted response? “Tosh and nonsense,” said Sir Michael Wilshaw.  “I don’t know of any headteacher who doesn’t believe that classroom observation isn’t anything other than a help. The fact that we are an inspectorate and we do make judgements has made a huge amount of difference.” According to the TES (13/9/13 p.8):

He said that new figures released this week, showing a 9 percentage point rise in the proportion of schools judged to be good or outstanding, proved that the watchdog’s tougher inspection regime had “galvanised the system”

This is, to my mind, a textbook example of the logical fallacy known as petito principii or “circular reasoning”. The form of this particular logical fallacy is as follows:

Logical Form:

     Claim X assumes X is true.

     Claim X is therefore, true.

Bennett, Bo (2012-02-21). Logically Fallacious: The Ultimate Collection of Over 300 Logical Fallacies (p. 82). Kindle Edition.

Let’s see what Wilshaw said again.

  • The inspectorate’s judgements make a huge amount of [presumably positive] difference.
  • Ofsted judgements show that more schools (9 percentage points!) are good or outstanding.

…therefore Ofsted judgements make a huge amount of positive difference.

Now please note that this does not necessarily mean that the conclusion (that Ofsted helps schools improve) is false, but merely that the argument put forward by the Chief Inspector to support that conclusion is fallacious. And it hopefully goes without saying that a fallacious argument is by definition invalid and must be dropped immediately.

The character Chief Brody in the film Jaws once remarked that they needed “a bigger boat”. The Chief Inspector needs a better argument. And in view of the large amount of taxpayers’ money going to support Ofsted, that new argument should be supplied sooner rather than later. As Professor Coe remarked (somewhat plaintively) in his excellent talk: “Just one would be nice.”

H’mm. More rigour, anyone?

Bloom Schloom; or, some research what I have auto-didactically done

“Use Bloom’s taxonomy here for a quick win with Ofsted!!!!”
— AHT giving lesson observation preparation advice, sometime in 2013. [Note: the multiple exclamation points are to give the reader some indication of the evangelical zeal with which this advice was imparted.]


 “I can’t remember the last time I met a teacher who knew if Bloom’s taxonomy was ever criticised”   — Tom Bennett, Teacher Proof, Kindle Locations 191-192. Taylor and Francis. Kindle Edition 2013


I must confess, at the outset, that Bloom’s taxonomy has never sat right with me: for example, is it always the case that creating is always more cognitively demanding than (say) applying? So, creating a story about how the dog ate my homework is more cognitively demanding than applying Einstein’s time dilation equation?

I thought I was alone in my scepticism until I came across Tom Bennett’s comment (quoted above). However, even our very own Ben Goldacre-style enfant terrible of the educational research world doesn’t put the boot in to Bloom’s flipping taxonomy any further, although he does do a good job on knocking down de Bono’s coloured hats (as well as several other pieces of educational “wisdom” that he reveals to be not so wise  — read the book!)

And so I present my Bennett-ian take on Bloom’s taxonomy, the fruit of at least one afternoon of casual internet research — I’m sorry I’ll rephrase that, Ernie Wise-style, as “the research what I have auto-didactically done”. (And please note that I do not mean to imply in any way shape or form that Tom Bennett’s research for his book was as slapdash and cursory as mine…)

A taxonomy is, in its essence, nothing more or less than a system of sorting or classifying. To my mind, Bloom’s taxonomy has more of the feel of a folk taxonomy than a scientific taxonomy. For example, the folk classification of the large plants in a garden as trees, shrubs or flowers would be more than adequate for the average layperson. However, a botanist or gardener would probably require a more rigorous classification system using actual detailed scientific observations of the characteristics of the plants, rather than a handwaving “it’s a bit bushy” or “it looks tree-y”.

At first glance, it might seem obvious that creating is more cognitively demanding than (say) applying. But is it? How do we know? It seems to me that in order to accept this as a fact we need a sound model of how the human mind actually works. Is it always the case that creating always trumps applying? From my (admittedly limited) understanding of neuroscience, it seems to me that creating involves many brain processes and that these are currently poorly understood. The same can be said of the brain processes involved in applying. As a consequence, to place the two in any sort of cognitive hierarchy is, at best, premature.

The danger is that Bloom’s taxonomy is prejudicial in the sense that it assigns relative value to certain nebulously-defined types of thinking. As psychologist Robert J. Sternberg says, such theories “often do not have the clarity in epistemological status” that is required of a scientific taxonomy. So what we are left with is a folk taxonomy common among educational practitioners.

But how common? As Brenda Sugrue notes, even fans of Bloom’s taxonomy do not always agree on the level of a given learning objective: “it might be classified into either of the two lowest levels [ . . . ] or into any of the four highest levels [ . . . ] by different designers.” Sugrue argues that Bloom’s taxonomy:

was developed before we understood the cognitive processes involved in learning and performance. The categories or “levels” of Bloom’s taxonomy … are not supported by any research on learning. The only distinction that is supported by research is the distinction between declarative/conceptual knowledge … and procedural knowledge (which enables application or task performance).

It might seem, therefore, that possibly Bloom’s taxonomy is not even a folk taxonomy within the educational community, but rather it is simply a taxonomy of personal preference with regard to educational objectives.

David Morrison-Love makes the point that “the contribution made by Bloom’s Taxonomy cannot be underestimated, as a communication system derived from classifying different types of exam questions”; but goes on to say that  he does “not view the elements in Bloom’s Taxonomy as successive levels, but simply a collection of equally important intellectual processes I wish to promote and develop in learners; the challenge of which I control.”

Many of the authors cited propose alternative systems to replace Bloom’s taxonomy. At the moment, I am not sure whether any of these are worth considering.

However, the point of this blog post is to warn you that if that ubiquitous multicoloured triangle is flashed without a caveat on to a training screen near you, it could be an indication that the presenter has not done his or her homework, and that his or her assurances that what they say is based on  what ” research shows” may not be as rock solid as they might appear.


O for a draught of vintage! (Or: Bring back POAE!)

O for a draught of vintage! that hath been

Cool’d a long age in the deep-delvèd earth

— John Keats, Ode to a Nightingale

The Northfarthing barley was so fine that the beer of 1420 was long remembered and became a byword. Indeed a generation later one might hear an old gaffer in an inn, after a good pint of well-earned ale, put down his mug with a sigh: “Ah! that was proper fourteen-twenty, that was!”

— J. R. R Tolkein, The Grey Havens, from The Lord of the Rings

I don’t know about anybody else, but I could do with a draught of the vintage good stuff right about now. I am that old gaffer in the pub muttering: “They should being back POAE, they really should.”

In all probability, only Science teachers of a certain generation (translation: old farts like me) will recognise the acronym P.O.A.E.

For the youthful pups who now seem to comprise the majority of the UK’s teaching workforce, it stands for “Planning, Evaluation, Observing and Evaluating”, the “strands” (dread word!) by which we used to mark practical skills in the good old days of yore, when the world was yet young.

And truth be told, they weren’t all that good. It is only in comparison with more modern iterations that they achieve their near-mythic ‘fourteen-twenty’ status.

One of the jobs I have been studiously avoiding over the summer holidays is to mark a portfolio of Y10 students’ controlled assessment practical work. I am dreading it. The reason is, I have to use the worst mark scheme every developed in the entire history of humankind. Or before. Or, applying a rigorous Bayesian statistical analysis of relevant probabilities, since.

Accuse me of hysterical hyperbole if you will, but take my word for it: this mark scheme is a turkey that out-turkeys all the Christmas lunches served over the past two millennia.

Let me explain. What is the purpose of marking students coursework or controlled assessment? Wearing our summative, assessment-of-learning hats for a moment, the essence of marking in this context is to generate a number that indicates a student’s relative performance. Ideally, another professional marking the same student’s work would generate a similar number.

Using the old-style POAE scheme, I would have to assess a student’s work against 25 hierarchical criteria which would give a “best fit” number out of a maximum of 30 marks. (Boy, this sure is a fun post, isn’t it?) From memory, moderators would tolerate a disagreement of plus or minus 3 marks before adjustment.

Using the modern, rubbish mark scheme, I have to assess a student’s work against, by my count, 67 hierarchical criteria which give a “best fit” number out of a maximum of 64 marks. This takes a while, as I challenge anyone to memorise or internalise the mark scheme.

And the end result: is a mark out of 64 ‘better’ than a mark out of 30? Does it allow a finer discrimination between the performance of students?

In theory: perhaps. In practice: no. It is just another example of assessment-itis:-itis” being the most appropriate suffix in this case as the entire system of assessment is, indeed, inflamed. More is, in fact, less.

As an example, under the old POAE-scheme, the P for Planning strand (dread word!) had 7 criteria and a maximum 8 marks. Using the new mark scheme, I mark the same set of skills which are now labelled as S for Strategy (“Mategy, Categy, Sategy”) and include two individual sub-strands (even more dread words!) with a total of 21 marking criteria and a maximum 16 marks. And . . . it doesn’t tell the student or the teacher anything that the older scheme did not.

It is, in my opinion, a badly-designed exercise in futility which provides no useful guidance or feedback for either student or teacher. Let it be sent forthwith to whatever corner of limbo that clapped-out assessment formats go to die. A curse upon it, and . . .

Sings to the tune of “My Bonnie Lies Over the Ocean”:

Bring back, bring back, O bring back my P-O-A-E, A-E!

Bring back, O bring back my P-O-A-E to me!