Saturday, November 16, 2013

the AEU and ACER evaluations of Direct Instruction

According to Mike Williss, an Australian Education Union (AEU) Research Officer, Direct Instruction threatens teacher autonomy, student engagement, creativity, curiosity and a socially critical curriculum.("Will you be directly instructed how to teach?", AEU Journal, November 2013, p. 16).

This correlates with the memes currently listed 4 ("real learning" claims), 5 (creativity) and 6 (philosophy) in my analysis (DI_indigenous_memes) of the factors that strongly influence the perception of the opponents of Direct Instruction.

Here are some quotes from the Williss article, which is accompanied by a Simon Kneebone cartoon of a teacher being zapped in a Skinner box for deviating from a Direct Instruction script.
There are considerable "highs" in our job as teachers and they are mainly associated with those occasions when we experience autonomy, when we are in control of what we do and are doing it because we want to, not because we have to.

Pretty much the same thing serves as the basis for student engagement with their learning.

So how would we feel if we were given a prepared lesson script, were told only to say those things on the running sheet, to only engage in the activities that it stipulated and in the required sequence?

At its worst, that is just what the US-inspired Direct Instruction approach to teaching means...

So in comes learning for rats, classrooms as Skinner boxes, and out goes creativity, curiosity and - God forbid! - a socially critical curriculum.
This poorly argued case for teacher freedom and student engagement fails to address the dialectic between freedom and necessity, or, rights and responsibilities. This argument would appeal more to an irresponsible teacher ("I do what I want to do" is not the same thing as socially responsible autonomy) than a teacher who is prepared to explore suggestions as to what might improve the learning of disadvantaged students. You can't be meaningfully creative or socially critical unless you have a strong knowledge base and are prepared to actually look and examine deeply the alternatives that go against your keenly felt ("ideological") world view. Like it or not behaviourist principles (that desirable behaviour when rewarded is often repeated) are used extensively by every successful teacher and parent.

Speaking personally, I have experimented with a wide variety of teaching methods and have found that Direct Instruction is the most effective for engaging students from severely disadvantaged, remote indigenous backgrounds. The simple reason for this is that students can be successful with it, they learn, and success is essential for engagement. My autonomy, creativity, curiosity and ability to be socially critical of commentators such as Williss remains intact.

The most important thing is whether Direct Instruction actually works to improve the learning outcomes of disadvantaged students. The history is that it did work in Project Follow Through, a decade long study in the USA in the 1970s. See Engelmann's For Readers Not Familiar With Project Follow Through. But what is the current state of the evidence in the Cape York trial?

This aspect is falsely addressed in one paragraph of the Williss article:
...the percentage of students in Pearson's schools at or above NAPLAN national benchmarks in all areas tested was substantially below not only the national percentage, but also the percentage for "indigenous Queensland students from remote and very remote areas"
Note the meaningless quotation marks which I have faithfully reproduced from the original article, the source of which is not referenced by this socially critical research officer.

Are the students in the schools in Noel Pearson's led Cape York Aboriginal Australian Academy (CYAAA), Coen, Hopevale and Aurukun, performing substantially below other indigenous Queensland students in remote and very remote locations? No, they are not. Williss does not even reveal the source for his incorrect allegation.

The Australian Council for Educational Research has prepared an independent analysis at the request of the Department of Education Training and Employment Queensland titled Evaluation of the Cape York Aboriginal Australian Academy Initiative, June 2013 (pdf, 93pp).

The findings of that report are disappointing from my perspective but they certainly don't support Williss's assertion. The ACER report has a quantitative and a qualitative assessment. The quantitative data informed assessment finds neither in favour nor against the CYAAA initiative. This is because irregular attendance rates by students often exceed 20% and hence the collected data (NAPLAN and other copious data much of it native to the DI programme itself) is ruled out for a high stakes programme such as this. But it is also fair to say that the data that was collected for attending students does not show a remarkable improvement of the type reported by Zig Engelmann and his supporters in other situations. This is disappointing.

On the other hand, the more anecdotal qualitative data does show strong support for the programme particularly from the teachers delivering it. Here is a quote from one of the longer term teachers:
Personally, having seen what it was like before and to see it now with the new structure, it blows me away. To see the kids reading, they start younger, and to see the Year 1s and 2s reading, it is wonderful. The older kids could not do that. To get them this far is amazing. (p. 29)
What can we conclude at this point? One of the core principles of the Direct Instruction philosophy is that anecdotal evaluations on their own are not good enough. Every good teacher can report a warm and fuzzy feeling emanating from some of their classes. But more than this is required to justify the implementation of a particular programme across a wider scale.

The teachers union article is ideologically driven from a stereotypical "left" position. The teacher's union is more interested in a fuzzy autonomy for its members than conducting real research into what works best for disadvantaged students. Nevertheless, the far more objective ACER report shows us that more research needs to be done to find the best answer to assisting the most disadvantaged students. The ACER report should be studied in more detail as part of an ongoing evaluation by those genuinely interested in helping the most disadvantaged students in Australia.


Mike said...

G’day Bill,

I still do check your blog occasionally and wasn’t surprised to see you had taken issue with my DI article.

Of course, it wasn’t balanced. I said “At its worst…” but not “At its best…”.

I didn’t point out that a teacher needs to play the piano with all ten fingers, and that a form of direct instruction is employed by most of us as part of that multifaceted approach without it being formally presented as DI.

What I objected to was the scripted and controlled curriculum delivery that is seen “at its worst” in material like this: . You will notice the wonderful advice given to meet the individual learning needs of students with disabilities!

The quote that I didn’t source was from the Business Model prepared for the CYAAA. That referred to 2009 NAPLAN results. I know claims have been made for improvements through DI since then, but there is no hard data. If you look at the ACER Report (p. 26), you will see that the number of NAPLAN tested areas which had more than 20% missing data doubled between 2009 and 2012 and that the lack of data made comparisons between CYAAA schools and others impossible. In the absence of data (Hattie) I would say that the case for improvements on NAPLAN since the 2009 statement made by the Business Plan does not exist, and that the Business Plan statement still stands.

Have you read John McCollow’s article in the Australian Journal of Indigenous Education on the CYAAA? I have my own hard copy, but I suspect it can only be accessed by payment, eg here:;dn=099092697165983;res=IELIND

John’s article is much more balanced, and I won’t put that in inverted commas. I’d recommend it. You will find some observations to support your view of DI, but they are not data – they are qualitative reflections of the type that you and Langer ridiculed when Hill wrote about the overthrow of the “Gang of Four” (“Marxism by the seat of his pants”, wasn’t it?). I repeat my point though, that the missing NAPLAN data does not help the advocacy of DI and indicates continuing problems with attendance and engagement.

The US study Project Follow Through cannot be cited uncritically. It is a contested study. Please read Richard Allington’s article here:

Lastly, while I know we haven’t spoken for a while and that you repudiate the – what is it? – “stereotypically ‘left’ position” that you once so fervently embraced and that I fondly remember you for, your sarcastic reference to “this socially critical research officer” and your implication that I am not “genuinely interested in helping the most disadvantaged students in Australia” is a bit low.


Mike said...

Bill, I forgot to add this article by Allan Luke: .


Mike said...

Bill, Sorry, but that was the link at the bottom of an article in the Australian Educator. However, it goes to a different study. The article I'd like you to look at is here:


Bill Kerr said...

hi Mike,

para : "the Business Model prepared for the CYAAA"

You still haven't provided a proper reference to this source. From clues now provided I can now see why it would be embarrassing for you to do that given the way you used it in your article.

The correct Allan Luke reference is On explicit and direct instruction . Important reference.

I haven't read the John McCollow article. I agree with your general point here that anecodotal evidence is not enough, as clearly stated in my original article.

I'm already familiar with Richard Allington's work and that new reference is useful. Everything about DI is contested, that is a focus of my research.(DI_indigenous_memes)

Your "at its worst" argument and subsequent elaboration is not internally coherent with respect to any meaningful purpose of, or, the flow of the argument in your original article. Either Noel Pearson has good or bad ideas for helping disadvantaged remote indigenous students. Your article argued they were bad. My position is that they are probably good but also need further critical study and refinement.

Bill Kerr said...

Mike Williss wrote above:

"The US study Project Follow Through cannot be cited uncritically. It is a contested study. Please read Richard Allington’s article here:"

That article by Richard Allington is comprehensively replied to by Zig Engelmann here: Allington leveled serious allegations against Direct Instruction

Shaun Killian said...

The scripted delivery can be altered once a teacher is aware of what the script is trying to achieve by being worded a certain way.

Plus, John Hattie's review of research is quite pertinent.