Categories
Education Research

July 2015 round of EEF projects

eefEvery now and then the Education Endowment Fund releases a series of reports. It’s interesting to see the sheer difference in media attention they create, and also the buzz on social media. I collated all the studies in a Google spreadsheet:

I tabulated the name of the project, the project lead, the money involved (to me it is unclear if this includes the cost of evaluation cost), whether the project was completed or in progress (at time of writing), the evaluator, type of project, number of schools, and I added an indicator of whether there were signs of ‘significance testing’. In this post I want to summarise the recently released reports. I make no claim to cover all aspects of the studies.

Philosophy for Children (EEF page)
This was an effectiveness trial with 40 schools which evaluated Philosophy for Children (P4C), an “approach to teaching in which students participate in group dialogues focused on philosophical issues.” A key conclusion in the report was that there was a positive impact on KS2 attainment, with “2 months progress”. This project caused the most discussion, mainly because of some crucial aspects in the design. There was a guest post by Inglis, and this post.

Word and World Reading Programme (EEF page)
This study was a pilot study of Core Knowledge (E.D. Hirsch inspired) materials from The Curriculum Centre. I was surprised the blogosphere did not really pick up on this study. A suspicious mind might think this might be because the results of this ‘Core Knowledge’ programme were quite underwhelming, and this does not fit the ‘knowledge’ preferences. But of course, that is just as suggestive 🙂 I will write a separate post on this.

Affordable Individual and Small Group Tuition: Primary (EEF page)
Key conclusion that stands out: “Due to the study’s design and problems recruiting schools to receive tuition or participate in the evaluation, this evaluation has not provided a secure estimate of the impact of the project on pupil outcomes.” and also “Participating pupils made slightly less progress in both English and mathematics than those in the matched comparison group. However, this finding was not statistically significant, meaning that it could have occurred by chance.”. Staff members were positive. Recommendations are to improve.

Affordable Individual and Small Group Tuition: Secondary (EEF page)
“Due to the limitations of the study design and the absence of a high-quality comparison group, this evaluation has not provided a secure estimate of the impact of the project on academic outcomes.” and “Participating pupils achieved slightly higher mathematics GCSE scores than pupils in the comparison group, and lower English GCSE scores than pupils in the comparison group. However, it is not possible to attribute either change to the tuition provided.” Staff members are positive.

Graduate Coaching Programme (EEF page)
This trial showed a positive effect with moderate security.”The programme had a positive impact on pupils’ attainment in reading, spelling and grammar, equivalent to approximately five additional months’ progress. The evaluation did not seek to prove that the approach would work in all schools, but did identify strong evidence of promise.” and . The cost was quite high, and the delivery was very varied.

Peer Tutoring in Secondary Schools (EEF page)
This study concluded “This evaluation does not provide any evidence that the Paired Reading programme had an impact on overall reading ability, sentence completion and passage comprehension of participating pupils. “. The security is high and cost relatively low.

Shared Maths (EEF page)
This evaluation (of a 750k+ project) does not provide any evidence that the Durham Shared Maths programme had an impact on attainment in maths, when used with Year 5 and 3 pupils.”, the evidence strength was high and the cost low.

Talk for Writing (EEF page)
This project is “an approach to teaching writing that encompasses a three-stage pedagogy”. Teachers were enthusiastic about the implementation and this went quite smoothly. There was mixed evidence, although teachers reported it had an impact (this seems a theme, teachers thinking something has impact but the evidence not being there?).

Some observations from these studies
What strikes me in most of these studies is that:

  • Most studies report quite small or no effects.
  • Most studies report that staff are positive about the interventions, which seems to suggest that effectiveness and teachers’ perception are only a little bit related.
  • Effects are often worded positively even if these are small or non-significant (with a few reports by one evaluator even making a case against Null Hypothesis Significance Testing which I understand but find strange given the majority of reports).
  • Some reports mention ‘redeeming factors’ for non-effects, for example low costs. Like the Maths Masters study it seems that ‘low cost’ automatically makes an intervention worthwhile, even when no or very small effects.
  • Pilots mainly concluded that (i) yes, the approach was feasible, (ii) mixed results, (iii) interventions needed to be developed for a full trial.
  • There are many ‘arguments’ for further study along the lines of “more time is needed” or “larger samples are needed”, even when initial studies have spent significant amounts of money and had decent samples

Why is this notable? Well, for me mainly because EEF reports have been proposed to tell us ‘what works’. I would be the first to acknowledge that we need a range of qualitative and quantitative research, and that means, in my book, that there *should* be space for smaller scale studies as well. However, this does not seem to be the premise of most of the studies conducted. If £2.4 million is spent on 8 projects I would hope that the conclusions would be a bit more informative than ‘we should try more and harder’. I think it would be good if the reports report *only* on the results and do not make recommendations.

 

Categories
Education Research

Predatory journals

More and more I’m being confronted with questions about journal publications. I devote some words to it in a session for our MSc programme in the module ‘Understanding Education Research’ and recently, in a panel discussion at our local PGR conference, there were questions about how to judge a journal’s reputation. Note that in answering this question I certainly don’t want be a ‘snob’ i.e. that only the conventional and traditional publication methods suffice. Actually, developments on blogging and Open Access are positive changes, in my view. Unfortunately there also is a darker side to all of this:

One place where I always look first when it comes to ‘vanity press’ and predatory journals is Beall’s List, which is “a list of questionable, scholarly open-access publishers.”. What I like about this list is that they are rather sensible about how to use the list: “We recommend that scholars read the available reviews, assessments and descriptions provided here, and then decide for themselves whether they want to submit articles, serve as editors or on editorial boards.”. The list of criteria for determining predatory open access journals is clear as well. One thing you can do is use the search function to see if a journal or publisher gets a mention. This is exactly what I did recently with some high profile research. I was surprised to find out articles were indeed published in such journals.

The first example is this high profile article mentioned in the Times Educational Supplement. It references a press release from Mitra’s university:  

 
The journal title did not ring a bell so I checked Beall’s list, and yes the journal and publisher are mentioned in this article on the list. Just a quick glance, also the comments, should make most scholars think twice to publish in here, certainly if it is ‘groundbreaking’ stuff. This is not to say that articles per se are bad (although methodologically there is much to criticisise as well, maybe later, although this blog does a good job at concisely flagging up some issues) but I am worried that high profile professors are publishing in journals like these (assuming it was done with the authors’ agreement, predatory journals sometimes just steal content to bump up their reputation). In the case of this person it has happened before, in 2012, when the ‘Center of Promoting Ideas’ (this name would be enough for me to not want to appear in their publications) published this article in a journal, which is also on Beal’s list. It is poignant that an Icelandic scholar really got into problems because of this. Some other examples: this article, CIR world also features on Beall’s list (Council for Innovative Research, again a name which raises suspicion by itself).

  

These publications serve as examples that even high end professors could fall victim of predatory journals. I do not mean that in a judgemental way; it shows that more education on the world of predatory journals is needed. Although I must admit, there might be some naivety at play here, experienced scholars should know ‘positive reviews only’, ‘dubious publishing fees’ and ‘unrealistic publication turnovers’ are very suspicious. Early Career Researchers often are targets of predatory journals and it therefore is important to be aware of this ‘dark side’ of Open Access publishing. Beal’s list covers these but recently there also are more and more ‘non open access’ journals that might be a bit dubious as well. In many cases it’s quite a challenge to judge the trustworthiness of publications. Certainly if in social sciences we would want to go away from the hegemony of the five big publishers, there is a lot to be gained in general skills to judge literature. Now, everyone has their own judgements to make when it comes where they want to publish, but I would be very concerned publishing in any journal (and for any publisher) on Beall’s list.

Categories
Education Research

Mindset #1 – measures

It’s fair to say I’m not a regular blogger. I always feel as if polishing lengthy posts is a waste of time. However, when I look at all the zillions of tweets I manage to cram out I feel that elaborating on some of the concise tweets I put out would sometimes be a good thing. So here goes; let’s see if I can get a twoweekly or so blog out. I’ll start with one on mindset, and specifically about measurement. I do not contend I can do any better than the (up until now) four fabulous posts on Slate Star Codex but will try to add some thoughts about measurement.

The first thoughts were based on the following tweet:

I think it refers to this study. (Claro & Paunesku, 2014). One of the authors is the is the person on Slate Star Codex’ blog. On the measurement of mindset in students it states the following:

mindset_chile

This seems to me a rather concise scale (in reports these two questions are even referred to as ‘questionnaire’). The Education Endowment Fund also recently published a report on mindset.( Education Endowment Fund, 2015). Disregarding, the results for now (maybe a later post) the following questions were asked to determine mindset. The methodology section states:

mindset3

So this gives three items. The same amount which is in this sample ‘mindset meter’ at http://survey.perts.net/take/toi. Mindset Interventions Are A Scalable Treatment For Academic Underachievement also gives more information:

“we assessed this belief using two items: “You can learn new things, but you can’t really change your basic intelligence” and “You have a certain amount of intelligence and you really can’t do much to change it” (α = .84; see Blackwell et al., 2007).”

(Paunesku et al., 2015, p. 4)

So two items again. The Blackwell article is here; this article explored “the role of implicit theories of intelligence in adolescents’ mathematics achievement” by using SEM. The 6 point Likert scale is consistent; this article still uses the 6 items, which apparently were reduced to three and later even two. This feels like quite a limited amount to base a construct on? Of course it’s not necessarily wrong but the warnings in this paper by Eisinga et al. (2013) are there for a reason. I also wonder about the wording (knowing that with language, asking unambigious questions is notoriously difficult): what does ‘not really’ mean? One question says ‘do much’, what is ‘much’? The questions have two components so when I score a question high do I agree with both aspects or the relationship? Furthermore they seem to be a variation of the ‘confidence’ theme, an aspect which has been widely researched. I probably misunderstand the concept of ‘mindset’ when I think that ‘self-confidence’ covers this? In the 2007 Blackwell article the now called mindset questions are part of a ‘Theory of Intelligence’ scale which is part of ‘Motivational variables’. So there you have another element: motivation. I surely see how the role of self-confidence and motivation would influence achievement; how is mindset different? Further questions which I might explore in later posts.

References
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246–263.
Claro, S. & Paunesku, D. (2014). Mindset Gap among SES Groups: The Case of Chile with Census Data. Paper presented at the SREE Fall 2014 Conference.
Dweck, C. S. (1999). Self-theories: Their role in motivation, personality, and development. Philadelphia: Psychology Press.
Eisinga, R., Grotenhuis, M., & Pelzer, B. (2013). The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown? International Journal of Public Health, 58(4), 637-642.
Education Endowment Fund. (2015). Changing Mindsets: Evaluation report and Executive summary. Retrieved from https://educationendowmentfoundation.org.uk/modals/pdf_download/projects/56/3/857
Paunesku, D., Walton, G.M., Romero, C.L., Smith, E.N., Yeager, D.S., & Dweck, C.S. (2015). Mindset Interventions are a Scalable Treatment for Academic Underachievement. Psychological Science.

 

Categories
Education

A response to a W3C priorities blog

blog1
This blogpost is a comment on this interesting white paper by Crispin Weston and Pierre Danet about setting W3C priorities. I first thought to comment on the site but it became rather lengthy so seemed more logical to post it here. Since coming to work in the United Kingdom I have not really been involved in standards for education, but the topic triggered my previous experiences. As from quite early on, 2004-ish, I did my fair share of ‘standards’ work (SCORM packages, programming the SCORM module in moodle, assessment standards for Maths via SURF) I thought it would be good to comment on the blog in more detail, not only because I disagree with some of it, but more importantly, I think any new development in this area *must* learn from previous experiences. I thought long and hard before writing this piece because I don’t want to come over, or be dismissed, as someone against innovation per se. But I must admit, even when I lived in the Netherlands, I did not really feel people really wanted too much innovation.

The context of the document

I hope that some of the claims and associations made in the first paragraph(s) will be reworded or evidenced more. At the moment one sentence combining underperforming education, PISA, MOOCs and elite courses seems far-fetched. I also wonder whether a statement saying that it’s a good time for 1:1 because touch is easy-to-use isn’t a bit too ‘easy’. On evidence I would also say that, when we run with the assumption that technology can do both good and bad, there might not be general evidence of the impact of technology on learning outcomes because it is seen as a means to and end. Like any tool, means are used correctly and incorrectly.

Three barriers

The authors of the piece see three key barriers to the emergence of a dynamic ed-tech market (why the word ‘market’?):
  • The lack of interoperability of systems and content that would allow different sorts of instructional software to work together in an integrated environment.
  • The lack of discoverability of innovative new products.
  • The institutional conservatism of many state-run education systems, which are often resistant to innovation and uncomfortable with the use of data as a management tool.
I certainly agree with interoperability (or lack thereof), which *is* the main topic of the article, as being a big barrier (some more comments later on). The second one, discoverability, is not really defined but if it leads to the conclusion that the W3C would be a good organisation to connect different players, then that’s fine as well. However, the article then primarily emphasizes how the W3C should “work with governments that are prepared to address the institutional conservatism to be found in their own formal education system.”. This is in line with the third barrier the authors define, adding that these systems “are often resistant to innovation”. I think such statements are not warranted and rather unproductive. It also neglects the role, at least on the topic of adoption of standards and interoperability, of companies that, in my view, in the last decades systematically have undermined standards, either intentional or through mis-management and neglect. In my view this thread runs from even HTML3.2 (Netscape vs Internet Explorer) to the current ePub3, Kindle and iBooks endeavours.

Requirements

The paper then goes to requirements analysis. In general terms this section is ok, and I certainly see a lot of good things in Analytics and Games. I do, however, miss some analysis there. How are these aspects effective in “other businesses”?,  why is that the case?, What characteristics make it this way? What’s to say it would work in education? And, crucially, why and how would you adopt a whole paradigm? To use this then to argue that little has been done, and subsequently to propose how to go forward has a bit of a ‘light touch’.

Interoperability

What I do find interesting and appropriate is the conclusion that interoperability between the two strands, let’s concisely call them analytics and content, is an important aspect. So although I think the analysis falls short, I think there would be no harm in, even potentially serve as catalyst, to have good interoperability. But that’s not something new, of course. Getting a standard is a ‘different cookie’ (Sorry, a Dutch joke). I also like the ambition to outsmart the proprietory market and be there first.
blog2Having used SCORM myself and even having modded the SCORM module in Moodle so that it made more use of the SCORM specification, I think the lessons to learn from it, are not complete. One only needs to look at software that can create SCORM packages like Reload/Weload, Xerte, but also commercial packages to see that it has been possible to make rich content. So I’m not really sure whether it’s the lack of standardisation and tools why it has not really taken off. When I extended the SCORM module most users did not really care, they just wanted a simple scoring mechanism. But now as well: when I see current adaptive systems they are mainly multiple choice and scores. When I look at game mechanisms they mainly are Points, Badges and Leaderboards. To me, that indicates users might not really want more sophistication (yet). Now I understand this might be seen as a chicken/egg issue i.e. when we finally *can* make sophisticated stuff it *will* happen. Perhaps this is true (although history tells us otherwise) but it certainly would be smart to analyze the situation more deeply. Not in the least with regard to the role of the current edtech industry who, in my view, have sometimes frustrated attempts to formulate standards.
This also brings me to a general frustration with the fact that millions have been spent on attempts to write specifications on standards and, even worse, metadata. Over the years this has resulted in thousands of pages of documentation. Why will it be different this time? I feel that before setting out on yet such a journey, that question needs to be answered extensively. The description of the  SCORM standard shows that we are dealing with experts. Given what I said previously I think there are more important reasons for SCORM’s waning than others. Apart from asking ourselves what factors, we also need to ask ourselves how it will be prevented this time. I also wondered whether there still was any scope in assessments standards like QTI. A thread, in my view, through almost all standards is the mis-management and frustration by organisations and enterprises. If W3C leads the process, that is at least a strong start. In how far W3C can confront the largest enterprises in the world, I don’t know.
A second point risk remains hardware and software. Hardware and software becomes obsolete or deprecated. Every time when it happens we are told there are good reasons for it, often inefficiency or security (e.g. java, but that’s also, again mis-management and perhaps personal feuds), but in any case: who’s to say this won’t happen again. In my opinion it certainly ties in again with the corporate aspect. The W3C should be strong enough to address it.

SCORM was under-utilized

From a technical point of view I have always thought the CMI had not been used as well as possible. I agree that it was partly because of the SCORM specification but also unfamiliarity with it, for example the ‘suspendState’ option. A package could communicate quite a lot of information through this field, needing two things: (1) packages that would create a suspendState, (2) a VLE that would actually read that state. I remember being involved in the Galois Project, a maths algebra project, where we tried to do just that. The former was done by creating a system that could produce algebra SCORblog3M packages which utilized the suspendState for storage. The latter had indeed needed to be obtained by reprogramming a VLE, which I did for moodle’s SCORM module. The annotated SCORM module was plugged on the moodle forums. As said previously, most people simply did not see the point in doing such a thing. Now, this was just a hack, but it did led me to believe that there’s more going on in the education world, in that technology is (and probably should be) linked to the preferred didactics/pedagogy. So: maybe we don’t even need a lot of sophistication. Why am I saying this: I think it would be good to use some Rapid Application Development here: get some working stuff out there first, rather than slave for years on specs and metadata.

Conclusion

Having said all that, I do think, given the developments, that a follow-up for SCORM is needed. And also that it is warranted that W3C would look into something like that. It is smart, I think, to take the theme of connectivity rather than content, to make it more W3C. It also provides a good reason to include Analytics. The fact the authors mention privacy and data Protection acknowledges awareness of the politics involved with such an initiative. So overall I think this is a good initiative, but ask attention for the following:
  • Traction and commitment with enterprise. How prevent frustration of the process?
  • Rather get technology working quickly than endless specification and metadata mashing.
  • Promote a more sophisticated use of technology as well.
  • Either refrain from sweeping statements about ‘conservatism’ in education and focus only on interoperability, OR get more evidence for the various claims (I doubt you will find this).
Categories
Education

My take on a new College of Teaching

Initial depicting Boethius teaching his students from folio 4r of a manuscript of the Consolation of Philosophy (Italy?, 1385)

A lot has already been said about the proposed new College of Teaching (CoT). Up until now I have only devoted tweets to the topic, and because it might almost seem as if I’m against such a college, I thought it would be good to write this post. Let’s say I’m positive but politically skeptical. One thing that certainly has become apparent that there is quite some momentum it seems, within the teaching community itself. I even became very enthusiastic about the passionate posts by David Weston and Tom Bennett

Initial depicting Boethius teaching his students from folio 4r of a manuscript of the Consolation of Philosophy (Italy?, 1385)
Initial depicting Boethius teaching his students from folio 4r of a manuscript of the Consolation of Philosophy (Italy?, 1385)

(and more at Claim your College). Also, the run-ins I had with Andrew Old were helpful, and showed a drive to make it work. The main discussions center around who could join the CoT and what the CoT would do.

Who could join?
There was some discussion about what teachers were envisaged. Although I initially thought a broad group of teachers should be able to join, after all teaching takes place at many levels, I was convinced by the argument that indeed some sub-groups already have opportunities or can create opportunities themselves. For example, although I have been a secondary school teacher for 14 years, now I’m a teacher/lecturer in HE. Well, the Higher Education Academy is for me then (unfortunately, their funding has been cut considerably), and they provide CPD and certification. So, yes, I can see a good case for a professional body for only schoolteachers, and it could contribute to a stronger professional identity within the sector. It could also certainly play a strong role in CPD and other professional development.

Dangers exaggerated
Mind you,  I think the dangers of allowing a wider group of ‘teachers’ to join have also been exaggerated. I’ve seen words suggesting that outside groups might ‘dominate’ the CoT. I firstly wonder on what evidence this statement is based, as it does fit too easily in a caricature of an educational establishment trying to keep education in a hold.  A caricature which is often ‘proven’, ironically, by providing anecdotal evidence which doesn’t stand the test of reasonable social science research. More importantly, I think that even if this was the case I think the presence of lobby groups will not be any less with or without these groups, just different. I think most organizations eventually will ‘suffer’ from lobby or dominant groups. I don’t think it will be any different in a CoT. Vocal bloggers express their opinion? They might very well be dominant. Some teachers are Google teachers, Apple educators etc. They might very well be overly positive towards technology and be influential. I think any organization gets the audience it deserves.

Teacher Training (Education!) and politics
My main point of political skepticism concerns the function of the CoT. Most people who wrote about it assumed that the CoT would be a ‘professional body’. I, however, am a bit concerned that some groups, especially the government, will see the CoT as something more. In the consultation document by the government the foreword explicitly mentions initial teacher training. Under initial functions in 2.9 it mentions “This might include functions that are presently exercised by the Secretary of State, or other roles undertaken by the Department for Education or the National College for Teaching and Leadership (NCTL) in relation to teacher training and development.”. Especially, the ‘in relation to’ at the end begs the question what functions are meant. The NCTL has the following responsibilities:

nctlNow, if the first two bullets are representative of what the government means with ‘in relation to teacher training and development.’ then to me it’s clear that more stakeholders than ‘schoolteachers’ are involved. Especially the allocation of teacher training places surely is a joint responsibility. I think it would be very undesirable if a new CoT would decide over this unless the brief and membership of it would include all the relevant stakeholders. It would also fly in the face of what Sir Carter mentions in the introduction of the recent Carter review of initial teacher training: “The truth is that partnership is the key.”. The review mentions more critical points for the future, for example in recommending that a future CoT might “develop a framework for core content for ITT”. Again, I would feel it inappropriate if a professional body of only teachers would decide on this when multiple routes into teaching need to be represented.  If anything, the government’s response to the Carter Review seems to agree with a big role in all of this for a new CoT. It is this prospect of perhaps some formal decision powers on Teacher Training which might cause the  CoT to become a political pawn. It is no secret that a school-led teacher training has been the main focus of the government, and this has led to the closure of some university providers. I think a CoT should try to prevent being used as a political pawn and should clearly state that they acknowledge that Teacher Training is a partnership affair which doesn’t mean that a professional body of schoolteachers alone decide on it.

So to summarize:

  • I’m positive about a professional body for schoolteachers. I agree that a restricted scope of ‘members’ could help build the profession and build a professional identity.
  • I’m skeptical about statements which imply that a restriction of membership is needed to prevent some forces to become too dominant. Like any organization this will happen any way.
  • If the College of Teaching gets formal powers (from the NCTL for example) for Teacher Training, then the scope should be broadened and include all stakeholders involved in Teacher Training.

Other comments
The current, most extensive blueprint for the CoT is based on documents from the Prince’s Teaching Institute.
The Teacher Development Trust also has written plans.
Both look pretty extensive and good but don’t really address the point of Teacher Training.
I also think that one critical point will be whether such a CoT would have anything to do with prescribing Standards or not. Maybe more on this in later posts (then again, maybe not).

Categories
Education Research

OFSTED reports

I’m working on a very-belated post on Robert Peal’s OFSTED report for Civitas. I’ve been working on OFSTED reports for quite some time, harvesting a couple of thousand of them. For a sneak preview I refer to the presentation I gave at the ICMT conference:

Categories
Games

My five best games of 2014

Some might know I’m an avid gamer. MY best five games (ok ok I smuggled in more) I played in 2014 are:

South Park: The Stick of Truth
This an excellent, basically, adventure-meets-RPG game. The originators of South Park had a hand in this and it shows. The game basically is an interactive, whole series worth of South Park material.

Watch dogs
This was presented as a blockbuster A* game and after some tweaking I got it working well enough on my PC. The open world game clearly adopted elements from Grand Theft Auto but I particularly liked the ‘hacking’ twist: several times I raised the bridges while racing across them. There was a lot to do about the ‘dumbed down’ graphics after demos at the E3 conference, but as I have a PC: no problem.

Alien: Isolation (I think the gamespot 6 is unfair)
After the disappointment of AlienS: Colonial Marines this title does one of the best SciFi movies ever justice. The game is not so much a First Person Shooter but more an atmospheric ‘stealth’ game. Especially the sections where you have to evade the xenomorph are gripping and tense. I recorded the end (spoiler!):

Far Cry 4 (still finishing this one)
This basically is Far Cry 3 all over again but in a new (and gorgeous) environment, new animals (rhinos, elephants) and the same, excellent gameplay.

https://www.youtube.com/watch?v=-P6FXTxBON8

Everything from Telltale: The Wolf Among Us, Season 2 of the Walking Dead, Tales from the Borderlands, Game of Thrones
This is just a whole category of games, with very limited and simple gameplay. But that’s ok because the games are ‘story-driven’. The stories are captivating. Choices made in the game influence the subsequent events in the game.

Note that I couldn’t include Assassin’s Creed: Unity and Dragon Age: Inquisition because they simply didn’t work well (enough) on my system. Other notable games are Shadow of Mordor and -for the kids but also enjoyable- LEGO Lord of the Rings and Minecraft 😉

Categories
Education Research Research Statistical Methods

Costs and Benefits of Initial Teacher Training Routes

Only recently did i manage to read the November 2014 report published by the Institute of Fiscal Studies (but joint with the Institute of Education and NFER, funded by Nuffield) on the Costs and Benefits of Different Initial Teacher Training Routes. It is an interesting read; it would go too far to comment on all of the content, but it was striking that different media outlets chose different ‘slants’ on the report.

IFS had quite an extensive press release highlighting several aspects, while the NFER chose three of the findings. The report is an interim report from a Nuffield project (I noticed Nuffield funds the IFS for more ‘public spending’ projects).

It is fascinating to see how outsiders reported or blogged about the results. John Howson seems to emphasize the monetarization and quantification of ITT routes. I agree with him that this could turn into an issue: it’s shouldn’t solely be about numbers. However, for public justification of expenses in principle it is important to explain how public money is spent. The Public Finance website had quite a factual report, and among other points, noted how the issies around student loans and repayments. The University and College Union (UCU) also picked up this point but, in my opinion rightly so, asks attention to the longterm effects of the changing ITT landscape, and the hidden costs involved. They emphasize the threat to university education departments caused by reducing the direct allocation of training places to universities. A school-based teacher training provision prefers to highlight (and not suprisingly agree with) the result that a higher percentage of school-based ITT responded that the benefits of the route outweigh the costs. In a more extensive piece in Academies Week many of all these findings come together in one piece. It also mentions the ‘benefits’ of ITT routes. As this also got a mention in some tweets at the time, I’d thought I’d look into how this benefit (and costs of course) was determined in the report.

Chapter 4 of the report breaks down this topic. It first addresses the ‘central costs’ in 4.1, in which scholarships, bursaries, tuition fees and maintenance loans, maintenance grants, NCTL grants to schools, and NCTL contracts are taken into account. The key table is below. I was wondering who the recipients of these costs were throughout the report. For example, a bursary will be received by a trainee, a tuiton fee loan paid (back) by students but going to HEI’s etc. etc.

t42

After this the indirect costs for schools are caclulated in 4.2. Note that throughout the report focus is ‘Inner London’, but both primary and secondary education are looked at. This was done by, per term for primary and secondary education, looking at costs for mentoring, observations, lesson planning and other costs. This is where I feel the estimates become a bit vague. The estimates for the costs were obtained by asking respondents to report the time involved with the indirect costs associated with a specific trainee. This was combined with information on the pay category of the lead staff member involved, also representing the ‘opportunity costs of training’. The largest cost associated with ITT for primary schools is mentoring, with an average cost of around £39 per week. For secondary schools the highest was ‘observations’ and I was struck by the difference between routes: Teach First costs £29 per week, HEI-led £81 per week. I seriously wonder how this can be the case. It certainly explains the secondary schools differences in table 4.5 below.

t45Section 4.3 then describes the benefits. I was particularly interested how the report would calculate (monetize) the benefits. Apparently it started with a simple question: respondents were asked to report the extent to which the specific trainee in their recent experience brought a number of benefits to their school/department. These benefits, and percentages ‘strongly agree’ or ‘agree’ are reported in tables 4.7 and 4.8.

t47t48The monetary value was calculated by asking an additional question “whether the benefit for their school or department was greater than, equal to or less than the cost associated with the route, and whether this was to a ‘large’, ‘some’ or ‘small’ extent. Now, this seems somewhat subjective, maybe captured by the report’s use of the word ‘perception’.

t49

For primary, it is reported, whether it is reported that benefits outweigh the cost is related to specific benefits, especially whether the school expects to hire the trainee. This seems understandable, because you would want a large (time and money) investment not to leave the next year.

For secondary to groups of people were asked: secondary subject leaders (Departments) and secondary ITT coordinators.

t410

t411

This is all quite informative, although interpretation is difficult. It’s the subsequent monetarization that made me scratch my head. This started with assuming that net benefit was a continous variable with the answers to the question whether the benefits were less, equal or larger and the extent, as underlying property. A next assumption then is that the benefit-cost ratio has a Gamma distribution. It is argued that ‘this is reasonable’ as it is flexible and because ‘it can approximate a large range of distributions depending on the parameters.’. I find this justification unconvincing. But the assumptions continue with the assumption that respondents’ interpretation of ‘large’, ‘some’ and ‘small’ extent is similar AND that the value for each is the same above and below benefits=costs. A final assumption concerns a margin of approximation (see p.49 and appendix D of the report). Then the gamma distribution was fit to the survey results, and then draws made from the optimal gamma distribution. The draws were averaged to provide average net benefit. For the three groups, primary, seocondary subject leaders and ITT coordinators the corresponding tables are 4.16, 4.17 and 4.18:

t416

t417

These tables seem to show what these assumptions do with the results. Take Teach First, the report mentions that, for ITT coordinators, the very high average net benefit is mainly caused by higher monetary costs as reported by the ITT coordinators, but also a higher estimated benefit-cost ratio. I thought the former was very strange as table 4.6 seemed to indicate that the costs were among the lowest. I find it difficult to understand what causes the difference between these observations. This is important to understand as high costs, combined with high benefit-cost ratio, imply more benefit. The calculation of the ratio also needs to be unpicked. I somehow feel that there are far too many assumptions here for such a conclusion, especially given the nature of the original questions. One could argue that it basically is a 7-point Likert scale with benefits to large extent less than costs, to, benefits to large extent more than the costs. With assumptions that these ‘steps’ are equal, Gamma distrubution assumptions, but also that it concerns the perception of benefits-costs ratio, seems problematic to me. Appendix D further explains the procedure and it seems that the first column is the average of the calculated benefit-cost ratios (those drawn from the Gamma distribution, I presume). It makes a big difference whether values are drawn to the right of the distribution or not. Now had I taken from table 4.11, for example, that benefits>cost percentage for HEI-led ITT was comparable to, for example Teach First. I have no idea why the benefit-cost ratio is lower. Overall, given all the assumptions I think the net benefits reported in terms of ‘monetarization’ are not really sound.

 

Categories
Education Research

Was the Sutton Trust report really about myths?

I’ve tweeted quite a lot about this report and true, this blog post comes quite late after its publication. I was quite positive about the report, but did notice that the press and the blogosphere seemed to mainly focus on the two (!) pages in the report on ‘ineffective practices’ (p22-p24). The executive summary -correctly- barely mentions them. reportI think this emphasis on ‘myths’ is a shame because the first half with effectiveness models, and the second half on frameworks for capturing teaching quality, are very interesting. In addition, the section on myths is rather light-weight regarding evidence, in my opinion. The character of the report, called a ‘review of underpinning research’, is not really shown in this specific section. Sure, this doesn’t mean that ineffective practices should not be uncovered, but surely this goal is better served with some more grounding. Something that should have been easy to do, as in some cases genuine reviews are referenced.

Let me give some examples:

Use praise lavishly
Praise for students may be seen as affirming and positive, but a number of studies suggest that the wrong kinds of praise can be very harmful to learning. For example, Dweck (1999), Hattie & Timperley (2007). Stipek (2010) argues that praise that is meant to be encouraging and protective of low attaining students actually conveys a message of the teacher’s low
What evidence is used here? Firstly Hattie and Timperley: their review on p96 is much more nuanced in my opinion. There are several sources that might have been used to support the claim in the report. Dweck’s article also seems to mainly reference her own work; I would hardly call it a review. Finally Stipek, a reference which I had trouble finding because it was placed wrongly in the reference list at the D, is a website. The website is an excerpt from a 2002 book. It could very well be that the contents also is in a more formal publication, but using this source as evidence for a myth does not seem to be such a strong case.
This happens with the other points as well:
Allow learners to discover key ideas for themselves
Enthusiasm for ‘discovery learning’ is not supported by research evidence, which broadly favours direct instruction (Kirschner et al, 2006). Although learners do need to build new understanding on what they already know, if teachers want them to learn new ideas, knowledge or methods they need to teach them directly.
The Kirschner et al. paper is excellent, and published in reputable journal. It would, however, been nice if there were some more references, even when they would be sources reviewed in exactly that article.
Group learners by ability
Evidence on the effects of grouping by ability, either by allocating students to different classes, or to within-class groups, suggests that it makes very little difference to learning outcomes (Higgins et al, 2014). Although ability grouping can in theory allow teachers to target a narrower range of pace and content of lessons, it can also create an exaggerated sense of within-group homogeneity and between-group heterogeneity in the teacher’s mind (Stipek, 2010). This can result in teachers failing to make necessary accommodations for the range of different needs within a supposedly homogeneous ‘ability’ group, and over-doing their accommodations for different groups, going too fast with the high-ability groups and too slow with the low.
This section again references the Stipek website (and indirectly book). Higgins et al. from 2014 is NOT in the reference list (sloppy, there are several of these errors), I assume it’s the 2013 reference to the Learning and Teaching toolkit. Within that am I to assume the report is referring to ‘setting and streaming’? There are references here, maybe some more taken from those references could have made the evidencing more substantial.
Encourage re-reading and highlighting to memorise key ideas
This finding has already been mentioned in summarising the review by Dunlosky et al (2013). Re-reading and highlighting are among the commonest and apparently most obvious ways to memorise or revise material. They also give a satisfying –but deceptive– feeling of fluency and familiarity with the material (Brown et al, 2014). However, a range of studies have shown that testing yourself, trying to generate answers, and deliberately creating intervals between study to allow forgetting, are all more effective approaches.
The Dunlosky et al. article is a strong source. Brown et al. refers to the book ‘Make it stick’ which again might be strong, but I would say that in general I’d rather see a peer-reviewed source. There also are some common names in both (Roediger for example), which might be a good thing (triangulation) but also could mean self-citation. I also think the mention of ‘a range of studies’ should have been evidenced.
Address issues of confidence and low aspirations before you try to teach content
Teachers who are confronted with the poor motivation and confidence of low attaining students may interpret this as the cause of their low attainment and assume that it is both necessary and possible to address their motivation before attempting to teach them new material. In fact, the evidence shows that attempts to enhance motivation in this way are unlikely to achieve that end. Even if they do, the impact on subsequent learning is close to zero (Gorard, See & Davies, 2012). In fact the poor motivation of low attainers is a logical response to repeated failure. Start getting them to succeed and their motivation and confidence should increase.
This reference refers to an extensive report by the Joseph Rowntree foundation. I assume p76 and p77 are the relevant pages, and they give me the impression that here too, the findings are more nuanced than qualifying it as an ‘ineffective practice’.
Present information to learners in their preferred learning style
A belief in the importance of learning styles seems persistent, despite the prominence of critiques of this kind of advice. A recent survey found that over 90% of teachers in several countries (including the UK) agreed with the claim that “Individuals learn better when they receive information in their preferred learning style (for example, visual, auditory or kinaesthetic)” (Howard-Jones, 2014). A number of writers have tried to account for its enduring popularity (see, for example, a clear and accessible debunking of the value of learning styles by Riener and Willingham, 2010), but the psychological evidence is clear that there
are no benefits for learning from trying to present information to learners in their preferred learning style (Pashler et al, 2008; Geake, 2008; Riener and Willingham, 2010; Howard-Jones, 2014).
This section has quite some references. The Howard-Jones one is about prevailing perceptions of teachers, but indeed Pashler et al. is a strong source. I would, however, frame it differently: ‘evidence it is not effective’ is not the same as ‘no evidence that it’s effective’. It is also ironic that the aforementioned Teaching and Learning Toolkit (Higgins et al., 2014) classifies Learning Styles as “Low impact for very low cost, based on moderate evidence.”.
Ensure learners are always active, rather than listening passively, if you want them to remember
This claim is commonly presented in the form of a ‘learning pyramid’ which shows precise percentages of material that will be retained when different levels of activity are employed. These percentages have no empirical basis and are pure fiction. Memory is the residue of thought (Willingham, 2008), so if you want students to remember something you have to get them to think about it. This might be achieved by being ‘active’ or ‘passive’.
I think here two separate issues are connected. The ‘learning pyramid’ (Cone of experience?), especially the specific numbers, has been debunked. However, is it true that the header of the section ‘is commonly presented in the form of the learning pyramid’? I read no supporting evidence for this. Of course, I think I know what is referred to, but I’d rather have some more substance included.
Overall, I think this section could have had more substance. The pages, in my opinion, do not warrant the current coverage of both the press and the blogosphere. Certainly not in the sense of soundbites like ‘the Sutton Report shows that <insert ‘myth’ here>’. It also did not seem the intention of the authors, as the executive summary barely mentions them (although I thought it was strange that this article on these ineffective practices came first, only a week later followed by what does work; also the press release starts with the ineffective practices). To conclude, I think too much emphasis has been put on two pages from a much longer report by the press and the blogosphere. Not only doesn’t the total length of the ‘ineffective practices’ warrant this, the section also is quite lightweight on evidence. I think the rest of the report, which is a better mix of teaching frameworks and teaching quality, subsequently was undervalued. This does not mean ‘myths’ need not be debunked, of course. Luckily, the topic is quite fashionable, so we can be sure that more rigorous articles and books will appear. I hope, however, that this will be a case of evidence, not a case of sound-bites.
Categories
Uncategorized

It’s been a while…

snaI’m updating the website and hope to make more time for blogging.

For a start I posted the materials for the Network meeting in Duisburg on October 21st, 2014 here.

 

Hopefully more to follow.