Tag: research

Categories
Education Research

Unpicking economic papers: a paper on behaviour

One of the papers that made a viral appearance on Twitter is a paper on behaviour in the classroom. Maybe it’s because of the heightened interest in behaviour, for example demonstrated in the DfE’s appointment of Tom Bennett, and behaviour having a prominent place in the Carter Review.

Carrell, S E, M Hoekstra and E Kuka (2016) “The long-run effects of disruptive peers”, NBER Working Paper 22042. link.

disrupt

The paper contends how misbehaviour (actually, domestic violence) of pupils in a classroom apparently leads to large sums of money that people will miss out of later in life. There, as always, are some contextual questions of course: the paper is about the USA, and it seems to link domestic violence with classroom behaviour. But I don’t want to focus on that, I want to focus on the main result in the abstract: “Results show that exposure to a disruptive peer in classes of 25 during elementary
school reduces earnings at age 26 by 3 to 4 percent. We estimate that differential exposure to children
linked to domestic violence explains 5 to 6 percent of the rich-poor earnings gap in our data, and that
removing one disruptive peer from a classroom for one year would raise the present discounted value
of classmates’ future earnings by $100,000.”.

It’s perfectly sensible to look at peer effects of behaviour of course, but monetising it -especially with a back of envelope calculation (actual wording in the paper!)- is on very shaky ground. The paper respectively looks at the impact on test scores (table 3), college attendance and degree attainment (table 4), and labor outcomes (table 5). The latter is also the one reported in the abstract.

table5There are some interesting observations here. The abstract’s result is mentioned in the paper “Estimates across columns (3) through (8) in Panel A indicate that elementary school exposure to one additional disruptive student in a class of 25 reduces earnings by between 3 and 4 percent. All estimates are significant at the 10 percent level, and all but one is significant at the 5 percent level.” The fact economists would even want to use 10% (with such a large N) is already strange to me. Even 5% is tricky with those numbers. However, the main headline in the abstract can be confirmed. But have a look at panel C. It seems there is a difference between ‘reported’ and ‘unreported’ Domestic Violence. Actually, reported DV has a (non-significant) positive effect. Where was that in the abstract? Rather than a conclusion along the lines whether DV was reported or not, the conclusion only focuses on the negative effects of *unreported* DV. I think it would be more fair to make a case for better signalling and monitoring of DV, so that negative effects of unreported DV are countered; after all, there are no negative effects on peers when reported.

 

 

Categories
Education Research

ResearchED 2015

Today I want to researchED at South Hampstead High School in London. It was a very fruitful day. Can’t say I heard a lot of new things, I knew most things, either because of my ‘job’ or because I had already read and heard of it via the blogosphere. But it was also very important: speaking from my background as a former secondary maths and computer science teacher and now lecturer/researcher (with some involvement in teacher training as well) I thresearchED-logoink it’s vital that practitioners (practice) and researchers work together in partnership. There are many obstacles for this to happen – I imagine I might know about the obstacles from both sides, practitioners and researchers, because I have experienced and am experiencing both. These are two distinct cultures that need to ‘bridge’ the gap. For this, government needs to ‘invest’ and not think -like with the maths hubs- volunteering can cover this. But ok, enough about that. A kaleidoscope of sessions I visited (although I did tweet about some ‘abstracts’ I read but couldn’t visit).

I started off with Lucy Crehan who reported on international comparisons and her experiences with visiting six countries to discover their education system. I liked this session a lot because I work quite a lot with international comparative but also with more qualitative data from, for example, the TIMSS video study. I try to combine both in the international network project ‘enGasia‘ in which England, Hong Kong and Japan collaborate to (i) compare geometry education in those countries, (ii) design some digital maths books for geometry, and (iii) test them both qualitatively through Lesson Study and more quantitatively through a quasi-experiment. Some of the work from John Jerrim was mentioned.

Then I finally got to see Pedro De Bruykere (slides here), whom I already knew to be a very engaging and funny speaker. He went through many of the myths in the book he wrote with Casper Hulshof and Paul Kirschner: Learning Styles, Learning pyramids, some TED talks (Sugata Mitra who featured in previous blogposts here and here, and Ken Robinson). I can recommend the book as a quick way to get up to speed to myths (and almost-myths). I liked how Pedro described how the section on ‘grade retention’ became more nuanced between the Dutch and English editions of the book because of the results of a new study.

Then back to international comparisons with Tom Oates. I already knew his report on textbooks and I agree that textbooks have to offer us a lot. But then again, I would think that: in the Netherlands maths (my subject) textbooks are used a lot and edited the proceedings of the International Conference on Mathematics Textbook Research and Development 2014. Tim’s talk covered quotes on international comparisons and unpicked the problems (fallacies, faulty arguments) with them. He had a measured conclusion:

After lunch I went to the #journalclub with Beth Greville-Giddings for some cookies. I had prepared by reading the article and making these annotations. The session explained the process of starting a journal club first and then we discussed the paper. One interesting moment was when people discussed the statistics in the paper. I agreed with comments that because we were dealing with a reputable journal the statistics probably were correct. But in my view there is another problem: statistical literacy. In this paper two things stood out for me (statistically, there were more like the definition of engagement): the term ‘significant’ and ‘variance explained’. With largescale data the sample size often is quite high, causing significance more quickly. Because of this reason ‘effect size’ is probably more appropriate. Secondly the statistics seemed to show that not much more extra variance is explained by adding engagement predictors. Any way, journal clubs to me seem like a worthwhile venture; might be good to forge partnerships with HE as well.

Crispin Weston then gave a lecture on how technology might revolutionise research. He framed this by first describing 7 problems and then showing how technology might ‘improve’ or ‘address’ them. It was an interesting approach which resulted in a matrix with ‘solved’ problems. Learning Analytics and standards (see my response to W3C priorities) had a prominent place. I’m a bit skeptical of it all will work. In the MC squared project we are implementing some Learning Analytics, including for creativity, and it’s bloody difficult.

Sri Pavar then talked about Cognitive Science. I think it’s good to summarise these principles. Principles like a memory model and relevant books (although some books referenced were not really about Cognitive Science) were presented. Cognitive Load Theory (CLT) had a prominent place. I couldn’t help tweeting some critical comments about CLT (a good summary here, a newer interesting blog on the measure used here). For example the ‘translation’ of research that a lower cognitive load is better: of course not, you wouldn’t learn anything. Or the often used measurement instrument:

Or the role of schemas: germane load was an (unfalsifiable) attempt at explaining schemas within the CLT framework but apparently some have abandoned it because of the unfalsifiable nature. But then what? And what does it add to existing information processing theories.

The final session was by Professor Rob Coe. He talked about several things pertaining to ‘what works’. He talked about Randomised Controlled Trials, logic, and took us back to last year’s ResearchEd, Dylan Wiliam’s talk.

I am with Coe here. Rob mentioned a little cited paper that sounded very interesting.

Tom Bennett finished the day. In the North Star it was great to meet a whole range of Twitterati, It was an interesting day and professionally I hope practitioners and researchers (Primary, Secondary Education and Higher Education) can grow towards each other:

Oh, and let me end on a contrarian note: some people have got to read up on all that cognitive psychology: most researchers are for more nuanced in their papers 😉

 

Categories
Education Research

Some work presented in the last months

snaSome work was presented in the last months.

At Sunbelt XXXV I presented this work on classroom interaction and Social Network Analysis:

At ICTMT and PME my colleague presented our work on c-books

Categories
Education Research

Predatory journals

More and more I’m being confronted with questions about journal publications. I devote some words to it in a session for our MSc programme in the module ‘Understanding Education Research’ and recently, in a panel discussion at our local PGR conference, there were questions about how to judge a journal’s reputation. Note that in answering this question I certainly don’t want be a ‘snob’ i.e. that only the conventional and traditional publication methods suffice. Actually, developments on blogging and Open Access are positive changes, in my view. Unfortunately there also is a darker side to all of this:

One place where I always look first when it comes to ‘vanity press’ and predatory journals is Beall’s List, which is “a list of questionable, scholarly open-access publishers.”. What I like about this list is that they are rather sensible about how to use the list: “We recommend that scholars read the available reviews, assessments and descriptions provided here, and then decide for themselves whether they want to submit articles, serve as editors or on editorial boards.”. The list of criteria for determining predatory open access journals is clear as well. One thing you can do is use the search function to see if a journal or publisher gets a mention. This is exactly what I did recently with some high profile research. I was surprised to find out articles were indeed published in such journals.

The first example is this high profile article mentioned in the Times Educational Supplement. It references a press release from Mitra’s university:  

 
The journal title did not ring a bell so I checked Beall’s list, and yes the journal and publisher are mentioned in this article on the list. Just a quick glance, also the comments, should make most scholars think twice to publish in here, certainly if it is ‘groundbreaking’ stuff. This is not to say that articles per se are bad (although methodologically there is much to criticisise as well, maybe later, although this blog does a good job at concisely flagging up some issues) but I am worried that high profile professors are publishing in journals like these (assuming it was done with the authors’ agreement, predatory journals sometimes just steal content to bump up their reputation). In the case of this person it has happened before, in 2012, when the ‘Center of Promoting Ideas’ (this name would be enough for me to not want to appear in their publications) published this article in a journal, which is also on Beal’s list. It is poignant that an Icelandic scholar really got into problems because of this. Some other examples: this article, CIR world also features on Beall’s list (Council for Innovative Research, again a name which raises suspicion by itself).

  

These publications serve as examples that even high end professors could fall victim of predatory journals. I do not mean that in a judgemental way; it shows that more education on the world of predatory journals is needed. Although I must admit, there might be some naivety at play here, experienced scholars should know ‘positive reviews only’, ‘dubious publishing fees’ and ‘unrealistic publication turnovers’ are very suspicious. Early Career Researchers often are targets of predatory journals and it therefore is important to be aware of this ‘dark side’ of Open Access publishing. Beal’s list covers these but recently there also are more and more ‘non open access’ journals that might be a bit dubious as well. In many cases it’s quite a challenge to judge the trustworthiness of publications. Certainly if in social sciences we would want to go away from the hegemony of the five big publishers, there is a lot to be gained in general skills to judge literature. Now, everyone has their own judgements to make when it comes where they want to publish, but I would be very concerned publishing in any journal (and for any publisher) on Beall’s list.

Categories
ICT Research

Convenience tooling

Saw_tooth_setter_kerfIn research we often refer to ‘convenience sampling’ as sampling where subjects are selected because of their convenient accessibility and proximity to the researcher. The most obvious criticism is sample bias. In working with other researchers, reading articles and PhD students there also is a danger of something I would like to call ‘convenience tooling’: choosing the tool first tool you see, you know or has a favorable image. Now, of course there could be many good reasons why a researcher chooses to do so. Maybe it’s because he/she has worked with or developed a certain tool. Maybe the tool in question is ‘the only tool’ that has a certain features. However, to at least have some sort of reasoning behind the tool choice, in my opinion, a good researcher should give arguments why he/she chooses a certain tool. Preferably, if the need to use a tool arises because of a certain research question or framework, a researcher should write down what features for a tool are needed to answer the research question. Then it should be argued how the chosen tool provides the features that are needed. You could compare this whole process with making a requirements document. In the end, it could very well be that the choice for a tool remains the same, but at least a researcher -just as with sampling- is ‘forced’ to make some of his/her tool choices more specific.

Categories
Math Education MathEd Research

BSRML conference – report

I have written three posts on the BSRLM day conference on November 17th, 2012.

The three posts are:

BSRLM conference part 1
BSRLM conference part 2 Alnuset
BSRLM conference part 3

Categories
Math Education MathEd Research

BSRLM conference part 3

The fourth session by Ainley reported on the Fibonacci project, integrating inquiry in mathematics and science education. It was good to hear that the word ‘utility’ that was used, did not refer to a utilitarian view of maths, i.c. that everything should have a clear purpose. I mention this as discussions about utility often tend to end in comments like ‘what’s the point of doing algebra’? Actually, I think that does have a purpose, amongst others ‘analytical thinking’ but I prefer steering clear from these types of pointless discussions. The best slide, I though, was a slide with science, statistics and mathematics in the columns and rows with a distinction in, for example, their purpose.

It formed a coherent picture of STEM. The two examples for integrative projects were ‘building a zoo’ which I didn’t like when it concerned the context of fences that had to be built. It’s the lack of creativity that often is in textbooks as well. the second project, on gliders, was more interesting but the mathematical component seemed to belong more in statistics used. I would loved to have seen a good mathematical example.

The fifth session by Hassler and Blair was about Open Educational Resources. The project, funded by JISC, acknowledged three freedoms: legal, technical and educational. It is a project that boasted a website with educational resources, free to use, keywords and with pdf creator. Although nicely implemented, to me, it seemed to be a bit ‘yet another portal’. The individual elements weren’t that novel either, with for example a book creator also in the Activemath project. The most interesting thing was the fact that the materials were aimed at ‘interactive teaching’.

The sixth and last session was a presentation by Kislenko from Estiona. She described how in Estonia a new curriculum was implemented for educating teachers in mathematics and natural sciences. It was an interesting story, although I was wondering how ‘new’ it was, as the title had the term ‘innovative’ in it.

Together with some networking these sessions made up an interesting and useful day in Cambridge.

Categories
ICT Math Education MathEd Research Tools

BSRLM conference part 2 Alnuset

The third session I attended was more a discussion and critique session, led by Monaghan and Mason, on the topic of ‘cultural affordances’. The basis was the work of Chiappini, who -in the ReMath project- used the software program Alnuset (see here to download it) to look at (its) affordances. Monaghan described the work (a paper on the topic, there will be a publication in 2013, was available) and then asked some questions. Chiappini distinguishes three layers of affordances: perceived, ergonomic and cultural. Engestroms cycle of expansive learning is used, as I understood it, to use activities as drivers for transformation of ergonomic affordances into cultural affordances. Monaghan then asked some critical questions, under which whether the theory of Engestrom really was necessary, wouldn’t for example Radfords work on gestures be more appropriate? Another comment pondered whether the steps for expansive learning were prescriptive or descriptive. I think the former: as the author has made the software with certain design elements in mind it is pretty obvious that they have a preconceived notion of how student learning should take place.  It was pretty hard to discuss these more philosophical issues in detail. I’m not really sure if I even understand the work. Although this could be solely because I haven’t read enough about it, I also feel a bit as if ‘difficult words’ are used to state the obvious. I could only describe what I was thinking off. The article that I took home afterwards gave some more pointers. To get a grasp of this I downloaded the software, that reminded me a bit of the Freudenthal Institute’s ‘Geometrische algebra’ applets, and tried out the software. I liked the idea behind the software. In this example I’ve made three expressions, and I can manipulate x. The other two expressions change with x. Some comments:

  1. I like the way expressions are made and the look and feel, as well as the way dragging changes the expression. Also ‘dividing by zero’ causes expressions to disappear. However, why does x=0 disappear as well when I drag x to 0? (see figure)
  2. I don’t see how the drawback of every tool that allows ‘dragging’, namely just pointless dragging, in this case just to line up the different expressions, is solved. Maybe this isn’t the main goal of the software.
  3. I think that the number line should be used in conjunction with tables and graphs, thus forming a triad expression-table-graphs. The addition of things like an algebraic manipulator and a Cartesian plane seems to indicate that the authors also like more than one representation.
  4. It has far too limited scope for algebra. The 30 day trial is handy here, as in my opinion the software doesn’t do enough to warrant the price.
Categories
Math Education MathEd Research

BSRLM conference part 1

On Saturday November 17th I visited the second day of the BSRLM conference (British Society for Research into Learning Mathematics). I’ve become a member as it’s ‘the place to be’ for maths education research. This time the conference was in Cambridge, and apparently I was the only one tweeting #bsrlm.

The first session I attended was by Anne Berit Fuglestadt from the university of Agder (soon, homebase of a Dutch researcher I know). She reported about teachers discussing inquiry-based teaching with digital tools.

The second half of the session consisted of discussions on instrumental and documentational genesis (The French School, Trouche is an important name). This was fitting, as one PhD student I (co-)supervise is studying instrumentation as underpinning framework for her study.

The second session was an interesting take on use of the Livescribe pen. At first it seemed as if the study, done by Hickman and Monaghan, seemed a bit of a waste of the livescribe pen. Emphasis was put on the audio recording facilities.

Luckily, as I could have expected, they did more with the pen. The pens were used to record student work while ‘thinking aloud’ and these materials (a sort of screencasts of what was written) were used for a combination of stimulated recall and task-based interviews (e.g. Goldin, 1997). Hickman showed some discourse by primary students that was recorded with the pen. It was nice to see student work being ‘constructed’ instead of just having static scans of their work. It also was nice that we could try out the pen ourselves. I did think more can be done with even the older generation of pens. For example, Dragon Naturally Speaking does doe a decent job of transcribing voice, just as long as it is trained to recognize it. It will certainly cut the amount of time you need for transcribing an hours worth of audio.

Another application to use would be Myscript, from the same company that brings a great online equation recognizer. The latest version of the pen also boasts Wifi and Evernote integration, so it looks interesting. It will certainly be worthwhile to check out this for our SKE+ group. A follow-up discussion could be whether these devices will eventually become obsolete if tablet technology with styli, like the Galaxy Note, takes off.