Jamie Dunlea is a researcher for the British Council specializing in English language assessment. Jamie joined the British Council in 2013 after a 20-year career in EFL education in Japan.
From 2007–2013, Jamie was the Chief Researcher for the Eiken Foundation with responsibility for designing the validation research agenda for a large-scale testing program. In his current role, Jamie works closely with the English Language Assessment Research team at the British Council to design and implement research for the Aptis testing system and other assessment related issues.
Topic: Incorporating the CEFR into language test development: using an international framework in local contexts
The presentation will first look at how the Common European Framework of Reference for Language has been used in assessment generally, including the criticisms and questions that have arisen.
The second half of the paper will look at some concrete examples of how the CEFR can be used as a springboard for task and rating scale development.
Video recording of the plenary session
You can watch the full recording of Jamie Dunlea’s talk on British Council Russia’s YouTube channel by clicking the link below:
Watch an interview with Jamie Dunlea speaking about what CEFR is and what make it different in terms of assessment here: http://www.viddler.com/v/5dc5fe03
Right after the plenary session Jamie Dunlea answered audience’s questions about Common European Framework of Reference for Languages.
Irina Ozolina: What are the examples where CEFR doesn't fit?
The first thing to stress, of course, is that the CEFR is more than the six-level descriptive scale of proficiency, the common reference levels, and the illustrative scales for specific communicative activities linked to the scale. When people ask this question, it is most often in reference to the common reference levels and illustrative scales, but there is a lot more in the CEFR itself which often deals with areas the scales do not. A good case in point is young learners. As one of the plenary speakers yesterday, Catherine Kneafsey pointed out, the CEFR scales are very much focused on older learners, and the descriptors describe activities appropriate for such learners. Others, for example Angela Hasselgreen in an article in a special addition of Language Testing on the CEFR (2005) have also pointed this out. At the same time, the actual document of the CEFR does recognize this, and suggests that scales focusing on areas of language use appropriate for children should be developed for use in primary schools. This is something that has also driven the approach to the European Language Portfolios, in which descriptors for activities and aspects of performance on those activities appropriate for learners in primary school have been developed in many countries. Another aspect which needs careful attention is English for Academic Purposes (and of course it could equally apply to other languages for academic purposes, but I am speaking from my own experience). The CEFR can provide the broad brush benchmarks which we might want to use as reference points for developing a test for language skills for academic purposes, but we would need to examine the Target Language Use domain in much more detail and devise performance standards appropriate to the academic context. At the same time again, we see how the CEFR is adapted, and adaptable to these needs. The scale on Notetaking in Seminars and Lectures was added based on a specific study in a specific context to address a skill relevant to the academic context. I would also point people to the many studies that have been done in varied contexts on linking examinations to the CEFR as good sources for identifying where things have, and haven’t worked appropriately, and the approach that has been taken in response. And remember the principle that adapt rather adopt as is usually the more appropriate approach.
Ekaterina Shadrova: Do tests based on CEFR assess learners' competences (their ability to use L2 in real life situations) or their knowledge of language?
Here, I would say that tests based on the CEFR can do either, or both. It is a matter of identifying the uses and interpretations for which we want to use the assessment. At the same time, as I stressed in my presentation, the common reference levels and illustrative scales in the CEFR are not assessor or constructor oriented scales, using the three-way distinction proposed by Alderson (1991). They were designed as user-oriented scales, and as such any assessment based on them needs to do a lot of work to create test specifications. An explicit model of validation needs to be used, and this will guide the developer to relevant aspects of task and test specification, including the cognitive processes engaged by the test tasks. The specific focus of the tasks, and the associated task features will be driven by the intended purpose and the construct/skill of interest. Of course when working with the CEFR, the aspects of language knowledge are much more underspecified than the aspects of language use associated with the communicative activities. This results from the language-neutral principle of the CEFR employed in its development. Language-specific aspects of language knowledge will need to be developed in much more detail for particular contexts. For English, for example, this is one of the primary focuses of the English Profile project. Fleshing out aspects of language knowledge will be particularly important in developing rating scales for tasks which have been developed from the CEFR, as sufficient information is definitely not in there to allow the scales to be used as rating scales without further development (and once again, they were not designed as assessor oriented scales, so this is no surprise).
Irina Kostyukovich: Can testing systems be influenced by a national context? Can't they be a standard followed by everyone and how can CEFR help establish such a standard?
Absolutely. Indeed they should be. But once again we should always remember that there is no one, right answer. It is a matter of uses and interpretations, purpose. For certain uses, it will be more relevant, indeed important to focus on transferrable, transparent information which is amenable to being carried across contexts, and is understandable and useful to those contexts. This is the “common” part of a common scale. But a common scale or framework can only be stretched so far before it becomes too general to be useful anywhere. In the of national-level education systems, particularly secondary school, we would definitely want to have a strong focus on the local context and assessment which is designed to be relevant to and reflective of the local curriculum and local needs. We definitely would not want to force a local testing system to fit an international framework. We would want adaption on both sides. Bringing the ideas, methods, and frameworks which are relevant to a wider context can of course be very useful, inspire innovation and change in the local context. But we always need to make sure we take notice of and are relevant to the local context. And here I am referring to a national-level education system as “local” in relation to a more generic, international system.
Irina Kostyukovich: In what aspect can CEFR make testing system more student-friendly?
Well, the focus on the positive aspects of what students can do is a defining principle of the approach in the CEFR. It is designed to allow for positive feedback and to identify goals for working towards what people can do with the language. That aspect in itself, when used within the classroom, can be very positive. The CEFR also offers self-assessment tools and encourages this aspect of ownership and involvement by learners in understanding their own proficiency and how it is assessed. I think that approach can be very positive, and that is part of the thinking behind the European Language Portfolio projects as well. This approach also gels well with a current trend in assessment which is assessment literacy. The idea behind assessment literacy is that we need to develop an understanding of how assessment is carried out, its purposes, and its limitation as well as uses among non-specialist users of language tests, including learners and teachers. Utilizing self-assessment and utilizing the can-do descriptors in the CEFR to set learning goals and work towards them, and to understand and participate in assessing them is beneficial for learners. It also helps develop more demanding users of tests, which is a good way of adding to quality assurance.
Vera Bobkova: What would you recommend to read to a teacher who wants to know more about your research?
A lot of what we discussed in the presentation is of course not my research, but that of colleagues and other who have put in the groundwork to give us the tools which I have applied.
For the socio-cognitive framework of test development and validation I would point you to Weir, C. J. (2005). Language Testing and Validation: an evidenced-based approach, and O’Sullivan & Weir (2011) O’Sullivan. Test development and validation. In B. O’Sullivan (Ed), Language Testing: Theories and Practice. How the socio-cognitive framework has been adapted and applied to the development of a specific testing program is described briefly in O’Sullivan, B. (2012). Aptis Test Development Approach. Aptis Technical Report ATR-1.
Tamara Oshchepkova: Designing tests is a really hard task. Would you advise teachers to use ready-made ones or design tests for their specific context with CEFR in mind?
Once again, it is always a matter of purpose, but also resources. There is no one answer. I think ready-made tests often serve very important purposes, and for many high-stakes usages it is important to bring the technical expertise and scale, for example to allow pretesting and equating, that “read-made” tests developed and delivered by specialized testing organizations, can provide. Of course there is the other end of the ready-made test spectrum which is cheap and efficient, and there are places for these too. But in terms of classroom practice, wherever possible teacher involvement in the development of assessments relevant to the situation is obviously preferable. The important thing is to stress that teachers should try to work together to develop assessments, build their assessment literacy and critique and provide input for each other’s testing solutions. Other kinds of ready-made assessments can be built into total solutions. In almost any situation, no one testing or assessment tool should be used in isolation, and various assessment tools and sources of information should be combined.
Tatiana Likintseva: Are there any training programmes to help teacher get the ropes of using CEFR and other frameworks in test development?
I am not aware of specific training programs or courses aimed this, though various organizations, such as ALTE may run workshops periodically. Often times training will be connected to particular national or regional projects rather than a course on offer for all. There is a great deal of information available on the “tool kit” webpage of the Council of Europe
http://www.coe.int/t/dg4/linguistic/Manuel1_EN.asp, and I would point people here to look for actual examples of performance at the different levels, and other tools. I would stress again that nothing should be taken at surface level, and you should approach the information from the perspective of brining your own important local knowledge. The European Association of Language Testing and Assessment also has a very useful online archive of conference presentations and other tools related the CEFR and its application in various contexts.
You can download Jamie Dunlea’s presentation “Incorporating the CEFR into language test development: using an international framework in local contexts” below