CEFR-based Short Answer Grading

A corpus of short answers written by learners of English and graded with CEFR levels

Abstract

The project through which the corpus was collected is concerned with the task of automatically assessing the written proficiency level of non-native (L2) learners of English. Drawing on previous research on automated L2 writing assessment following the Common European Framework of Reference for Languages (CEFR), we investigate the possibilities and difficulties of deriving the CEFR level from short answers to open-ended questions, which has not yet been subjected to numerous studies up to date.

The object of our study is twofold: to examine the intricacy involved with both human and automated CEFR-based grading of short answers. First, we compiled a learner corpus of short answers graded with CEFR levels by three certified Cambridge examiners. Next, we used the corpus to develop a soft-voting system for the automated CEFR-based grading of short answers.

Corpus v1.0

Download

© ALTISSIA International & CENTAL, UCLouvain, Belgium · cental@uclouvain.be

Citation

More information can be found in the following paper. When using the corpus in your research or publication, please cite this paper as well.

@inproceedings{W17-5018,
    title = "Human and Automated CEFR-based Grading of Short Answers",
    author = {Tack, Ana{\"\i}s  and
      Fran{\c{c}}ois, Thomas  and
      Roekhaut, Sophie  and
      Fairon, C{\'e}drick},
    booktitle = "Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications",
    month = sep,
    year = "2017",
    address = "Copenhagen, Denmark",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/W17-5018",
    doi = "10.18653/v1/W17-5018",
    pages = "169--179",
    abstract = "This paper is concerned with the task of automatically assessing the written proficiency level of non-native (L2) learners of English. Drawing on previous research on automated L2 writing assessment following the Common European Framework of Reference for Languages (CEFR), we investigate the possibilities and difficulties of deriving the CEFR level from short answers to open-ended questions, which has not yet been subjected to numerous studies up to date. The object of our study is twofold: to examine the intricacy involved with both human and automated CEFR-based grading of short answers. On the one hand, we describe the compilation of a learner corpus of short answers graded with CEFR levels by three certified Cambridge examiners. We mainly observe that, although the shortness of the answers is reported as undermining a clear-cut evaluation, the length of the answer does not necessarily correlate with inter-examiner disagreement. On the other hand, we explore the development of a soft-voting system for the automated CEFR-based grading of short answers and draw tentative conclusions about its use in a computer-assisted testing (CAT) setting.",
}

Examples

Describe your hobbies.
(at least 30 words)

My hobbies are: cooking, gardening, sewing, learning chinese (Mandarin), play with my grandchildren. I also like to improve my English language for example “Memrise” Chinese - English. With “Skritter” reproduce caractères in Chinese.

Grades
A1
/
A2
A1?
A2
Minimal text, but mainly accurate English.
Majority vote
A2

You are invited to a friend's birthday. You respond thanking him/her. Suggest some different ways that you can help him/her.
(at least 60 words)

Hello, I've just received your invitation and I wouldn't miss it for the world! Thanks a lot!! I know organising a party can be difficult: there is so much to plan, to think of... So if you need my help, don't hesitate to ask. I could for example come tomorrow morning to help you with the decoration, or run errands for you this afternoon. Let me know ;) Bye

Grades
C1
Correct register for task set
Imposs.
C1 or C2. There are no errors, but a B2 could write this. A more demanding task is needed.
B2
Text completely correct, but does not show a variety of language. An accurate text message, for example!
Majority vote
C1

Acknowledgements

We would like to thank the Centres de Langues (CLL) and the Institut des langues vivantes (ILV) of Louvain-la-Neuve, Belgium for their indispensable contributions to the corpus collection and annotation.

This work received the financial support of the ALTISSIA e-learning company.