Analysis and results
The area of analysis is managed by CITO’s psychometrician Gunter Maris.
A brief overview
Prior to the Main Study in 2011, SurveyLang will conduct a Field Trial: the analysis following the trial will provide information on the quality of the test items (both language tests and questionnaires) and in-country translations. This will allow any arising issues to be resolved before the Main Study.
After the conclusion of the Main Study, the Language Tests completed by the students sampled in participating countries will be marked and the Questionnaires’ results analysed. This process not only examines students’ responses, but also the way that the tests themselves have functioned. At this stage, particular attention will be paid to any differences found across languages and countries.
The data from the Language Tests and Questionnaires will also be analysed together to study the relationships between performance and concepts covered by the Questionnaires. A key focus will be looking at the relationships, both similarities and differences across languages and countries. These will be presented in the Final Report. The European Commission may use the analysis in the report to make future policy recommendations. Countries participating in the survey will also be able to examine their own data and report on the findings relevant to their country and context.
Marking of writing
Unlike the listening and reading tests and the Questionnaires, which can be easily marked as responses are predominantly multiple-choice, the marking of writing requires a different approach. SurveyLang has developed a markscheme specific to its assessments and will provide exemplar tasks and training on use of this markscheme. Marking will be completed in-country with a sub-sample quality controlled centrally by SurveyLang.
Data analysis methodology
The language testing data will be analysed using a One Parameter Logistic Model (OPLM; Verhelst & Glas, 1995; Verhelst, Glas & Verstralen, 1995). The basic approach is that the data sets for each of these three skills represent a unidimensional scale. Exploratory and confirmatory factor analysis will be used to examine the relationships in more detail.
Further advanced data analysis techniques such as cluster analysis and multidimensional scaling will be applied to detect and analyse systematic differences between countries.
The most important part of the analysis will be devoted to studying the relationships between the Questionnaire data and the language test data. The basic technique used is called latent regression analysis.
The standard setting section provides details on how SurveyLang assessments are related to the CEFR.