The CAA International EALTS and the VRIP principles

Following best practice in English language examinations for speakers of other languages, the development and production of the CAAi EALTS has been underpinned by a consideration of the four essential examination qualities of validity, reliability, impact and practicality, referred to collectively as the VRIP principles. 1

Validity is generally considered to be the most important examination quality; it concerns the appropriateness and meaningfulness of an examination in a specific context and the specific inferences made from examination results and refers to the extent to which an examination measures what it is intended to measure and can be shown to produce scores or ratings which are an accurate reflection of the candidate’s actual level of language ability in the skills assessed as set against the criteria by which they are assessed. The validity of an examination is often described in relation to a number of related concepts for which evidence of validity can be provided:

  • Construct validity – the extent to which the examination results conform to the model or models of language ability underlying the examination.
  • Content validity – the extent to which the examination covers the full range of knowledge and skills relevant and useful to real world situations and authentic language use.
  • Criterion-related validity (predictive and concurrent validity) – the extent to which examination scores correlate with a recognised external criterion which refers to or measures the same area of knowledge or ability.
  • Face validity – the stakeholder acceptance of the examination as an appropriate means of measurement of whatever it is the examination claims to measure.

In language testing, validity is one of the ways in which the overall fairness of an examination can be interpreted, with validity being seen as the degree an examination measures what it is supposed to measure. Aviation language tests have high stakes. It is important for safety and for the integrity of the industry, particularly the operators and for test-takers themselves, that language tests be fair and accurate. 2

Reliability is a key concept in any form of measurement and contributes to overall validity. In language assessment, reliability concerns to the overall stability of an examination - the extent to which examination results are fair, consistent, accurate and free from errors of measurement, both in relation to a single candidate and across candidate populations, and therefore the extent to which the results can be depended on for making decisions about a candidate.

For tests of speaking where the assessment of candidate performance involves judgement by examiners or assessors, it is important to consider reliability in terms of the accuracy and consistency of the assessments which are made. The reliability of subjective assessments (i.e. examiner judgements) is usually estimated using correlations (e.g. intra- and inter-rater reliability correlations).

With validity, reliability is the second of the two ways in which the overall fairness of a test can be interpreted, with reliability being seen as the degree that the test produces consistent and fair results. 3

Impact refers to the effects, educational, professional, social, economic or political which an examination may have on the candidates and other test users. The effects of this impact may be positive, negative or neutral.

It is recognized that examinations can have a major impact on language training processes and on society in general. This impact operates on at least two levels:

  • a macro level: in terms of general language training processes
  • a micro level: in terms of the individuals (stakeholders) who are affected by examination results

Positive impact on teaching and learning is an important aspect of impact which operates on both levels (macro and micro). It is in this context that the notions of "face validity" (or test appeal) and wash-back are considered.

In the context of English for Aviation language proficiency testing for ICAO compliance, the overall impact of any assessment means should lead to improved operational practice in aeronautical communications and enhanced aviation safety. 4

Practicality is an integral part of the concept of examination usefulness and affects many different aspects of an examination. It refers to the extent to which an examination is practicable and sustainable in terms of the balance of resources required to produce, administer and support it (including the funds, time and expertise) in its intended context of use with the resources available to do so.

The practicality of any examination, involves two factors:

  • the resources that are required to produce an operational examination that has the appropriate balance of qualities (validity, reliability and impact) for the context in which the examination will be used
  • the resources that are available in all aspects of the development and administration of the examination

A practical examination is one that does not place an unreasonable demand on available resources.

It is the consideration given to these four essential assessment criteria that assures the quality of the CAAi English for Aviation Language Testing System. 5

1 c.f. www.ALTE.org for content and further details. Also ICAO Cir 318-AN/180 Ch.1 Fundamental Principles of Language Testing 3.3

2 c.f. Ch.2 Test Validity and Reliability 2.1. Statements of evidence for test validity in plain, layperson language accessible to test stake-holders are available in the CAA International EALTS Test Specification 2009.

3 c.f. ICAO Cir 318-AN/180 Ch.2 Test Validity and Reliability 2.1 Statements of evidence for test reliability in plain, layperson language accessible to test stake-holders are available in the CAA International EALTS Test Specification 2009.

4 c.f. ICAO Cir 318-AN/180 Ch.1 Introduction to Language Testing 3.4. Further details of test impact in the form of a Test Wash-back statement are provided in the section EALTS Test Wash-back in this Handbook.

5 c.f. ICAO Cir 318-AN/180 Ch.1 Introduction to Language Testing. Further details in relation to test administration and delivery are provided in the section Test Administration in this Handbook.