The PAIR framework applied to assessment in HE
The PAIR framework applied to assessment in HE
The PAIR model enables lecturers to systematically integrate GenAI tools into assessment
(This post is a translation generated by AI of my original post in Spanish; if you spot any inconsistencies or mistakes, please email me at: maricruzgarciavallejo@gmail.com)
This blog post is based on my Spanish adaptation of the PAIR model for the module “CAIE24: Developing AI Competencies Applied to Assessment: Toward a more authentic assessment”, which I teach at the University of Las Palmas de Gran Canaria.
PAIR (an acronym for Problem, AI, Interaction, and Reflection) is a framework for assessment and teaching developed by Oguz A. Acar, a marketing professor at King’s College London. This framework was first published in June 2023 in the article “Are Your Students Ready for AI? A 4-Step Framework to Prepare Learners for a ChatGPT World” for Harvard Business Publishing Education. With the rise and popularization of GenAI applications such as ChatGPT, the higher education sector has shifted from its initial resistance to student use of GenAI tools to consider how best to introduce these tools to facilitate learning, how to integrate them into assessment with clear pedagogical goals, and what guidelines and recommendations to provide students to ensure ethical use and the development of technical skills and higher-order cognitive processes, broadly encompassed by the term "AI competencies."
The terms "AI competencies" and "AI literacy” are relatively recent, and emerged following the popularization of generative AI in education. In principle, “competency” and “literacy” are not equivalent terms (competency refers to the practical application of specific expertise or skill in a given context and for a particular purpose, while literacy refers to the skill or knowledge in a broader sense).
To know more, I recommend UNESCO´s AI competency frameworks for students and teachers.
Oguz based his framework on his research in AI psychology and student learning, operating on the premise that GenAI could only enhance learning if students first developed five fundamental cognitive “skills” of higher-order cognition (skills that, initially, were unrelated to technical abilities for managing GenAI applications). The cognitive skills Oguz lists in his article are:
Formulation
Exploration
Experimentation
Critical Thinking
Willingness to learn
These five skills form the basis of the four steps in the PAIR framework, which gained popularity after being included in the inaugural edition of the short course “Generative AI in Higher Education” developed by King’s College London (an excellent resource for university educators looking to incorporate GenAI into their teaching practices).
What Does the PAIR Framework Entail?
The PAIR framework provides a structured methodology to facilitate student use of GenAI in the context of both summative and formative assessment. Specifically, PAIR promotes authentic assessment, such as problem-solving, inquiry, or project-based learning, and supports active learning.
First, the instructor designs an assessment task that requires students to complete a project or solve a problem or inquiry, typically related to the professional or occupational field of the student’s discipline (for example, creating a report, analyzing a dataset, conducting research or a market study, producing a video, or developing a podcast on a professional topic).
The student then follows a four-step process to complete the assessment task:
(The graphic in English and the original formulation of the PAIR framework can be found on the King’s College London website)
Problem Formulation: The student defines the problem, inquiry, or challenge and determines what constitutes a solution. The type of question or challenge can vary:
Open Inquiry: Students are given autonomy to identify and formulate their own project, inquiry, or problem.
Closed Inquiry: A structured format where a predefined problem is provided to students.
Semi-Guided Inquiry: The instructor offers broad topics or loosely defined problems, allowing students to focus on specific aspects of the issue.
Selection of AI Tools: The student identifies generative AI's role in solving the problem and selects the most suitable AI tool(s) to assist in the solution.
Interaction: The student engages with the problem or challenge by interacting with AI; this involves experimenting with various inputs (additional data, prompting techniques, format of instructions) and critically evaluating the AI-generated output, improving and refining the result until the desired outcome is achieved.
Reflection: The student critically reflects on the interaction process with generative AI, considering how the inputs and instructions provided to the AI were modified to achieve an acceptable result.
Benefits of Using the PAIR Framework
Unlike other frameworks or methodologies for integrating GenAI into assessment, PAIR focuses on the individual (in this case, students, though it can also apply to teacher training courses) and on developing higher-order cognitive skills that enable individuals to use AI critically, ethically, and with full awareness.
PAIR is a framework designed to foster critical AI awareness, and it is grounded in three key principles:
Human-Centred Approach: AI is viewed as a tool to expand and complement—rather than replace—human intuition, judgment, and creativity.
Focus on Human Cognitive Skills: AI usage emphasizes the development of mental processes and skills essential for holistic individual growth (such as problem formulation, exploration, experimentation, critical thinking, and reflection), rather than technical mastery of software or specific applications.
Responsibility-Centred: Critical reflection on the role of AI in assessment is embedded throughout the framework, particularly in the final step. Here, students select the tool to solve a problem and reflect on the overall experience. This requires an awareness of AI's limitations, risks, and potential—topics that lecturers should explore alongside students to encourage mature reflection on AI integration in assessment, allowing students to take an active role.
The PAIR framework enables the intrinsic development of competencies and skills characteristic of AI competency and literacy frameworks, including:
Formulating a problem, inquiry, or challenge in a way that AI can address.
Using prompting techniques to break down a complex problem into subtasks that can be addressed by generative AI applications.
Applying prompting techniques to improve or refine initial results until the desired outcome is achieved; this refinement process is often referred to as pre-training or fine-tuning, depending on the approach.
Having a foundational technological understanding of available GenAI applications and software to select the most effective tool for the problem.
The final step of the framework, reflection, also serves to strengthen AI competencies by linking reflection with ethical considerations. This could include evaluating the use of GenAI tools for specific professional activities, recognizing data protection and privacy risks associated with using these tools or identifying potential racial, cultural, or social biases in the information or initial outputs presented by the AI.
Since the PAIR framework involves the formulation of a problem or inquiry, its use in assessment methods supports the development of students’ own assessment literacy. Students are the ones who formulate the problem or inquiry, enabling them to connect the theoretical and/or practical knowledge they have acquired in an academic discipline with the responsibilities they will undertake in their professional lives. Students also select and use the AI tool in the way a professional in the field or discipline would be expected to.
What is Assessment Literacy?
Assessment literacy refers to a student’s ability to:
understand the purpose of assessment within their academic discipline,
relate the assessment to specific learning objectives,
comprehend the instructions provided in an assessment task,
critically evaluate the project or task provided as evidence that relevant knowledge in the discipline has been acquired, and determine whether this knowledge meets the quality standards defined by the assessment criteria.
Disadvantages of Using the PAIR Framework
As mentioned earlier, the PAIR framework requires students who are sufficiently mature and have a baseline level of assessment literacy. Rather than adopting a passive role, the student becomes an active examiner, capable of formulating a complex problem or inquiry that aligns with the purpose of the assessment and meets the course learning objectives. The student must be prepared to assess others, or self-assess, according to a set criterion or assessment metric.
To some extent, the PAIR framework assumes that students possess prior knowledge of cognitive skills and competencies typical of advanced-level education, such as critical thinking, the ability to deconstruct a complex problem into sub-problems or subtasks, and the ability to engage in reflective thinking and express it in writing using minimally academic and professional language. Given the varying skill levels of new university students worldwide, it is likely that, in the early years of university studies, not all students will have developed mature critical thinking or reflective skills (or be able to articulate these in the language of instruction for their discipline). The PAIR framework should be used with students who are sufficiently mature and have acquired basic academic competencies, such as academic reading and writing, researching, breaking down complex problems into subtasks, and formulating a problem.
For students to be able to select and evaluate GenAI tools, they must possess a foundational understanding of these tools, which may not always be the case. Therefore, before implementing the PAIR framework as part of an assessment task, it is necessary to consider:
The level of AI competencies students possess.
The GenAI tools to which students have access to avoid technological discrimination; for instance, if an assignment or coursework requires the use of a text-to-text generator to perform basic tasks such as summarising, analysing, and synthesising related literature, students who can afford subscriptions to the highest-performing LLMs will have an advantage over those limited to free versions (which often do not include the latest model developments). Institutions should ensure that only institutionally recommended tools are used, providing equal access for all students.
The initial GenAI training required students to complete steps 2 and 3 of the PAIR model.
Finally, an assessment method based on the PAIR framework places the responsibility on the student to be honest about the role that AI has genuinely played in solving the problem or task, or in generating the final product or outcome. It is expected that the student will interact with this tool and provide a reflective discourse on their interaction with the AI tool. However, a poorly designed, or, overly, simplistic assessment, carries the risk that the AI tool will solve the task or problem following an initial prompt and with minimal intervention from the student. We must avoid formulating excessively simple problems or queries where the student’s added value is negligible. There is, certainly, a risk that the student may not be truthful in their final reflection, particularly regarding the extent of their interaction with AI to achieve the final result.
Additionally, the student may use unauthorised AI tools or those that pose risks concerning personal data protection, infringe copyright rights, or violate national and supranational AI regulatory frameworks, either consciously or unconsciously. The final decision regarding an acceptable-and ethical-use of AI rests with the student.
The PAIR framework presents both advantages and disadvantages, like any framework applied to assessment. Educators must carefully consider the maturity of the students regarding assessment and their level of competencies and knowledge in AI before adopting the framework in their teaching practice.