1. Summary
The AI Evaluation Team, part of the GDS IDEA Unit (Insight, Data Science, Economics, and Analysis), leads and supports the impact evaluation of AI programmes across HMG.
We work to build a robust and credible evidence base that demonstrates the impact of AI initiatives across government via GDS-led initiatives. As part of our core offer, we assist policy and delivery teams in planning and executing AI impact evaluations.
The framework provides a common structure and shared language for AI impact evaluation, designed to help teams measure the value realised from AI investments and support more informed decisions around adoption, scale, and future investment. It encourages consideration of a broad set of impact categories while preserving flexibility in how teams select and apply metrics. It is intended for evaluation leads both within our team and across policy and delivery teams, to help design their evaluations.
1.1 A note on this guidance
This framework was developed by the AI Evaluation Team in consultation with the Evaluation Task Force, the wider AI and evaluation community across government, and academic AI experts.
This is the first iteration of the framework. It establishes core principles, impact categories, and example metrics, and will evolve alongside technological developments and feedback from stakeholders and live evaluations. Future iterations may provide more detailed methodological guidance on data collection, quality, and measurement approaches suited to different tool types and use cases.
2. How to use this framework
2.1 About the framework and where it sits in the AI lifecycle
An impact evaluation is a systematic assessment of whether an AI tool caused a specific change in outcomes and by how much. A good way to think about it is in terms of what the AI tool does to people, services, and organisations.
Impact evaluation is one piece of evidence collected along the AI evaluation cycle. The diagram below, developed in collaboration with the Evaluation Task Force and i.AI, provides useful context for where impact evaluation sits within the broader AI lifecycle. As shown in the top right of the diagram, impact evaluation takes place once a tool has moved through incubation, alpha, and beta stages and is in live deployment.
The AI impact evaluation framework is built around shared impact categories, defined in section 3, with clear definitions that teams across government can adopt consistently.
Some deployments will require consideration of additional, more specialised impact categories beyond those set out here. For example, a policing AI tool may require further consideration of privacy, civil liberties, or security impacts; a healthcare tool may require closer attention to patient safety, clinical risk, and equalities impacts. Teams are strongly encouraged to consult subject matter experts and relevant stakeholders to ensure they have identified the full range of intended and unintended impacts relevant to their specific context before finalising their evaluation design.
Teams should identify the categories most relevant to their deployment and select metrics that best answer their evaluation questions, adapting measures to their context, tool type, and available data.
What is in and out of scope
The framework is scoped to deployment impact: what the AI does to people, services, and organisations. The related areas below sit outside its scope but are important complements to it. Future work should more explicitly explore and address interdependencies among these strands.
Model evaluation
Addresses whether the model itself is performing well, including whether it is accurate, fast, and unbiased. This should be completed before, alongside and after deployment as part of separate assurance processes, which may also include privacy assessments and accountability governance. Use this framework alongside the AI Testing and Assurance Framework for the Public Sector (opens in new tab).
User research
Explores how users engage with tools, identifies potential barriers and enablers to their effective use, and how well tools meet their needs. It can form part of a process evaluation or impact evaluation. It is also used earlier in tool development, to strengthen a product before piloting.
Process evaluation
Examines how the intervention was implemented in practice, what worked, what did not, and why. It helps explain impact findings and surface implementation barriers. Teams are strongly encouraged to conduct process evaluation alongside impact evaluation.
Value for money and costing
Includes cost-benefit analysis and cost-effectiveness assessment. Where efficiency metrics such as time saved are collected, these should not automatically be converted to cashable savings; doing so requires a separate analytical step and a number of assumptions that may not hold in all contexts.
2.2 Designing your evaluation
Teams are encouraged to develop a Theory of Change and clearly defined evaluation questions, and to select metrics from there. This grounds measurement choices in a clear account of how and why the AI intervention is expected to produce change. For further guidance on AI impact evaluation design, including how to develop a Theory of Change, refer to the Guidance on the Impact Evaluation of AI Interventions in HM Treasury's Magenta Book (opens in new tab).
Findings from short-term studies should, in most cases, be treated as indicative only. The appropriate evaluation window varies by impact category:
- some outcomes, such as efficiency gains or initial adoption rates, could be assessed over shorter horizons where sample sizes are sufficient
- others, including social and user impact and public and user trust, are inherently time-dependent and require a longer window to avoid incomplete or misleading conclusions
- second-order effects and behavioural changes typically only become visible once a tool is embedded in real working practices
Short evaluation windows are also structurally prone to capturing novelty effects rather than sustained impact, particularly for AI tools used in decision-making workflows.
Teams should also apply judgement about which impact categories are in scope for their deployment, but are encouraged to document early observations across all relevant categories, even qualitatively, rather than omitting them entirely.
2.3 Checklist for robust impact evaluation
- Use your Theory of Change to select metrics.
- Define clear evaluation questions that the metrics will answer.
- Measure real human behaviour and real-world outcomes, not just system outputs.
- Randomised Controlled Trial: include a control or comparison group. Aim to randomly split your target sample of users into a control group (status quo) and a treatment group (who receive the AI tool) to avoid self-selection, akin to an A/B test. If this is not possible, find another suitable comparison group.
- Use a sufficiently large and representative sample of users. If this is not possible, clearly document and report sample and methodology limitations.
- Combine quantitative and qualitative methods. While numbers capture what changed, interviews, focus groups, and case studies capture why, and surface context, intangible benefits, and unintended effects.
- Use adoption and engagement data (section 3.3) as contextual evidence to interpret the reach and extent of impact findings.
- Be cautious when relying on self-reported data for efficiency or productivity gains, as these can be biased or unreliable. Objective measures are preferable where available. Surveys, interviews, and other qualitative methods remain valuable for assessing areas such as staff wellbeing and user experience.
- Incorporate time-to-benefit realisation metrics against key gains to inform decision-making. Impact rarely appears the day after deployment, so track the timeline from implementation to the first meaningful gain.
- Assess Technical and Organisational Readiness (TRL/ORL). This typically happens ahead of an AI pilot. Evaluating the gap between a tool's technical maturity and the organisation's capacity to adopt it is a prerequisite for understanding its potential for scale.
- Pair impact evaluation with pre-, during and post-deployment model evaluation so that technical performance, safety and real-world impact are assessed in tandem. Refer to the AI Testing and Assurance Framework for the Public Sector (opens in new tab) for guidance on model evaluation.
- Pair impact evaluation with process evaluation to understand how the intervention was implemented, what worked, and what did not.
3. AI impact evaluation framework
The framework is grouped into seven broad impact categories, each accompanied by illustrative KPIs and example measures. KPIs are illustrative only; teams should select measures relevant to their context.
The table below sets out each impact category and its definition. Sections 3.1 to 3.7 then provide detail on example KPIs and measurement methods, including definitions to aid interpretation. Section 4 explores broader considerations which we recommend as areas to document rather than measure in a pilot.
| No. | Impact category | Definition |
|---|---|---|
| 1 | Productivity | The extent to which AI enables staff, teams, or departments to deliver more or higher-quality outputs with the same or fewer inputs. |
| 2 | Efficiency | The extent to which an AI intervention enables a process or service to achieve the same or better outputs using fewer inputs, such as reductions in time, cost, or manual effort. |
| 3 | Adoption and real-world performance | The extent to which users consistently engage with AI tools and the degree to which those tools perform reliably in live environments. |
| 4 | Staff wellbeing | The effect of AI on employee experience, workload balance, and perceived meaning or satisfaction in work. |
| 5 | Public and user trust | The degree to which citizens, users, or stakeholders have confidence in AI-enabled tools, including their transparency, accuracy, and fairness. |
| 6 | Social and user impact | The direct effects of the AI deployment on users and those around the service, including changes to confidence, anxiety, exclusion, dependency, or working relationships. |
| 7 | Scalability | The potential for an AI intervention to be replicated or expanded across departments or service areas. |
3.1 Productivity
Productivity refers to the extent to which AI enables staff, teams, or departments to deliver more or higher-quality outputs with the same or fewer inputs.
| Illustrative KPI | Example measure |
|---|---|
| Cases processed per FTE | Admin data / output monitoring |
| Median tasks completed per user per week | System logs / dashboards |
| Quality-adjusted output (for example, % approved without revision) | Supervisor review / QA data |
| Quality of policy output | A/B comparison with non-AI-assisted outputs / structured expert review / independent scoring against quality criteria |
| Error reduction rate (pre/post automation) | QA logs / audits |
KPI definitions
- Cases processed per FTE
- The number of cases, requests, or work items completed per full-time equivalent member of staff over a given period. Used to track whether AI enables teams to handle greater volumes of work.
- Median tasks completed per user per week
- The midpoint number of discrete tasks completed by individual users in a given week. Median is preferred over mean to reduce the effect of outliers.
- Quality-adjusted output (% approved without revision)
- The proportion of AI-assisted outputs, such as documents, decisions, or recommendations, accepted or approved without requiring further amendment. A higher rate indicates improved output quality.
- Quality of policy output
- An assessment of whether AI-assisted policy work meets defined quality standards.
- Error reduction rate (pre/post automation)
- The percentage change in errors or defects identified in outputs before and after AI deployment. Measured against a consistent quality standard using QA logs or audit data.
3.2 Efficiency (time savings)
Efficiency refers to the extent to which an AI intervention enables a process or service to achieve the same or better outputs using fewer inputs, such as reductions in time, cost, or manual effort.
| Illustrative KPI | Example measure |
|---|---|
| Median time saved per task / FTE | A/B testing / survey |
| Average handling time reduction | Process mining / time logs / diary study |
| Typical use of time saved at work | Survey |
KPI definitions
- Median time saved per task / FTE
- The midpoint time saving per completed task or per staff member as a result of AI assistance. Measured by comparing task completion times before and after deployment.
- Average handling time reduction
- The change in average time taken to process or resolve a case, request, or interaction end-to-end. A reduction indicates efficiency gains in the overall workflow, not just individual tasks.
- Typical use of time saved at work
- A qualitative or survey-based measure capturing how staff are using time freed up by AI. Important for understanding whether time savings translate into productive reallocation or are simply absorbed.
3.3 Adoption and real-world performance
Adoption and real-world performance refer to the extent to which users consistently engage with AI tools and the degree to which those tools perform reliably in live environments.
| Illustrative KPI | Example measure |
|---|---|
| Active user rate (% using tool weekly) | System logs / dashboards |
| Tool drop-off rate post-trial | System logs |
| Training / onboarding completion rate | Learning management system |
| Accuracy | Human-in-the-loop audits / SME testing |
| Degradation over time, change in accuracy or consistency | Periodic SME audits / test battery |
KPI definitions
- Active user rate (% using tool at least weekly)
- The proportion of intended users who engage with the AI tool at least once per week. A proxy for whether the tool has become embedded in working practice rather than used sporadically.
- Tool drop-off rate post-trial
- The percentage of users who stop using the tool after an initial trial or onboarding period. High drop-off may indicate usability issues, lack of perceived value, or inadequate support.
- Training / onboarding completion rate
- The proportion of intended users who have completed required training or onboarding before or during deployment.
- Accuracy of outputs over time
- The proportion of AI outputs rated as correct or fit for purpose by subject matter experts, tracked at regular intervals across the deployment period. Indicates whether the tool maintains quality standards in real-world use.
- Degradation over time
- The change in accuracy or consistency of AI outputs between early and later stages of deployment. Identifies whether tool performance is stable, improving, or declining in real-world use over time.
3.4 Staff wellbeing
Staff wellbeing refers to the effect of AI on employee experience, workload balance, and perceived meaning or satisfaction in work. Pre/post comparisons require sufficient deployment time to be meaningful.
| Illustrative KPI | Example measure |
|---|---|
| Meaningful work score | Pre/post deployment survey |
| Workload balance score | Pre/post deployment survey |
| Self-reported reduction in repetitive tasks | Survey |
| Digital confidence / skills uplift | Pre/post capability self-assessment |
| Change in burnout / job satisfaction score | Pulse survey / HR data |
| Staff turnover / retention | HR data |
| Disruption metrics, for example change in number of steps in a process | Survey / interviews |
KPI definitions
- Meaningful work score
- A self-reported measure of whether staff feel their work is purposeful and valuable following AI deployment. Captures whether automation of routine tasks has enhanced or diminished perceived job meaning.
- Workload balance score
- A self-reported measure of whether AI deployment has improved, worsened, or had no effect on staff experience of workload distribution and manageability.
- Self-reported reduction in repetitive tasks
- The proportion of staff reporting a decrease in time spent on routine or repetitive work following AI deployment. Indicates whether the tool is delivering on its intended efficiency purpose from the user's perspective.
- Digital confidence / skills uplift
- A measure of change in staff confidence and capability in using digital and AI tools. Relevant for understanding workforce readiness and training effectiveness.
- Change in burnout / job satisfaction score
- The change in standardised burnout or job satisfaction scores before and after deployment. Captures broader wellbeing effects beyond task-level experience.
- Staff turnover / retention
- The rate at which staff leave their roles during or following AI deployment. Elevated turnover may signal negative wellbeing or cultural impacts; improved retention may indicate positive effects.
- Disruption metrics
- Qualitative or survey-based indicators of the degree to which AI deployment has disrupted established working practices, team dynamics, or role expectations. This tells us if AI is highly disruptive, moderately disruptive, or not very disruptive. For example, measuring the change in the number of steps in a process as a result of using the AI tool.
3.5 Public and user trust
Public and user trust refers to the degree to which citizens, users, or stakeholders have confidence in AI-enabled tools, including their transparency, accuracy, and fairness. Trust takes time to establish and measure reliably.
| Illustrative KPI | Example measure |
|---|---|
| Transparency / explainability rating | User testing |
| Perceived fairness / impartiality score | Public survey / deliberative panels |
| % of users reporting trust in AI outputs | Public or internal user survey |
| Complaint / appeal rate (AI-assisted decisions) | Administrative data |
KPI definitions
- Transparency rating
- A user-assessed score of how well the AI tool communicates what it is doing, including what data it uses, its purpose, and its limitations. Measures whether users understand how the system works at a general level.
- Explainability rating
- A user-assessed score of how well the AI tool explains why it produced a specific output or recommendation. Measures whether users can understand the basis for individual AI-generated results.
- Perceived fairness / impartiality score
- A measure of whether users or citizens believe the AI tool treats people consistently and without bias.
- Percentage of users reporting trust in AI outputs
- The proportion of users who report confidence in the accuracy and reliability of the tool's outputs. A key trust indicator best tracked over time to assess whether trust builds or erodes with experience.
- Complaint / appeal rate (AI-assisted decisions)
- The rate at which users or citizens formally contest decisions made with AI assistance. An elevated rate may indicate perceived unfairness, lack of transparency, or actual errors in outputs.
3.7 Scalability
Scalability refers to the potential for an AI intervention to be replicated or expanded across departments or service areas. Readiness can be assessed during a pilot; evidence of actual replication accumulates over time.
While scalability is not strictly an impact measure, it is included because it is a priority consideration for senior leaders assessing pilot outcomes, and a small number of KPIs collected during deployment can provide useful early indicators of scaling potential.
| Illustrative KPI | Example measure |
|---|---|
| Replication readiness score (governance, data, documentation) | Self-assessment / peer review / survey |
| Repeatable use cases identified | Output and performance monitoring |
| Cross-department interoperability score | Technical evaluation |
| Successful replications (solutions adopted by other teams) | Output and performance monitoring |
KPI definitions
- Replication readiness score
- A composite assessment of whether the AI deployment has the governance, data infrastructure, and documentation in place to be replicated in other teams or departments.
- Repeatable use cases identified
- The number of distinct use cases within the deployment that have been assessed as suitable for replication elsewhere, based on performance and transferability criteria.
- Cross-department interoperability score
- A technical assessment of whether the tool's architecture, data standards, and interfaces are compatible with systems used in other departments, a prerequisite for scaling.
- Successful replications (solutions adopted by other teams)
- The number of confirmed instances where another team or department has adopted the tool or approach. The most direct evidence of scalability in practice.
4. Broader considerations: areas to document rather than measure
The following areas are unlikely to be measurable within a short pilot. Evaluation teams should document relevant risks and early observations and build formal measurement into longer-term deployment plans where the tool is scaled.
4.1 Lessons learned
Capturing what worked, what did not, and why, is essential for building institutional knowledge and improving future AI deployments across government. We strongly recommend that teams document lessons learned.
Example questions
- What are the lessons learned from the pilot?
- What were the project challenges?
- What blockers, if any, materially changed trajectory?
- Are there any recommendations for future AI projects?
- What time-to-benefit realisation can we expect for benefits identified, for example efficiency or productivity?
- How did the tool's technology and operational readiness (TRL/ORL) evolve, and what gaps remain before wider deployment?
- How scalable is the solution, and what constraints or dependencies could limit scaling?
- What changes (technical, operational, governance, or resourcing) would be required to scale effectively?
- What risks or performance issues might emerge at scale that were not evident during the pilot?
- What is the expected timeline and pathway to scaling and realising benefits?
4.2 Environmental sustainability
AI systems consume significant computational resources, and the energy and carbon costs of training and running models at scale are increasingly material. Documenting environmental considerations from the outset ensures these impacts are visible and can be managed responsibly as deployments grow.
Example questions
- Has the environmental cost of training and inference been estimated?
- Is compute infrastructure using renewable energy where possible?
- Has data volume been minimised relative to requirements?
4.3 Long-term societal impact
AI deployments can have effects that extend well beyond the immediate users and services involved, affecting communities, labour markets, and public institutions over time. These effects are real but typically emerge beyond the pilot window, making early documentation and planning for longer-term monitoring essential.
Example questions
- Which communities or groups are likely to be affected, and how?
- Has a scenario modelling, horizon scan or risk workshop been completed?
- Is there a plan for longer-term monitoring if the tool is scaled?
3.6 Social and user impact
Social and user impact refers to the direct effects of the AI deployment on users and those around the service, including changes to confidence, anxiety, exclusion, dependency, or working relationships. These are observable through qualitative methods within a pilot even where formal measurement is not feasible.
KPI definitions