New Guidance for Evaluating the Impact of AI Tools

The Evaluation Task Force have recently published a new annex to the Magenta Book which covers best practice for impact evaluation of AI tools and technologies.

In December the Evaluation Task Force published a new annex to the Magenta Book, focusing on best practice for evaluating the impact of AI evaluation methods (click here to read the guidance). The guidance will enhance the safety and confidence with which government departments and agencies can adopt AI technologies, ensuring that public sector innovation keeps pace with the private sector. It reflects an understanding of the unique challenges posed by AI and the need for tailored approaches to address these challenges.

The guidance has been coproduced with the Department for Transport and Frontier Economics, in consultation with leading AI specialists. It is expected to be a valuable resource for policymakers, public sector professionals, and digital specialists working to integrate AI solutions into government operations. Moving forwards, the guidance will be co-owned with the Central Digital and Data Office (CDDO). 

The guidance details best practice, including evaluation design, methodology, and timing, for evaluating the impact of new AI tools and technologies being introduced in the public sector. In particular, it advocates for the use of Randomised Control Trials when testing a new AI product to produce high quality evidence on the intended and unintended impacts of introducing these new technologies. The guidance also includes a series of hypothetical case studies to illustrate possible high-quality approaches to evaluating the impact of different types of AI tools.