Information Commissioner's Office
Automated Decision Making: the role of meaningful human reviews
In the first detailed element of our AI framework blog series, Reuben Binns, our Research Fellow in AI, and Valeria Gallo, Technology Policy Adviser, explore how organisations can ensure ‘meaningful’ human involvement to make sure AI decisions are not classified as solely automated by mistake.
This blog forms part of our ongoing work on developing a framework for auditing AI. We are keen to hear your views in the comments below or you can email us.
Artificial Intelligence (AI) systems often process personal data to either support or make a decision. For example, AI could be used to approve or reject a financial loan automatically, or support recruitment teams to identify interview candidates by ranking job applications.
Article 22 of the General Data Protection Regulation (GDPR) establishes very strict conditions in relation to AI systems that make solely automated decisions, ie without human input, with legal or similarly significant effects about individuals. AI systems that only support or enhance human decision-making are not subject to these conditions. However, a decision will not fall outside the scope of Article 22 just because a human has ‘rubber-stamped’ it: human input needs to be ‘meaningful’.
The degree and quality of human review and intervention before a final decision is made about an individual is the key factor in determining whether an AI system is solely or non-solely automated.
Board members, data scientists, business owners, and oversight functions, among others, will be expected to play an active role in ensuring that AI applications are designed, built, and used as intended.
The meaningfulness of human review in non-solely automated AI applications and the management of the risks associated with it are key areas of focus for our proposed AI Auditing Framework and what we will be exploring further in this blog.
What’s already been said?
- Human reviewers must be involved in checking the system’s recommendation and should not “routinely” apply the automated recommendation to an individual;
- reviewers’ involvement must be active and not just a token gesture. They should have actual “meaningful” influence on the decision, including the “authority and competence” to go against the recommendation; and
- reviewers must ‘weigh-up’ and ‘interpret’ the recommendation, consider all available input data, and also take into account other additional factors’.
Are there additional risk factors in complex systems?
The meaningfulness of human input must be considered in any automated decision-making systems however basic (e.g. simple decision trees). In more complex AI systems however, we think there are two additional factors that could potentially cause a system to be considered solely-automated. They are:
- Automation bias
- Lack of interpretability
Latest News from
Information Commissioner's Office
Statement in response to media enquiries about the Data Protection Impact Assessment for the NHSX’s trial of contact tracing app11/05/2020 09:15:00
An ICO spokesperson said: “We are reviewing the Data Protection Impact Assessment for NHSX’s pilot of its contact tracing app in the Isle of Wight.”
Blog: Information Commissioner sets out new priorities for UK data protection during COVID-19 and beyond06/05/2020 09:10:00
Blog posted by: Elizabeth Denham, Information Commissioner, 05 May 2020.
COVID-19 contact tracing: data protection expectations on app development05/05/2020 09:10:00
Information Commissioner Elizabeth Denham and Executive Director of Technology and Innovation Simon McDougall appeared before the Human Rights Joint Committee yesterday (4 May 2020).
Statement in response to details about an NHSX contact tracing app to help deal with the COVID-19 pandemic27/04/2020 09:10:00
Statement given recently (24 April 2020) in response to details about an NHSX contact tracing app to help deal with the COVID-19 pandemic.
Blog: Combatting COVID-19 through data: some considerations for privacy20/04/2020 09:10:00
Blog posted by: Elizabeth Denham, Information Commissioner, 17 April 2020.
Blog: Video conferencing: what to watch out for17/04/2020 09:10:00
The COVID-19 crisis is changing the way we live our lives. Keeping our distance means many of us are working from home for the first time and adapting to new ways of doing our jobs.
How we will regulate during coronavirus16/04/2020 09:10:00
The ICO yesterday published a document setting out our regulatory approach during the coronavirus pandemic.
ICO statement on investigating coronavirus scams09/04/2020 09:10:00
ICO are supporting businesses eager to stay in touch with customers during the Covid-19 pandemic.
Winner of the ICO’s Practitioner Award for Excellence in Data Protection 2020 announced07/04/2020 12:25:00
Recognising the increasingly vital role played by data protection professionals, the third ICO Practitioner Award for Excellence in Data Protection is awarded to Barry Moult, Information Governance and Privacy Consultant, and former Head of Information Governance at an NHS Trust.