Information Commissioner's Office
Fully automated decision making AI systems: the right to human intervention and other safeguards
Reuben Binns, our Research Fellow in Artificial Intelligence (AI), and Valeria Gallo, Technology Policy Adviser, discuss some of the key safeguards organisations should implement when using solely automated AI systems to make decisions with significant impacts on data subjects.
This post is part of our ongoing Call for Input on developing the ICO framework for auditing AI. We encourage you to share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
The General Data Protection Regulation (GDPR) requires organisations to implement suitable safeguards when processing personal data to make solely automated decisions that have a legal or similarly significant impact on individuals. These safeguards include the right for data subjects:
- to obtain human intervention;
- to express their point of view; and
- to contest the decision made about them.
These safeguards cannot be token gestures. Guidance published by the European Data Protection Board (EDPB) states that human intervention involve a review of the decision, which
“must be carried out by someone who has the appropriate authority and capability to change the decision”.
The review should include a
“thorough assessment of all the relevant data, including any additional information provided by the data subject.”
In this respect, the conditions under which human intervention will qualify as meaningful are similar to those that apply to human oversight in ‘non-solely automated’ systems. However, a key difference is that in solely automated contexts, human intervention is only required on a case-by-case basis to safeguard the data subject’s rights.
Why is this a particular issue for AI systems?
The type and complexity of the systems involved in making solely automated decisions will affect the nature and severity of the risk to people’s data protection rights and will raise different considerations, as well as compliance and risk management challenges.
Basic systems, which automate a relatively small number of explicitly written rules (eg a set of clearly expressed ‘if-then’ rules to determine a customer’s eligibility for a product) are unlikely to be considered AI. It should also be relatively easy for a human reviewer to identify and rectify any mistake, if a decision is challenged by a data subject because of system’s high interpretability.
However other systems, such as those based on machine learning (ML), may be more complex and present more challenges for meaningful human review. ML systems make predictions or classifications about people based on data patterns. Even when they are highly accurate, they will occasionally reach the wrong decision in an individual case. Errors may not be easy for a human reviewer to identify, understand or fix.
While not every challenge on the part of data subject will be valid, organisations should expect that many could be. There are two particular reasons why this may be the case in ML systems:
- The individual is an ‘outlier’, ie their circumstances are substantially different from those considered in the training data used to build the AI system. Because the ML model has not been trained on enough data about similar individuals, it can make incorrect predictions or classifications.
- Assumptions in the AI design can be challenged, for example a continuous variable such as age, might have been broken up (‘binned’) into discrete age ranges, eg 20-39, as part of the modelling process. Finer-grained ‘bins’ may result in a different model with substantially different predictions for people of different ages. The validity of this data pre-processing and other design choices may only come into question as a result of an individual’s challenge.
What should organisations do?
Many of the controls required to ensure compliance with the GDPR’s provisions on solely automated systems are very similar to those necessary to ensure the meaningfulness of human reviews in non-solely automated AI systems.
- consider the system requirements necessary to support a meaningful human review from the design phase. Particularly, the interpretability requirements and effective user-interface design to support human reviews and interventions;
- design and deliver appropriate training and support for human reviewers; and
- give staff the appropriate authority, incentives and support to address or escalate data subjects’ concerns and, if necessary, override the AI system’s decision.
- The use of solely automated systems to make decisions with legal or significant effects on data subjects will always trigger the need for a Data Protection Impact Assessment(DPIA). DPIAs are a compliance requirement, but also a helpful tool for organisations to reflect carefully on the appropriateness of deploying a solely automated process. In the case of AI systems, DPIAs should give particular consideration to the level of complexity and interpretability of the system, and the organisation’s ability to adequately protect individuals and their rights.
- Our ExplAIn project is currently looking at how, and to what extent, complex AI systems might affect an organisation’s ability to provide meaningful explanations to data subjects. However, complex AI systems can also impact the effectiveness of other mandatory safeguards. If a system is too complex to explain, it may also be too complex to meaningfully contest, to intervene on, to review, or to put an alternative point of view against. For instance, if an AI system uses hundreds of features and a complex, non-linear model to make a prediction, then it may be difficult for a data subject to determine which variables or correlations to object to. Therefore safeguards around solely automated AI systems are mutually supportive, and should be designed holistically and with the data subject in mind.
- The information about the logic of a system and explanations of decisions should give data subjects the necessary context to decide whether, and on what grounds, they would like to request human intervention. In some cases, insufficient explanations may prompt data subjects to resort to other rights unnecessarily. Requests for intervention, expression of views, or contests are more likely to happen if data subjects don’t feel they have a sufficient understanding of how the decision was reached.
- The process for data subjects to exercise their rights should be simple and user friendly. For instance, if the result of the solely automated decision is communicated through a website, the page should contain a link or clear information allowing the individual to contact a member of staff who can intervene, without any undue delays or complications. Organisations are expected to keep a record of all decisions made by an AI system, as well as whether a data subject requested human intervention, expressed any views, contested the decision, and whether the decision was changed as a result.
- Organisations should monitor and analyse this data. If decisions are regularly changed in response to data subjects exercising their rights, organisations will be expected to amend their systems accordingly. Where the system is based on ML, this might involve including the corrected decisions into fresh training data, so that similar mistakes are less likely to happen in future. More substantially, they may identify a need to collect more or better training data to fill in the gaps that led to the erroneous decision, or modify the model-building process, ie by changing the feature selection.
In addition to being a compliance requirement, this is also an opportunity for organisations to improve the performance of their AI systems and, in turn, build data subjects’ trust in them. However, if grave or frequent mistakes are identified, organisations will need to take immediate steps to understand and rectify the underlying issues and, if necessary, suspend the use of the automated system.
We would like to hear your views on this topic and genuinely welcome any feedback on our current thinking. Please share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
Latest News from
Information Commissioner's Office
ICO statement in response to an announcement made by the Metropolitan Police Service on the use of live facial recognition24/01/2020 15:15:00
In October 2019 we concluded our investigation into how police use live facial recognition technology (LFR) in public places.
ICO's blog on its information rights work23/01/2020 16:10:00
Colleagues from the ICO’s access to information and compliance department share their experiences and involvement in raising awareness of our regulation of access to information legislation.
ICO publishes Code of Practice to protect children’s privacy online22/01/2020 16:33:00
The Information Commissioner’s Office yesterday published its final Age Appropriate Design Code – a set of 15 standards that online services should meet to protect children’s privacy.
Blog: Adtech - the reform of real time bidding has started and will continue17/01/2020 16:25:00
A blog by Simon McDougall, ICO Executive Director of Technology and Innovation
National retailer fined half a million pounds for failing to secure information of at least 14 million people10/01/2020 13:25:00
The Information Commissioner’s Office (ICO) has fined DSG Retail Limited (DSG) £500,000 after a ‘point of sale’ computer system was compromised as a result of a cyber-attack, affecting at least 14 million people.
ICO launches consultation on draft direct marketing code of practice09/01/2020 09:10:00
The Information Commissioner's Office (ICO) has launched a public consultation on a draft direct marketing code of practice.
Blog: The benefits of sharing personal data – what can we learn from Open Banking?07/01/2020 13:20:00
The ICO’s Regulators’ Business Innovation Privacy Hub has recently been looking at the key data protection considerations for innovators who are working in the Open Banking space.
Trust, technology and slippers with torches02/01/2020 14:10:00
Jonathan Bamford holds up a tatty bundle of papers. They’re scrumpled, time worn, ripped and held together with yellowing Sellotape, but with the Royal coat of arms crown still proudly visible on the cover.