Information Commissioner's Office
Data Protection Impact Assessments and AI
Simon Reader, Senior Policy Officer, discusses some of the key considerations for organisations undertaking data protection impact assessments for Artificial Intelligence (AI) systems.
This post is part of our ongoing Call for Input on developing the ICO framework for auditing AI. We encourage you to share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
Several of the blogs in this series have referred to the importance of carrying out a Data Protection Impact Assessment (DPIA) for AI systems that will process personal data. DPIAs offer an organisations an opportunity to consider how and why using AI systems to process personal data and what the potential risks could be.
The ICO has produced detailed guidance on DPIAs that explains when they are required and how to complete them. This blog sets out some of the things organisations should think about when carrying out a DPIA for the processing of personal data in AI systems.
DPIAs under the General Data Protection Regulation (GDPR)
The GDPR states that, DPIAs are required (at least)
- before the deployment of innovative technological solutions;
- for the processing of special category personal data at large scale; or
- for automated decision-making, profiling, or for the expected denial of a service to an individual.
The use of AI for processing personal data will therefore usually meet the legal requirement for completing a DPIA.
If the result of an assessment indicates residual high risk to individuals that cannot be reduced, data controllers must consult with the ICO.
What should be assessed in a DPIA
A DPIA needs to describe the nature, scope, context and purposes of any processing of personal data.
It needs to make clear how and why AI is going to be used to process the data. It will need to detail:
- how data will be collected, stored and used;
- the volume, variety and sensitivity of the input data;
- the nature of the data controller’s relationship with data subjects; and
- the intended outcomes for individuals or wider society and for the data controller.
In the context of the AI lifecycle, a DPIA will best serve its purpose if undertaken at the earliest stages of project development. It should feature, at a minimum, the following key components.
1. A systematic description of the processing
A DPIA should include a systematic description of the processing activity, including data flows and the stages when AI processes and automated decisions may produce effects on individuals. It can also explain any relevant variation or margins of error.
Where automated decisions are subject to human intervention or review, the processes being implement to ensure this is meaningful and that decisions can be overturned should also be detailed.
Unless there is a good reason not to do so, organisations should seek and document the views of individuals, or their representatives, on the intended processing operation during a DPIA. It is therefore important to be able to describe the processing in a way that is accessible to those who are consulted.
However, it can be difficult to describe the processing activity of a complex AI system. It may be appropriate therefore, to maintain two versions of an assessment. The first presenting a thorough technical description for specialist audiences. The second containing a more high-level description of the processing and explaining the logic of how the personal data inputs relate to the outputs affecting individuals.
A DPIA should set out the roles and obligations of the data controller and any processors. Where AI systems are partly or wholly outsourced to external providers, both organisations should also assess whether joint controllership has been established under Article 26 of the GDPR; and if so, to collaborate in the DPIA process as appropriate.
Where a data processor is used, some of the more technical elements of the processing activity can be illustrated in a DPIA by reproducing information from that processor. For example, a flow diagram from a processor’s manual. However, the data controller should generally avoid copying large sections of a processor’s literature into their own assessment.
2. Assessing necessity and proportionality
The deployment of an AI system to process personal data needs to be driven by the proven ability of that system to fulfil a specific and legitimate purpose; not by the availability of the technology. By assessing necessity in a DPIA, an organisation can evidence that these purposes couldn’t be accomplished in another reasonable way.
By undertaking a DPIA, organisations can also demonstrate that the processing of personal data by an AI system is a proportionate activity. When assessing proportionality, the interests of the organisation need to be weighed up against the rights and freedoms of individuals. In relation to AI systems, organisations need to think about any detriment to data subjects that could follow from bias or inaccuracy in the algorithms and data sets being used.
Within the proportionality element of a DPIA, organisations need to assess whether data subjects would reasonably expect the processing to be conducted by an AI system. If AI systems complement or replace human decision-making, it should be documented in the DPIA how the project might compare human and algorithmic accuracy side-by-side to better justify its use.
Organisations should also describe any trade-offs that are made, for example between accuracy and data minimisation, and document the methodology and rationale for these.
3. Identifying risks to rights and freedoms
The use of personal data in the development and deployment of AI systems may not just pose risks to individual’s privacy and data protection rights.
For instance, machine learning systems may reproduce discrimination from historic patterns in data, which could fall foul of equalities legislation. Similarly, AI systems that stop content being published based on the analysis of the creator’s personal data could impact their freedom of expression. In such contexts, data controllers should consider the relevant legal frameworks beyond data protection.
The DPIA process will help organisations to objectively identify the relevant risks. A score or level should be assigned to each risk, measured against the likelihood and the severity of the impact on data subjects.
4. Measures to address the risks
It is important that data protection officers and other information governance professionals are involved in AI projects from the earliest stages. Clear and open channels of communication must be established between them and the project teams. This will ensure that risks can be identified and addressed early in the AI lifecycle.
Data protection should not be an afterthought, and a DPO’s professional opinion should not come as a surprise at the eleventh hour.
A DPIA can be used to document the safeguards put in place to ensure the individuals responsible for the development, testing, validation, deployment, and monitoring of AI systems are adequately trained and have an appreciation for the data protection implications of the processing.
Organisational measures to ensure that appropriate training is in place to mitigate risks associated with human error can also be evidenced in a DPIA. Along with the technical measures designed to reduce risks to the security and accuracy of an AI system.
Once measures have been introduced to mitigate the risks identified, the DPIA should document the residual levels of risk posed by the processing. These must be referred to the ICO for prior consultation if they remain high.
5. A ‘living’ document
While any DPIA must be carried out before the processing of personal data begins, they should be considered a ‘live’ document. This means they are subject to regular review or re-assessment should the nature, scope, context or purpose of the processing alter for any reason.
For instance, depending on the deployment, it could be that the demographics of the target population may shift, or that people adjust their behaviour over time in response to the processing itself.
Questions for comment
We are very interested to hear about how organisations are approaching DPIAs in the context of AI. We would particularly like to know:
- if you feel your organisation needs to take substantially different approach to DPIAs in relation to AI systems?
- whether your organisation has a policy of publishing DPIAs for AI systems and why you do/ do not?
Please share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
Latest News from
Information Commissioner's Office
ICO call for views on the application for powers under the Proceeds of Crime Act11/11/2019 09:10:00
The Information Commissioner invites views on her office being granted access to investigation and other associated powers under the Proceeds of Crime Act 2002 (POCA).
Information Commissioner reminds political parties they must comply with the law ahead of General Election06/11/2019 09:10:00
The Information Commissioner has sent the following letter to the political parties in relation to the use of data in political campaigning.
Blog: Live facial recognition technology – police forces need to slow down and justify its use31/10/2019 13:10:00
Blog posted by: Elizabeth Denham, Information Commissioner, 31 October 2019.
Statement on an agreement reached between Facebook and the ICO30/10/2019 15:10:00
In 2017 the Information Commissioner's Office ("ICO") commenced a formal investigation into the misuse of personal data in political campaigns.
Blog: Embedding accountability – we want to hear from you29/10/2019 13:20:00
Blog posted by: Ian Hulme, Director for Regulatory Assurance, 28 October 2019.
AI Auditing Framework Call for Input: final considerations and next steps29/10/2019 09:10:00
As the initial Call for Input into the development of the ICO AI Auditing Framework comes to an end, Simon McDougall, Executive Director for Technology and Innovation, reflects on some of the overarching themes that have emerged in the first phase of our work.
Blog: How coming to work at the ICO on a secondment can benefit both of us22/10/2019 15:05:00
Blog posted by: Simon McDougall, ICO Executive Director for Technology and Innovation, 22 October 2019.
Blog: Global collaboration and how it helps the UK22/10/2019 12:20:00
Blog posted by: Elizabeth Denham, Information Commissioner, 21 October 2019.