Data Protection Impact Assessments and AI

23 Oct 2019 12:44 PM

Simon Reader, Senior Policy Officer, discusses some of the key considerations for organisations undertaking data protection impact assessments for Artificial Intelligence (AI) systems.

This post is part of our ongoing Call for Input on developing the ICO framework for auditing AI. We encourage you to share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk. 

Several of the blogs in this series have referred to the importance of carrying out a Data Protection Impact Assessment (DPIA) for AI systems that will process personal data. DPIAs offer an organisations an opportunity to consider how and why using AI systems to process personal data and what the potential risks could be.

The ICO has produced detailed guidance on DPIAs that explains when they are required and how to complete them. This blog sets out some of the things organisations should think about when carrying out a DPIA for the processing of personal data in AI systems. 

DPIAs under the General Data Protection Regulation (GDPR)

The GDPR states that, DPIAs are required (at least)

The use of AI for processing personal data will therefore usually meet the legal requirement for completing a DPIA. 

If the result of an assessment indicates residual high risk to individuals that cannot be reduced, data controllers must consult with the ICO. 

What should be assessed in a DPIA

A DPIA needs to describe the nature, scope, context and purposes of any processing of personal data.

It needs to make clear how and why AI is going to be used to process the data. It will need to detail:

In the context of the AI lifecycle, a DPIA will best serve its purpose if undertaken at the earliest stages of project development. It should feature, at a minimum, the following key components. 

1. A systematic description of the processing

A DPIA should include a systematic description of the processing activity, including data flows and the stages when AI processes and automated decisions may produce effects on individuals. It can also explain any relevant variation or margins of error. 

Where automated decisions are subject to human intervention or review, the processes being implement to ensure this is meaningful and that decisions can be overturned should also be detailed. 

Unless there is a good reason not to do so, organisations should seek and document the views of individuals, or their representatives, on the intended processing operation during a DPIA. It is therefore important to be able to describe the processing in a way that is accessible to those who are consulted. 

However, it can be difficult to describe the processing activity of a complex AI system. It may be appropriate therefore, to maintain two versions of an assessment. The first presenting a thorough technical description for specialist audiences. The second containing a more high-level description of the processing and explaining the logic of how the personal data inputs relate to the outputs affecting individuals.

A DPIA should set out the roles and obligations of the data controller and any processors. Where AI systems are partly or wholly outsourced to external providers, both organisations should also assess whether joint controllership has been established under Article 26 of the GDPR; and if so, to collaborate in the DPIA process as appropriate. 

Where a data processor is used, some of the more technical elements of the processing activity can be illustrated in a DPIA by reproducing information from that processor. For example, a flow diagram from a processor’s manual. However, the data controller should generally avoid copying large sections of a processor’s literature into their own assessment.

2. Assessing necessity and proportionality 

The deployment of an AI system to process personal data needs to be driven by the proven ability of that system to fulfil a specific and legitimate purpose; not by the availability of the technology. By assessing necessity in a DPIA, an organisation can evidence that these purposes couldn’t be accomplished in another reasonable way. 

By undertaking a DPIA, organisations can also demonstrate that the processing of personal data by an AI system is a proportionate activity. When assessing proportionality, the interests of the organisation need to be weighed up against the rights and freedoms of individuals. In relation to AI systems, organisations need to think about any detriment to data subjects that could follow from bias or inaccuracy in the algorithms and data sets being used.

Within the proportionality element of a DPIA, organisations need to assess whether data subjects would reasonably expect the processing to be conducted by an AI system. If AI systems complement or replace human decision-making, it should be documented in the DPIA how the project might compare human and algorithmic accuracy side-by-side to better justify its use. 

Organisations should also describe any trade-offs that are made, for example between accuracy and data minimisation, and document the methodology and rationale for these.

3. Identifying risks to rights and freedoms

The use of personal data in the development and deployment of AI systems may not just pose risks to individual’s privacy and data protection rights.

For instance, machine learning systems may reproduce discrimination from historic patterns in data, which could fall foul of equalities legislation. Similarly, AI systems that stop content being published based on the analysis of the creator’s personal data could impact their freedom of expression. In such contexts, data controllers should consider the relevant legal frameworks beyond data protection.

The DPIA process will help organisations to objectively identify the relevant risks. A score or level should be assigned to each risk, measured against the likelihood and the severity of the impact on data subjects. 

4. Measures to address the risks

It is important that data protection officers and other information governance professionals are involved in AI projects from the earliest stages. Clear and open channels of communication must be established between them and the project teams. This will ensure that risks can be identified and addressed early in the AI lifecycle.

Data protection should not be an afterthought, and a DPO’s professional opinion should not come as a surprise at the eleventh hour.

A DPIA can be used to document the safeguards put in place to ensure the individuals responsible for the development, testing, validation, deployment, and monitoring of AI systems are adequately trained and have an appreciation for the data protection implications of the processing.

Organisational measures to ensure that appropriate training is in place to mitigate risks associated with human error can also be evidenced in a DPIA. Along with the technical measures designed to reduce risks to the security and accuracy of an AI system.

Once measures have been introduced to mitigate the risks identified, the DPIA should document the residual levels of risk posed by the processing. These must be referred to the ICO for prior consultation if they remain high.

5. A ‘living’ document

While any DPIA must be carried out before the processing of personal data begins, they should be considered a ‘live’ document. This means they are subject to regular review or re-assessment should the nature, scope, context or purpose of the processing alter for any reason.

For instance, depending on the deployment, it could be that the demographics of the target population may shift, or that people adjust their behaviour over time in response to the processing itself.

Questions for comment 

We are very interested to hear about how organisations are approaching DPIAs in the context of AI. We would particularly like to know:

Please share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.