Information Commissioner's Office
Reuben Binns, our Research Fellow in Artificial Intelligence (AI), and Valeria Gallo, Technology Policy adviser, discuss how using AI can require trade-offs between data protection principles, and what organisations can do to assess and balance them.
This post is part of our ongoing Call for Input on developing the ICO framework for auditing AI. We encourage you to share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
AI systems must satisfy a number data protection principles and requirements, which may at times pull organisations in different directions. For example, while more data can make AI systems more accurate, collecting more personal information has implications in terms of erosion of privacy.
Organisations using AI need to identify and assess such trade-offs, and strike an appropriate balance between competing requirements.
The right balance in any particular trade-off will depend on the specific sectoral and social context an organisation operates in, and the impact on data subjects. However, in this blog we discuss some considerations for assessing and mitigating trade-offs that are relevant across use cases.
We start off with a short overview of the most notable trade-offs that organisations are likely to face when designing or procuring AI systems.
Privacy vs accuracy
Machine Learning (ML) uses statistical models to predict behaviour or classify people. In general terms, the more data is used to train and run the ML model, the more likely it is for the latter to capture any underlying, statistically meaningful relationships between the features in the datasets.
For instance, a model for predicting future purchases based on customers’ purchase history will tend to be more accurate the more customers are included in the training data. And any new features added to an existing dataset may be relevant to what the model is trying to predict; for instance, purchase histories augmented with additional demographic data might further improve the predictive accuracy of the model.
However, collecting additional personal data can have an adverse impact on the privacy. The more individuals included in the dataset, the more information collected about each person, the greater the impact.
Accuracy and Fairness
As we discussed recently, the use of AI systems can lead to biased or discriminatory outcomes. Organisations can put in place various technical measures to mitigate this risk, but most of these techniques also tend to reduce the accuracy of the AI outputs.
For example, if an anti-classification definition of fairness is applied to an AI credit risk model, any protected characteristics, as well as known proxies (eg postcode as a proxy for race) would need to be excluded from consideration by the model. This may help prevent discriminatory outcomes, but it may also result in a less accurate credit risk measurement. This is because the postcode may also have been a proxy for legitimate credit risk features, for example job security, which would have increased the model’s accuracy.
There may not always be a trade-off between accuracy and fairness. For example, if discriminatory outcomes in the model are driven by a relative lack of data on a minority population, then both fairness and accuracy could be increased by collecting more relevant data.
However, in that case, the organisation would face a different trade-off, between privacy and fairness.
Privacy and Fairness
Privacy and fairness might conflict in two ways. Firstly, as described above, an organisation may find that its system is unfair, due in part to a relative lack of data on a minority population. In such cases, it may want to collect data on more people from such groups so that its system is more accurate on them.
Secondly, in order to test whether an AI system is discriminatory, it would normally be necessary to collect data on protected characteristics. For instance, to measure whether a statistical model has substantially different error rates between individuals with different protected characteristics, it will need to be tested with data that contains labels for those characteristics. If this data needs to be collected in order to perform the testing, then the organisation faces a trade-off between privacy (not collecting those characteristics) and fairness (using them to test the system and make it fairer).
Explainability and Accuracy
As discussed in the interim report of our ExplAIn Project, the trade-off between the explainability and accuracy of AI decisions may often be a false dichotomy.
Very simple AI systems can be highly explainable. In simple and relatively small decision trees, for example, it is relatively easy to understand how inputs relate to outputs. And although it is more challenging, there are also ways to explain more complicated AI decision-making systems.
Nevertheless very complex systems, such as those based on deep learning, can make it hard to follow the logic of the system. In such cases, there may be a trade-off between accuracy and explainability, which will be also considered in greater depth in the ExplAIn project final guidance.
Explainability and Security
Providing data subjects with explanations about the logic of an AI system can potentially increase the risk of inadvertently disclosing private information in the process.
Recent research has demonstrated how some proposed methods to make ML models explainable can unintentionally make it easier to infer private information about the individuals whose personal data the model was trained on. This is a topic which we will cover in a future upcoming blog on privacy attacks on ML models.
Some literature also highlights the risk that in the course of providing an explanation to data subjects, organisations may reveal proprietary information about how an AI model works. Our research and stakeholder engagement so far indicate this risk is quite low. However, in theory at least, there may be cases where a trade-off will need to be struck regarding the right of individuals to receive an explanation, and the right of organisations to maintain trade secrets.
Both of these risks are active areas of research, and it is not yet known how likely and severe they are likely to become. Organisations should monitor the latest research and consider realistic threat models in their given context.
Latest News from
Information Commissioner's Office
Statement regarding the government’s initial response to Online Harms White Paper consultation13/02/2020 09:10:00
Elizabeth Denham, Information Commissioner, yesterday gave a statement regarding the government’s initial response to Online Harms White Paper consultation.
Joint statement warning FCA-authorised firms and insolvency practitioners to be responsible when dealing with personal data10/02/2020 09:10:00
Joint statement from the Financial Conduct Authority (FCA), the Information Commissioner’s Office (ICO) and the Financial Services Compensation Scheme (FSCS) (07 February 2020).
ICO celebrates excellence in data protection with third annual award for practitioners05/02/2020 12:25:00
The Information Commissioner is looking for data protection practitioners who have made an outstanding impact within their organisation.
Statement on data protection and Brexit implementation – what you need to do30/01/2020 12:25:00
The UK will leave the European Union on 31 January and enter a Brexit transition period.
ICO launches latest phase of privacy innovation grants programme29/01/2020 12:25:00
Applications are now open for the third round of funding from the Information Commissioner’s Office’s (ICO) grants programme.
Data Protection Day 202028/01/2020 11:43:00
The ICO marked this year’s annual Data Protection Day (27 January 2020) by highlighting data sharing resources and guidance.
ICO statement in response to an announcement made by the Metropolitan Police Service on the use of live facial recognition24/01/2020 15:15:00
In October 2019 we concluded our investigation into how police use live facial recognition technology (LFR) in public places.
ICO's blog on its information rights work23/01/2020 16:10:00
Colleagues from the ICO’s access to information and compliance department share their experiences and involvement in raising awareness of our regulation of access to information legislation.