EPIC Comments on UK ICO Draft Guidance on Automated Decision-Making

EPIC has submitted feedback in response to the UK Information Commissioner’s Office (ICO) consultation on the draft guidance they have produced about automated decision-making (ADM), including profiling. The updates are meant to incorporate the additions and changes to Articles 22A-22D of the UK General Data Protection Regulation (UK GDPR) and to address common areas of confusion around ADM systems, making clear to companies how and where they can legally be used and to the public what rights they have relating to these systems.

EPIC has long engaged in policy and research discussions around ADM systems, contributing letters of support for bills with strong consumer protection against automated decision making systems, speaking at state stakeholder sessions on the issue and contributing testimony, and issuing in-depth reports on how automated decision making in one particular jurisdiction is affecting people’s lives. We also regularly engage on issues of international law and policy, including through previous comments to the ICO and beyond on proposed regulations and guidance.

The ICO guidance attempts to address an extremely narrow area of ADM – automated decision making that has a consequential effect on an individual and has no meaningful human involvement. We fear such a narrowly-drawn scope leaves a massive loophole: when human actors take an action based solely on the ADM conclusion. If the guidance does not apply in these cases, it allows the simple loophole of a human pressing a button while the ADM makes all the meaningful decisions.

In looking at the practical challenges and potential downsides and risks of these systems, we feel the ICO underestimates the time, resources, and opportunity lost by individuals fighting unfair and inaccurate decisions. Even when there is a clear way to contest a decision, the individual must ensure they are following the proper process and have the correct information. They must gather evidence for their argument and wait on a response. In the meantime, they could be without access to critical resources or services, such as public housing or food resources. In some cases, an ADM rejection may be unfixable, as when a person is passed over for a job position that is then given to someone else.

The guidance also sets forth conditions conditions that must be in place if a company wants to use children’s personal information to “influenc[e] their choices or behavior,” implying that this may be permissible in some circumstances. There are no conditions that make manipulating children acceptable. None. ADM must be flatly rejected for any use on children.

At its core, the guidance hinges on the assumption that ADMs will make critical processes more efficient, effective, and fair. This assumption is not borne out
by experience or studies. The changes in speed and ‘precision’ enabled by ADM are undercut by its lack of transparency and the cost of constant review and correction of its mistakes. Moreover, the systems are not built for the kind of accountability or explainability that can be exercised with humans. While human review also is subject to bias and mistakes, a human can be asked to explain their reasoning and can be assigned clear liability when things go wrong. ADMs based on generative AI are unable to offer similar explanations and it is unclear where liable and responsibility will be placed when the systems inevitably malfunction or err.

Finally, the guidance itself states that that there will always be a margin of error in profiling. We ask the ICO what is an acceptable margin of error for life-changing decisions made by machines about individuals? Is it one person unfairly incarcerated, denied work, or rejected from housing? Two? When the risks are this severe and we can see the human costs, arguments about efficiency ring hollow.

EPIC regularly engages on work around AI systems, international policy, and human rights and dignity. For more, please contact us or check our website at epic.org.