Here’s the reason why you can never design the perfect explanation from an AI

Sign up for our newsletter, Artificiality, to get our latest AI insights delivered directly to your email inbox.

Explainable AI is a foundational concept and one that everyone agrees is important. In US law, citizens have a right to an explanation when an algorithm is used to make credit decisions, for example. But beyond certain legal rights, a user-centric explanation and justification is simply good design.

Recent research in this area has revealed yet another AI irony – the autonomy paradox. In order to respect someone’s autonomy, the designer of the AI must make certain assumptions about what information will be valuable to users. There are many different reasons why the designer cannot know everything that could be important – the realities of life for users are usually not fully imaginable or necessarily even finite. There may not be a clear link between the recommendation made by the AI and an action in the real world. And there’s no way for a designer to know how outcomes vary from person to person.

All this means that the choices that designers need to make about what to disclose and how to explain an AI decision can have unintended consequences for users, consequences which could have been avoided if the users had disclosed different information about themselves.

This research reveals a basic power imbalance that is not easily remedied: given the informational position of the designer, there is simply no way to fully maintain commitment to a user’s autonomy.

The answer that many reach for (and indeed, perhaps one of the reasons why it’s taken until recently for this paradox to be revealed) is to collect more data. Maybe all of it! But this isn’t a solution. The very act of collecting more data disrupts a person’s autonomy because privacy is fundamental to autonomy. Hence the paradox.

This invites the question: is there any way that giving up more information can be autonomy-enhancing? The answer depends on the power structure and its underlying incentives. For example, we give up highly personal information to professionals all the time in ways that increase our decision making ability. Many of these people are legally required to act in our best interest – for example, lawyers. So here’s the insight – resolving the autonomy paradox relies more on the relative power and constraints on the decision maker than it does on the quality of the explanation.

This work highlights the unique difficulties with AI explanations. The designer (and by implication, the decision maker) has no choice but to make decisions about what to explain and how to factor in various assumptions about the real world. The right (or desire) for an explanation from AI sets up unintended power which accrues to the designer – whenever there is ambiguity in an individual’s preferences, the designer has the power to resolve the ambiguity however they choose. The authors note:

“This leaves the decision maker with significant room to maneuver, the choice of when and where to further investigate, and more degrees of freedom to make choices that promote their own welfare than we might realize.”

Solon Barocas, Andrew D. Selbst, and Manish Raghavan

This concern exists with any complex algorithmic system but machine learning makes it worse, in part due to the nature of explanation itself. Broadly speaking, there are two types of explanation – principal reason and counter-factual.

Principal reason explanations tell users what dominated a decision. Principal reason explanations come from the law (as opposed from computer science) and lack precision as they do not make use of decision boundaries. The intent is education; they are more justification than guidance.

Counter-factual explanations are intended to be more practical and actionable. They provide an “if you change this, then that” type of explanation. Some explanations go so far as to seem like promises – do a thing and get this thing.

In practice, AI explanations can never map clearly to actions. There are simply too many interdependent variables at play. Add to this that new data changes decision boundaries and the model output can drift against the explanation architecture itself. Only the designer (writ data scientist) has enough understanding of the domain and causal features to make the necessary decisions about what matters and how to explain the most relevant relationships.

This leaves us with the role of UX design. Here it gets even more interesting. There are two possible approaches to deal with the autonomy paradox – make more interactive UX for the decision subjects (users) and/or make more interactive UX for the decision makers (designers).

In designing a UX for decision subjects (ie users), more interactive tools can allow users to explore the effect of making changes to certain features. This gives the user a greater sense of freedom and allows them to maintain autonomy because they can play around; it is their own knowledge of their own constraints and choices that matters.

In designing a UX for decision makers (ie, themselves), the decision makers can go about finding out even more information about users; what do they like/not like, what other preferences do they have?

Both of these approaches run into the next constraint: revealing enough of the model to be able to reconstruct it. Too much transparency can raise IP, proprietary or trade secret issues. It also incentives gaming and, because AI can be more correlation than causation, a successful gaming strategy may not map to a successful real-world outcome. So while a user may cheat the model, they may only be cheating themselves.

Ugh, is there an end to this? Research into the UX of explanation is on-going and we can expect much more to come. From a regulatory perspective, the idea that there is now an “information fiduciary” role is an interesting and viable path to consider. Even if it doesn’t become law, it’s got legs as human-centric design.

Photo by Thiago Cardoso on Unsplash

About Sonder Scheme: We are on a mission to help humans win in the age of AI by making AI design easier, more inclusive, more ethical and more human. Through our workshops, learning journeys and the Sonder Scheme Studio, we enable people to use design thinking-based tools to create innovative, effective and ethical human-machine systems. Both offline and online, Sonder Scheme empowers companies around the world to design human-centered AI.

Share on email
Share on facebook
Share on linkedin
Share on twitter