How people react to over-personalization

We are all individuals. We all feel we have a unique core identity. We are not interchangeable. It is this truth that lies at the heart of problematic over-personalization.

Algorithms use data sourced from the masses to draw highly accurate predictions about individual preferences and behaviors. According to recent research, this can challenge our sense of individuality. When consumers believe their future choices can be predicted by a machine based on their past choices, some will assert their autonomy by choosing a less-preferred option. To reestablish their own autonomy, these individuals would rather violate their own preferences by choosing to be unpredictable.

This idea has many implications for AI designers.

If interactions are too efficient, users have no ability to express their individuality. For brands, this can be a missed opportunity to help customers express their autonomy and deepen their emotional connection with a product or service. Designers should consider ways that customers can interact with AI such that they feel their experience is theirs alone.

The desire for autonomy results in a unique challenge for AI designers — machines aren’t good at anticipating human capriciousness. How can a finely-honed machine learning algorithm account for a user’s “freedom to do otherwise?” The more a company expects a customer to act by following their past choices, the more some will want to break with those past choices. The researchers suggest some simple tips — say, messaging with agency-supportive language such as “consistent” as compared to “predictable” which can be perceived as agency-decreasing. It’s a good idea to invite customers to pursue an already-chosen path rather than risk a customer deviating from it in order to assert their autonomy.

Once again, context proves vital to successful personalization. Sometimes deviating from past choices is in the customer’s best interest. Algorithms can detect a specific pattern in buying behavior called “precommitment” when someone is ready to make a positive lifestyle change, say to their diet. AI can be agency-enhancing by identifying people who fear they may lack the willpower to act and personalizing the opportunity to create some self-imposed constraints that make it costly or difficult to succumb to temptation.

A precommiting customer will, for example, buy unhealthy snacks in smaller sizes than they had previously. Companies can use AI to find these signals and then design a range of options to help customers act on their long-term preferences. Nudging can be used to help customers declare their long-term goals and aspirations. Research shows that when people reflect on their behavior they become more aligned with other, deep-seated preferences.

To be fully agency-preserving, personalization relies on careful design of user feedback and control. What does a user need to know and when do they need to know it to preserve their long term autonomy? AI can be beneficial when users can declare their metapreferences — desires about desires — and have an AI serve as a tool to clarify and simplify choices as well as allow users to choose something different.

Perhaps the biggest challenge for AI designers is when working with AI systems that use data that humans are not aware of such as mouse clicks and eye tracking. People can feel significantly more threatened when they feel they are being watched while they make up their minds as compared to when they have already decided before preferences are revealed to others. Designers should consider ways to explain to users how the AI is working behind the scenes as people make decisions.

Personalization will always be imperfect. Sometimes people will choose to interact only with a human because they do not believe that a machine will be able to account for their unique situation. Many people don’t want to feel as though their problems are generic and crave the personal attention of a human. Designers need to consider how a user will evaluate the roles of human versus machine when designing for personalization. Which users want a highly personalized experience, but from a human and not from a machine?

Personalization should be seen as a delicate balance. The promise of AI is to provide efficiency and a seamless experience but it can create a response in humans that machines can’t predict.

Human-centered AI design is agency-preserving.

Photo by Dan Meyers on Unsplash

Share on email
Share on facebook
Share on linkedin
Share on twitter