Designing for Intelligence: Embracing Inconsistency (Part 2 / 5ish)

This is part two of a five-ish part series. Here’s part one.

Consistency is one of the most misunderstood concepts in product design. As Jared Spool wrote back in 2005,

The problem with thinking in terms of consistency is that those thoughts focus purely on the design and the user can get lost. “Is what I’m designing consistent with other things we’ve designed (or others have designed)?” is the wrong question to ask. Instead, the right question is, “Will the user’s current knowledge help them understand how to use what I’m designing?”

Put another way: Software will feel consistent when it matches a user’s expectations in context.

What does it mean to match a user’s expectations in context? Context is itself constantly shifting, which means we’re trying to achieve a sense of consistency while responding to something inherently inconsistent.

One way to think about it is in terms of modes. In UX-speak, a mode is a state-change that causes your UI to behave in fundamentally different ways. Done wrong, modes can be jarring — and indeed some of UX’s greatest thinkers have suggested they be avoided entirely.

For example: Have you ever thought a button was broken—clicking on it repeatedly with no response—and eventually realized it’s blocked by a dialog box? That frustration is due to the modal nature of the dialog: the UI has switched modes, from one where you can click anywhere to one where you have to dismiss the dialog in order to proceed.

Modes are sometimes unavoidable, and aren’t universally bad. When I’m writing this post and I move the mouse, the cursor moves around the screen. When I’m playing Minecraft and do the same thing, my entire view shifts as I look around the virtual world. I’m in a “gaming mode,” and I adapt without thinking.

So the goal isn’t to avoid modes entirely, but to ensure we create modes that match those in the user’s brain. Let’s call this contextual consistency — again, matching the user’s expectations in context.

Part of the allure of intelligent apps is their ability to adapt — to adjust themselves based on what they know about us, our behavior, and our current situation. Those changes often introduce uncertainty — effectively, probabilistic modality — and the degree of uncertainty tends to increase with the sophistication of the algorithm.

Here’s a simple example. Amazon’s Kindle lets you press, hold, and drag to select text onscreen, revealing options like dictionary definitions, Wikipedia entries and highlighting. Until recently, it looked like this:

For definitions and contextual info, the Kindle provides a row of cards. To highlight the selection for later, hit the Highlight button.

A recent update changes this UI: Shorter selections work as they did before, but selecting more than four words produces this:

The informational cards aren’t useful for longer selections. So Amazon assumes, correctly, that I want to highlight the phrase and just does it. That’s smart, right? If I select a paragraph, it’s near-certain I want to highlight; if I only select a word, it’s probably a lookup. It’s great when one can build a “smart” feature with a simple heuristic.

The problem is mode mismatch. A significant minority of the time, I might want to highlight a short phrase. (“Measure twice, cut once,” is only four words long. So is, “Don’t shoot the messenger.”) When I’m selecting, I’m not counting words — and four-word phrases can take up more space onscreen than five-word ones. So the software mode (highlight vs. lookup) doesn’t reliably reflect my intent.

Errors make it even more problematic. Selecting on a Kindle is imprecise: it’s easy to miss, to select an extra line by accident—turning a two-word selection into twelve. Because such errors are common (and frustrating), it should be effortless to abort or correct. But Amazon’s new, adaptive UX means that the “abort” action is unpredictable: there are two variants (Highlight or simply click out), each producing the opposite result from the other.

This example is especially acute because for common actions, we have the opportunity to create expectations at a subconscious level. We call this muscle memory: When we perform actions (driving, showering, etc.) frequently enough that we don’t have to think about them. The process of developing muscle memory is habituation, and it’s a powerful tool for product designers. Products that make effective use of it feel fast and effortless. Products that don’t feel cumbersome and interrupt the user’s train of thought: conscious effort is diverted from the task to the tool.

In our Kindle example, Amazon has created two distinct modes:

1. Select <4 words: Show definitions and a Highlight button.
2. Select ≥4 words: Automatically highlight and show Undo.

But these two modes don’t match the user’s own mental mode, which could reasonably include:

1. Select <4 words: Started falling asleep and pressed the screen.
2. Select ≥4 words: Meant to get a word’s definition and missed.
3. Select <4 words: Wanted to highlight three words.

…and so on. Because of this mode mismatch, the UI that appears is inconsistent with expectations—that is, it’s surprising. The user is constantly asked to re-habituate, blocking the formation of muscle memory. A common action that could occur seamlessly and subconsciously remains a perpetual interruption.

Muscle memory and habituation matter in non-modal situations, too. Here’s a recent version of the Xbox One home screen. Its most prominent feature is a list of apps and games—in this case Netflix, Amazon Video, and Destiny:

Look what happens after I open Amazon Video:

The list is a smart feature, driven by recency. Every time I open an app the screen rearranges itself, with the most recent app taking a different position and appearance. Sure, my most common apps are front and center; but I have to hunt around for the one I want. It’s not really a modal interface—though I suppose one could argue it’s in a constantly-shifting mode—but it is a UX that prevents habituation.

Apple introduced an adaptive launcher of its own in iOS 9, with a better approach:

Apple’s smart launcher sits on the search screen but doesn’t interfere with the task of searching.

The primary launcher is still the traditional, deterministic home screen, which permits habituation since things don’t move around. The adaptive launcher appears on the search screen, which makes sense: Search is generally invoked for less commonly-used apps, and that infrequency means habituation won’t apply.

But search does have a muscle-memory component of its own: pull down, type, look for results. And Apple’s design supports that. With the input focused and the keyboard up, nothing gets in the way of your intended activity. If the desired app is in the smart launcher and you notice, it’ll save you a little effort; if not, no harm done.

I started by talking about consistency. If consistency is a goal, we must reject intelligence in software, since the very adaptiveness that makes it powerful also makes it unpredictable. Indeed, participants in early-stage user studies tend to reject smart, adaptive experiences because they’re unpredictable and feel out-of-control.

But consistency isn’t the goal; perceived consistency is. And perceived consistency is about matching expectations, synchronizing human and software modality. Marrying perceived consistency with adaptive algorithms is a little like doing a magic trick: Get it right, and your user is fooled into perceiving an inconsistent experience as even more predictable than a consistent one.

And like a magic trick, it stops working when you explain it. When we ask prospective users about adaptive functionality and they reject it, it’s like revealing how the trick is done before you perform it.

How, then, do we build great experiences that are also smart and adaptive? By recognizing that the algorithm and the UX go hand in hand, just as they always have; that the type of adaptiveness we build is as much a part of the experience as the rounded corners on the buttons.

To achieve this, we must learn to co-evolve our intelligent algorithms with our UX. To stop asking, “What’s the right UX for this intelligent system?” and to start asking, “What should we build to meet this user need? What type of intelligence will serve it well?”

That, in turn, requires new knowledge and new collaboration. PMs and designers need a high-level understanding of the algorithms that power smart software, and machine-learning engineers need a basic grasp of UX fundamentals. Teams need to change their composition and operating rhythms. User researchers need to find new ways to test that avoid new types of self-reporting bias.

I’ll dive into how that might work in my next post—but we’re still figuring this out as an industry and I’d welcome your thoughts. In the meantime, thanks for reading.