Android 6.0 up close: Google Now on Tap is almost amazing

Can you believe it? After months of waiting and anticipation, Google’s Android 6.0 Marshmallow release is finally on its way into the world.

I’ll have a detailed overview of what’s different with Marshmallow and why it all matters for regular users soon. (Update: See my in-depth Marshmallow FAQ for more.) First, I wanted to take an up-close look at one of Android 6.0’s most interesting features: Google Now on Tap. As I mused when Google gave us our first glimpse at Now on Tap this summer, this feature really seems like the future of Android — like something that has the potential to change the way we interact with our mobile devices.

In essence, Now on Tap aims to turn Google Now into Android’s connective tissue — tying everything on your device together with potent Google intelligence. You just tap and hold your device’s Home button from anywhere in the system, and Google Now pulls up a series of cards with contextual info about whatever’s on your screen.

So if you activate Now on Tap while looking at a text message that mentions a particular restaurant, the system might bring up a card showing reviews of the food along with links to jump to relevant apps like Maps or Yelp. If you activate it while viewing an email that mentions a movie, Now on Tap might give you cards with reviews of the film and an option to see the trailer or read more in IMDB.

Now on Tap also lets you use voice control to ask questions about what’s on your screen: All you do is say “Okay, Google” to get the system’s attention and then say whatever’s on your mind. You might ask “What album was this song on?” while watching a music video, for example, or “How far is this from me?” while looking at an email from a local business.

Sounds pretty intriguing, right? It is — but having spent some time with Now on Tap in the final Android 6.0 release, I’m not convinced it’s consistent enough to be truly transformative. Yet.

Getting to know Google Now on Tap

Let’s start with the good: When Now on Tap works, it feels — to borrow a term I’ve heard somewhere — magical. Like a true intelligent assistant that knows just what you want and delivers it on command.

I pressed and held my Home button while viewing a text message that mentioned a dinner at Olive Garden, for instance. Within a second or two, Now on Tap placed two cards atop my screen: one with a link to reviews about the restaurant along with icons to pull up directions and a general Internet search for the name, and the other with a quick-link to create a calendar event for the time mentioned in my message.

Another message came in that mentioned the movie The Martian. I activated Now on Tap again, and this time, it gave me a third card with reviews about the movie and a link to its trailer.

At this point, I started getting excited. This thing is reading my mind! So I decided to take things up a notch and try the voice-activated “Okay, Google” feature to ask for specific context. Still in that same message thread, I asked: “When’s that playing?” The system is supposed to be able to interpret those sorts of natural-language commands and figure out what you’re saying, so that oughta be easy…

Or so you’d think. Now on Tap took me to a generic Google search for the phrase “When’s that playing” — which didn’t do me much good.

I tried another voice-activated search while viewing an article I wrote about the second-gen LG Watch Urbane. My question: “How much does that cost?”

The first time, the system interpreted it as “How much does it cost?” and gave me a generic Google search for that term. I tried again, and it got my words right on the second go-round — but it failed to pick out the actual relevant term from the page.

Whoops. That’s not what I want.

The same sort of thing happened when I was viewing a news story about a solar panel company called SolarCity. I asked: “Where is that?” — and Now on Tap gave me a generic search for that exact phrase. Not too useful, either.

I had better luck when viewing an article I did on the new Moto 360. That time, I asked: “Where can I buy this?” — and Now on Tap took me to a search for “Where can I buy Moto 360.” I didn’t get any shopping-specific results, which would have been nice, but it was at least in the right ballpark. 

Tap, tap, tap: A little more magic

Things were starting to look up — but man, I wanted that magical feeling again. The feeling that this feature was really reading my mind and doing something special.

I found that lovin’ feeling while watching videos in YouTube — first, while viewing a music video by the band Audioslave. As the music played, I said: “Okay, Google, who’s the lead singer?” And bam:

Back in the video, I tried something else: “What album is this song on?” Ka-ching:

My favorite was when I pulled up a clip from the TV show Scrubs and asked: “Who’s that actor?” Google got it:

But I think I might have given the system too much credit there. The character’s name, “JD,” was in the video’s title — and that seems to be what Now on Tap actually picked up on. When I tried it again with a couple of other clips that didn’t have such obvious text-based clues, I didn’t have any luck.

Maybe I was getting overzealous, though. At Now on Tap’s launch, Google did say the system would focus on text at its launch — though the company hinted that it might eventually scan and recognize images as well. Maybe one day.

For now, text ends up being enough some of the time. I pulled up a video by the one and only MKBHD and tapped and held my tablet’s Home button — and right on cue, Now on Tap gave me context not only to the guy behind the video but also (sort of) to the product he was reviewing:

(As you can see there, too, the system knows what apps you have installed on your device and provides direct links within apps when relevant — like it did with Twitter in the example above.)

Now on Tap even gave me MKBHD’s age when I asked (“Okay, Google — how old is he?”). But remember: His name was on the screen as text. When I tried the same thing while watching a clip of Seinfeld on Letterman in 1988, Now on Tap gave me Jerry’s age even though Dave was being shown at the time (presumably because Dave’s name was first in the title, and Now on Tap had no way of knowing which person I meant).

Hello, Jerry. It’s far less magical when the trick doesn’t pan out.

Tapped out: Now on Tap on the home screen

One last factor worth mentioning is how Now on Tap works on the home screen. When I talked to Google about the feature back in June, I was told that the tentative plan was to have Now on Tap come up when you tap and hold the Home key from an app — but to have that same command take you to the full Google Now interface when performed from the home screen. In other words, tapping and holding the Home button from the home screen would work just like swiping up from the Home button did in the past.

The idea, I was told, was to make tapping and holding the Home key a universal gesture for Google Now. Where the gesture is performed would determine what exactly transpires: When you do it in an app, you might be craving context — but when you do it on the home screen, you probably just want to go to the full Google Now interface. So it makes sense to use different implementations for different scenarios.

Well, evidently, plans changed. In Android 6.0, if you tap and hold the Home key while on your home screen, Now on Tap activates. (Swiping up from the Home key no longer does anything.) Since there’s no real info to be digested there, though, the system doesn’t have anything to show you — and you just end up getting this slightly weird message:

In order to get to the full Google Now interface, you have to tap the card that says “Google Now” (which eventually gets shortened to just a multicolored “G” Google logo). Provided you’re using the Google Now Launcher, you can still swipe over to the left of your main home screen panel to get there as well — but if you’re using any other launcher, you’ll probably need to put a shortcut to the Google app on your home screen if you want quick and easy access.

(Unless you want to disable Now on Tap altogether, that is. If you do that, the tap-and-hold gesture will load the full Google Now interface like the swipe-up gesture does now — but Now on Tap obviously won’t be available.)

Tons of potential — not yet fully tapped

So what to make of this wild new feature? Based on the time I’ve spent with it so far, I’d say this: Now on Tap is delightfully impressive at times, but consistency is its core challenge. Plain and simple, it can be hard to know what you can and can’t ask the system — and when it will or won’t deliver what you want. (Google says it works best with “places, people, movies, and more,” but that leaves a lot of uncertain ground.) And even in instances where you’re using a known available command — or you’re tapping the Home button on what seems like a can’t-miss, obvious-place-for-context scenario — the system’s responses aren’t always predictable.

I learned that again when I tried asking “How far is that from me?” in the middle of a messaging thread where someone had mentioned a specific business. Now on Tap got my question right — but it picked the wrong business, focusing not on the business mentioned most recently but instead on a different company name mentioned earlier in the thread.

And that gets at the tricky nature of this kind of system: No matter how intelligent an algorithm is, it can’t actually read your mind — and that can be an issue in the real world, where your screen isn’t usually limited to a single sentence or keyword at a time.

Regardless of the reason, there are enough instances when Now on Tap struggles to deliver that I suspect it’ll end up being quicker and simpler for most people to rely on good old-fashioned searches in the long run. Once the novelty’s worn off, a feature like this has to nail it almost flawlessly to be worth using. If it fails to get you the info you need even a quarter of the time, it gets frustrating fast and starts to become more of a time-consumer than a time-saver.

But all things in perspective: This is version one of an extremely ambitious project, and it does a lot of things incredibly well. The potential here is tremendous. Now let’s see if Google can figure out how to tap into that potential and take Now on Tap from being a neat and sometimes useful trick to being a consistent, reliable, and indispensable part of the Android experience.