Artificial Intelligence and the Future of Search

Artificial Intelligence and the Future of Search

One of the least appreciated cognitive skills we humans use is finding stuff. Unless you’re too old to remember where you left your keys, searching for information is seen as a minor ability, certainly less important than playing chess, or reading literary works, or inventing wonderful new products. And yet this skill is hugely important, because we live in an age of free access to infinite online information – and we can’t find what we need in all that wealth.

The solution to this challenge came from the same online world that made it necessary, and it is interesting to see how that solution has been evolving while the Internet exploded from a minor DARPA project in the late sixties to the global mega-structure it is today.

The first answer to the problem of finding stuff on the Internet was online directories, starting from text-based ones like Gopher (1991), which gave a hierarchical directory of online resources, and later Yahoo! (1994, in its original implementation) and DMOZ (1998), which did the same thing. These depended on human intelligence to curate the directory, and so were bound to run out of steam sooner or later.

Next came Archie (1990) and Veronica (1993), early text-based search engines for finding information on FTP and Gopher servers respectively. And in 1993 came JumpStation, the first search engine that used a software “robot” to crawl the web and automatically create an index that people could search. This was the idea behind all subsequent search engines, and it gave us a long list of increasingly powerful tools including WebCrawler, Mosaic, Netscape, AltaVista, and ultimately Google.

Google’s great advantage originated from the PageRank algorithm, which allowed it to break ahead of the competition by prioritizing search results according their expected quality. This algorithm has been endlessly refined over the years to keep Google at the leading edge in its domain, and it should come as no surprise that Artificial Intelligence has recently been added to the mix. Around 2015 Google disclosed its use of RankBrain, an AI learning algorithm that analyzes the search query and uses its artificial smarts to understand the intentions of the user even for never-before-seen phrases and keywords. Details are understandably kept under wraps, but there is no question that leading edge AI is allowing Google to significantly improve its search results.

So where is this leading? AI and machine learning algorithms are obviously here to stay, in Search as in a growing number of other fields. The fact that we don’t really know what these methods are doing – how they learn to achieve better results than human-defined rules – may be vaguely disturbing, but can’t offset the advantage of better search results, which benefit both the users and the search provider’s bottom line. One can only conjecture what will come next, but it is highly likely that future generations of these tools will merge an understanding of language, an understanding of culture, and a deep rapport, as it were, with the individual user’s preferences and personality to create better search experiences we can’t yet imagine. We already have Google responding to our queries with a gentle “Did you mean?” when we mistype our query, but on the whole, it still responds to the specific question we asked; the next step would be to reply to what it thinks we wanted to ask but didn’t. And the queries themselves will be much more natural – instead of trying to guess what phrase would appear in the site we need, we’ll just tell the computer – in spoken language, much of the time – what we’re trying to achieve, whether it be “find relevant sources for my class project on the meaning of Hamlet’s famous monologue” or “I feel like taking a vacation to some interesting destination that isn’t too boring… how about it?”.

And then – the search engine may react to what it thinks we ought to have asked to fill our need (as it perceives it) even before we realize what that is. And how, you ask, could it possibly know what we need when we ourselves don’t? Well – given how much data a company like Google collects from our behavior in various ways and given its knowledge of what a billion other users are doing around the planet and around the clock, it has all the big data it needs to train its AI programs until they figure it out with more certainly that we ourselves can have. We may think we’re intelligent, but our intelligence is fairly fixed, whereas that of the AI we’re creating is growing exponentially – is it any wonder it can outsmart us?…

Exciting? Scary? Depends on your point of view. Think about it… or just Google it!

Nathan Zeldes
Nathan Zeldes

Nathan Zeldes is a globally recognized thought leader in the search for improved knowledge worker productivity. After a 26 year career as a manager and principal engineer at Intel Corporation, he now helps organizations solve core problems at the intersection of information technology and human behavior.

Leave a Reply

Your Email address will not be published. Required fields are marked *