Artificial Intelligence, Information Overload, and the Library of the Future

Artificial Intelligence, Information Overload, and the Library of the Future

I was invited to give a keynote lecture at the XV International Conference on University Libraries in Mexico City last month. The conference debated the changes occurring or need to occur in university libraries towards the United Nations’ “Agenda 2030” horizon. My lecture, titled “Libraries and Knowledge in the Age of Information Overload”, took a close look at the impact of the rampant Info Overload that is affecting the entire knowledge landscape on the academic library, and vice versa.

The main thing is that where in centuries past knowledge – carried in books – was scarce and precious, today the internet has created a huge overload of information, dubious information and fake information; we are all struggling to survive this info tsunami. As a result, if the libraries of the past were like serene shrines where people would come to bask in the glow of accumulated knowledge, today libraries must serve as sanctuaries where users can get help to survive the aforementioned tsunami, and gain the skills and support to allow them to find insight yet discriminate against the fake and the useless.

But interestingly, the more I thought about the information overload problem during the conference, the more my thinking veered towards AI as part of the solution. Indeed, Artificial Intelligence seems to be the best way that computers can help us extricate ourselves from the hole they’d helped us dig for ourselves. At last, they’ve attained the power required to help us deal with the overload, in a variety of ways. Some examples:

Incoming mail management

Tools like Knowmail apply AI techniques to figure out for us what we should be doing at any given moment about our inbox, based on the context of our workday at that specific moment. No matter that you have 800 messages clamoring for your attention: Knowmail – not you – will hone in on the two that are critical for the meeting you’re walking to right now. Sounds like the future of email is already here.

  • Medical IO

    With millions of research papers out there, doctors have no way to remain updated. The result is chilling: a patient comes in, the doctor is unaware of a study made in a far country last month that suggests a diagnosis or a cure, the patient dies. But IBM is repurposing its flagship AI machine Watson (of Jeopardy fame) to solve this problem: the doctor submits the patient’s history and symptoms, Watson gives recommendations based on reading and understanding the entire body of published articles, the patient lives.

  • Scholarly research

    The flood of papers covers all areas of science, but Semantic Scholar, a new search tool, uses data mining, natural language processing and computer vision to extract valuable information from 40 million studies. Yes – it actually reads, compares, correlates and evaluates the papers to figure out what’s relevant. In fact, the plan is to evolve this tool until it can comb research papers and suggest future studies, expose missing links, and nudge researchers towards more productive avenues.

  • Intelligence analysis

    One reason U.S. intelligence agencies failed to connect the dots and detect the September 11 plot is that the clues were submerged under a flood of other data. Today big data techniques are able to extract valuable insights from the billions of data points in such a flood – the more data the better, in fact.

All of which also made me think specifically of how academic libraries can use AI to get better at what they will need to do, which is, help students and faculty home in on the knowledge they need. In addition to existing search tools, a university could actually apply AI and big data in new and useful ways.

Some ideas that occurred to me:

  • Use the data in the university’s computers to direct students towards improved study outcomes

    (Imagine a computer telling the student “I see you’re taking book A for your course in astrophysics; but you should know that in past years students in that course that took this book before reading book B didn’t do as well in their course exam, on average, than those who started with book B. Better rethink your steps”.)

  • Generate personalized book recommendations

    Instead of relying on the generic reading list from their professor, students could have an AI-capable computer give them recommendations based specifically on what they know and what they need, allowing them to get the best grades and education with the least time investment. (Here the computer might say “I see you’re checking out book C, but from what I’ve read in your submitted reports and exams I think it’s too advanced for you. I think book D would be more beneficial to start with for someone at your level”.)

  • Help identify reliable sources among all the fake news

    With tools like Semantic Scholar reading and understanding written content, why not use this capability to scour the internet, identify who cites whom, who plagiarized what, and which sources are the most reliable?

  • Help solve the Open Access problem

    Everyone wants more open access to research papers, but we still need respected (and costly) commercial journals to check reliability of papers and citations. I see no reason why a computer couldn’t do this for us, once it knows the entire body of publications in a scientific field.

  • Visualize knowledge space

    OK, I know, this borrows from William Gibson’s cyberspace… but a visualization and navigation tool that allows to navigate visually, perhaps even in Virtual Reality, through a multidimensional taxonomy of books and online resources would be very useful (not to say, pretty cool!).

These tools will need to be developed, but the basic technology and science already exist. There would be hurdles – privacy concerns, for one – but we should strive to overcome them. The sooner we start, the better our libraries will be, and the better job they will do of helping us get over information overload.


Nathan Zeldes
Nathan Zeldes

Nathan Zeldes is a globally recognized thought leader in the search for improved knowledge worker productivity. After a 26 year career as a manager and principal engineer at Intel Corporation, he now helps organizations solve core problems at the intersection of information technology and human behavior.