The Q&A Professor

By Michelle Lee Twan Gee

SMU Office of Research & Tech Transfer – Professor Jiang Jing is driven by questions and answers.

Specifically, the Associate Professor of Information Systems at SMU and Deputy Director of the Living Analytics Research Centre (LARC) is on a quest to uncover a better way by which machines can answer questions.

Today machines are answering our queries in just about every sphere of our life. At work, at home (Alexa and SIRI) and in public spaces, machines are giving us the weather report, telling us how to place online orders, turning our air-conditioner on and instructing us on how to navigate the way to our destinations.

All this is enabled by the question answering (QA) technology. QA is a task in the fields of information retrieval and natural language processing (NLP) that is concerned with building systems that automatically answer questions posed by humans in a language understood by the average man on the street (that is, through words or signs).

Question answering differs from search engines in that it is about information extraction rather than document retrieval: it seeks to provide exact answers to questions rather than merely present a bunch of web pages. In short, question answering systems deliver answers rather than merely suggest documents where one can find possible answers.

Typically in question answering, a computer programme finds answers by searching either within a structured database of knowledge or information (a knowledge base) or by pulling answers from an unstructured collection of natural language documents. Such unstructured collections could include a collection of reference texts, an organisation’s set of documents and web pages, compiled news reports, Wikipedia pages or web pages.

Question answering systems may be asked to handle closed-domain questions or open-domain questions.

Closed-domain questions are questions that fall under a distinctive domain such as sports or medicine. They can also be questions that have been narrowly restricted, such as questions asking for descriptive rather than procedural information.

Open-domain questions are questions about anything under the sun. And these are precisely the questions that are befuddling question answering systems right now.  

The era of instant answers 

Question answering systems are central to our lives today in this era of instant answers. Today the explosion of information available online and our increasingly short attention span and patience makes us demand information at the snap of our fingers. At multiple points during the course of our daily lives, we seek answers to diverse questions and we want the answers NOW. 

In response to this urgent demand, the last four decades has seen the creation of hundreds of question answering systems. Despite this, the current capability of QA systems remains unsatisfactory.

“People are aware of the limitations of neural networks and are really trying to look into new ways to do things. For instance, we are a still long way off from neural networks being able to answer ‘why’ and ‘how’ questions,” commented Professor Jiang. “Neural networks are good at recognising patterns but not good with answering questions with many possible answers. Computer scientists are making some progress but current technologies remain very preliminary.”

Professor Jiang dreams of a day when machines can work things out by themselves. “The end goal is to have AI networks that are able to reason. Ideally, when asked a “why” question, instead of having AI systems find the answer from some documents, we want AI engines to be able to infer from information that they have, and explain that because of A, B and C, therefore the answer is D.”

On the hunt for smarter QA systems

In her paper ‘Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering,’ Professor Jiang, together with fellow SMU researchers and researchers from IBM Research AI, proposed using evidence aggregation to get answers.

A popular approach to answering open-domain questions is to first search for question-related passages and then apply reading comprehension models to extract answers. Existing methods usually extract answers from single passages independently. But some questions require a combination of evidence from across different sources to answer correctly.

Professor Jiang and her colleagues suggest improving open-domain QA by explicitly aggregating evidence from across multiple passages.

Explained Professor Jiang: “Open-domain question answering answers questions from a broad range of domains by marshalling evidence from large open-domain knowledge sources. Studies have made progress in bringing together evidence from large data sources, but they predict an answer to the question with only a single retrieved passage at a time.

“However, answer accuracy can often be improved by using multiple passages. In some cases, the answer can only be determined by combining multiple passages.”

The paper sets out two models that make use of multiple passages to generate their answers. Both use an answer re-ranking approach, where possible answers generated by an existing state-of-the-art QA model are reordered. The two models – strength-based re-ranking and coverage-based re-ranking – will then make use of the aggregated evidence from different passages to better determine the answer.

In strength-based ranking, possible answers are ranked according to how often their evidence occurs in different passages. The re-ranking is based on the idea that if a possible answer has multiple pieces of evidence, and each passage containing some evidence tends to predict the answer with a relatively high score, then this answer is more likely to be correct. The passage count of each possible answer, and the aggregated probabilities for the respective answers, reflect how strong its evidence is, and thus in turn suggest how likely an answer is the corrected answer.

In coverage-based re-ranking, a possible answer is ranked higher if the union of all its contexts in different passages could cover more aspects included in the question. To achieve this, all the passages that contain the possible answer are linked together. The result is a new context that aggregates all the evidence necessary to submit an answer as the right answer for the question.

When tested, both models delivered state-of-the-art results on three public open-domain QA datasets: Quasar-T, SearchQA and the open-domain version of TriviaQA. In fact, the models performed better on two datasets, showing an eight-percentage-point improvement over the best results achieved thus far with other models.

Another way forward proposed by Professor Jiang is using topic units.

In the paper ‘Knowledge Base Question Answering with Topic Units,’ a team of three SMU researchers including Professor Jiang postulates that only relying on entity linking to look for answer candidates may not be sufficient. Instead, the team suggests performing topic unit linking using a generation-and-scoring approach to gradually refine the set of topic units.

Testing of this method on three commonly used benchmark datasets has shown that this method is not only consistently accurate, it also outperforms the previous best performance achieved on two of the datasets.

Back to Research@SMU Feb 2020 Issue