The Professor and her learning machines

By Michelle Lee Twan Gee

SMU Office of Research & Tech Transfer – Professor Sun Qianru is asking all sorts of intriguing “what if?” questions.

Questions such as: “What if you can train a machine to recognise an object in different settings after feeding it just one image of that object?” and “What if you can train a machine to predict the social relationship between the people in a photograph just based on that photograph alone?”

These intriguing questions posed by the Assistant Professor of Information Systems at SMU and the answers she gets from her research are making computer scientists excited.

One particular question she investigated together with a team of researchers yielded a paper that received the highest citation among related work.

The paper titled Pose Guided Person Image Generation asked the question: “Can a machine generate a convincing image of a person in any new pose after being fed an example image of this person?”

The answer is yes. The team developed a Pose Guided Person Generation Network that is able to generate images of a person in different poses after being fed one image. For example, when an image of a person sitting has been fed into the network, the network can generate very high-quality images of the same person dancing, walking or jumping.

There are two stages involved: pose integration and image refinement. In the first stage, the network generates a coarse image of the person in the target pose (for example dancing). The network then refines this initial blurry result by training a refining generator to create the final dancing image with convincing details.

The paper was accepted at top-tier conference of Machine Learning – the Proceedings of 31st Conference on Neural Information Processing Systems (NIPS 2017).

Professor Sun finds the idea of Pose Guided Person Image Generation fascinating. “It is quite amazing how the software can generate new images that are so convincing,” she remarks.  But the technology is also terrifying: it could possibly be used to incriminate a person or embroil a victim in a fake scandal. Realising the danger, Facebook AI recently published a dataset for inspiring related research on de-fake and dedicated more than $10 million to fund the effort for solving problem.

“In my opinion, technologies of generating fake images and de-fake are adversarial, and both progress by playing off each other,” she notes. “For example, more powerful generation models offer more efficient samples for training de-fake models. There is no conflict between faking and de-faking research.”

Research in progress

Professor Sun joined SMU in August 2019 and already she is knee-deep in projects.

One of the current projects that she is most excited about is Few Shot Learning problems. This is the science of teaching a machine to identify an object after it has been fed only a few images of that object. Recognition is difficult because the learning software doesn’t have enough images to get trained up to recognise the object.

“Solving Few Shot Learning problems is very challenging. It is also exciting because it is very useful with many possible applications,” describes Qianru.

Take the medical field. With a rare disease, there may only be a few images of that disease available. How then do you get your software to recognise that disease when it next encounters it?  

One solution offered by Professor Sun is Meta-Transfer Learning. This means training the software on another disease that is similar and that has a lot of existing images. After the software has been fed many images of this similar disease, it is then fed the few images of the target disease and fine-tuned to recognise this target disease.

Few Shot Learning would also be useful in surveillance. It would allow a monitoring system to become super-efficient at recognising faces. When a new faculty member joins a university for instance, the university needs to feed only one or two images of the new professor for the monitoring system to be able to recognise him, instead of having to feed the system dozens of his images. The extended research has just been awarded by Alibaba Group with the Alibaba Innovative Research (AIR) grant.

In another research paper titled A Domain Based Approach to Social Relation Recognition, a team led by Professor Sun showed that it is possible to train a computer to accurately predict the social relationship between individuals in a photograph by teaching the computer to recognise social cues such as activities and poses.

A Lise Meitner Award winner – an award given out by Max Planck Institute for Informatics to outstanding female computer scientists – Professor Sun obtained her PhD at Peking University only three years ago. Despite her youth, Professor Sun has been highly prolific in research. What drives her?

“I want to leave behind concrete, solid and impactful work that can help other researchers in their own research,” she explains. “My goal is to develop efficient algorithms for visual recognition and create an ideal system that can learn by itself.”

Why the focus on machine learning? “My belief is that the machine is going to be more powerful than the human, and I will be super excited to reveal and announce that powerfulness someday,” she declares.

Back to Research@SMU Nov 2019 Issue