Food recognition and logging: New research focuses on local Singaporean food

By Stuart Pallister

SMU Office of Research – Two Singapore Management University researchers have embarked on a three-year project, funded by Singapore’s Ministry of Education, to ‘de-bias’ digital food recognition and develop a more robust machine learning system capable of correctly identifying Singapore’s multiracial food.

The two researchers from SMU’s School of Computing and Information Sciences, Professor Ngo Chong Wah and Associate Professor Rajesh Balan, already have extensive experience in food recognition and logging. For example, Professor Balan had previously developed a smart watch-based solution called Annapurna which was able to detect automatically when the user was eating, capture images of the food eaten, and then upload those to a cloud service where they were recognised.

“The whole idea of the current project is food recognition,” Professor Ngo told the Office of Research. Not only does the user take a picture of the food being eaten, but when the image is uploaded, the algorithm should be able to recognise the type of food consumed correctly, find the right recipe, ingredients, and cooking steps, plus highlight the nutritional value.   

“But that’s only part of the state of the art,” he said, adding that “with the technology available today, we can’t do this.”

“That’s the target we want to reach: the recipe, along with detailed information which would help dieticians calculate the nutritional value of the meal for whatever purpose.”

And that is at the core of the research. When you are given dietary advice to avoid, say, certain foods, you may be initially willing to keep track of what you are consuming, but in practice you may find it a chore to log everything every day. Then there is the issue of dishes being recognised incorrectly.

“The reason is simple,” Professor Ngo said. “If I take a picture when eating at home and it’s recognised wrongly, I may need to correct the system multiple times before it learns to recognise the same food correctly. Not every user is willing to make such corrections.”

One theme of the research project, he said, will be to ensure the artificial intelligence (AI) recognition system is robust and not biased towards a particular cuisine, to avoid having a system which is “sometimes correct and sometimes wrong. We want to build a model that’s more robust so that it really does learn over time and can do recognition better and better.”

On the one hand, the system should be able to recognise as many dishes as possible, and on the other there should be a good user experience, he said. The research project will involve some 40 people participating in food logging over four weeks.

“That means the AI system needs to be robust. If I’ve been eating Indian food all week and the system insists I’ve been eating Chinese food, then it would be very frustrating.”

Singaporeans are proud of their local cuisines and the SMU project will focus on the plethora of dishes available in multiracial Singapore, whether Chinese, Indian, Malay or Peranakan (a fusion of Chinese and Malay). “We have all kinds of food so Singapore’s the right place to do the research.”

Currently food recognition is biased towards dishes which are more popular on the internet, “but if you talk about localised food, for example, then it doesn’t do well. So, we’re trying to find an algorithm to ‘de-bias’ AI recognition.” In terms of Singapore’s local food, there is a system bias towards Chinese food which is predominant due to the large ethnic population.

SMU is already working with government agencies and external companies on food recognition services. “We can only recognise about 1,000 popular Singaporean dishes,” Professor Ngo said, “but beyond that we can’t do much.” So, the researchers are aiming to collect as many recipes as possible to add to the database, so that “whenever the user takes a picture of food, we can identify it from a large pool of recipes. So that’s actually the purpose of the research.”

Along with potential health benefits, the research may also have commercial applications, although at this stage Professor Ngo said that will depend on the progress of the project.

As AI systems develop, we may be embarking on a brave new world in which restaurant kitchens become increasingly automated to reduce labour costs. Although this is beyond the scope of the current project, as machine learning becomes more and more sophisticated, it does not seem far-fetched.

“In my opinion we want to enjoy food from real humans rather than robots,” Professor Ngo said. “People may want to see what the food will look like if I cook it in a certain way. Or if I use only a few ingredients, I could ask the AI system what the food would look like. Maybe there’s a possibility of this.”

The presentation of food is important but so too, if not more so, is the taste. “How it tastes, we don’t know,” Professor Ngo conceded. At this stage, it is all about developing a machine learning system that will, at least, recognise food images correctly.  

Back to Research@SMU August 2023 Issue