New Image-Based Food Scanner Could Simplify Tracking Calories

Whether it's step counts, daily workouts, or food intake, what gets measured gets monitored. For example, 2023 research suggests that wearing a fitness tracker that logs physical activity and provides feedback may help motivate some people to achieve their fitness goals.

For people who want to lose weight or have a health condition like type 2 diabetes, tracking the nutritional profile of each meal is critical. However, counting calories and determining the amount of fat, protein, and other nutrients can be daunting.

Food calculator apps can help, but they still require searching for food items and adding them to a daily log. Some apps scan food product bar codes and automatically add them to a list, but this could prove challenging with homemade meals or restaurant dishes.

The entire food tracking process involves time and effort, and calorie, fat, and protein counts might be inaccurate due to variances in serving sizes and ingredients.

Since it can be such a hassle, some people might give up tracking their food intake, which could pose risks, especially for people with health conditions like type 2 diabetes.

Now, a new technology has emerged that might make food tracking and calorie counting more convenient.

The new food tracking system, developed by NYU Tandon School of Engineering researchers, uses advanced algorithms to recognize food items in images and calculate their nutritional content, including calories, protein, carbohydrates, and fat.

To track food intake, a person simply snaps a picture of their meal, and the new system handles the rest.

Using smartphone images to count calories

NYU Tandon School of Engineering researchers were inspired to create new, more accurate food image scanning technology after NYU's decade-long Fire Research Group project revealed that a significant number of career and volunteer firefighters had a high body mass index (BMI) or obesity.

Typically, when researchers conduct studies requiring information on the participants' daily food intake, the data is self-reported. This means that each participant must track and make note of the type and amounts of food they eat each day — a notoriously unreliable method.

The researchers aimed to improve the reliability of food tracking data and remove human error by utilizing images of foods to log nutritional intakes.

While the idea seems simple, the research team soon discovered several challenges with developing an AI-based food recognition system.

"The sheer visual diversity of food is staggering. Unlike manufactured objects with standardized appearances, the same dish can look dramatically different based on who prepared it," said Sunil Kumar, a professor of mechanical engineering at NYU Abu Dhabi and NYU Tandon, in a news release. "A burger from one restaurant bears little resemblance to one from another place, and homemade versions add another layer of complexity."

AI's ability to estimate serving sizes and issues with processing power were additional challenges the team had to overcome.

To solve these issues, the team used YOLOv8, an image-recognition technology, and ONNX Runtime, an AI engine that allows AI programs to run more efficiently to create the new image-based system.

Testing revealed that the system correctly identified the nutritional values of simple food like pizza to more complex bakery items like baklava. The NYU team expects to refine its "proof-of-concept" technology for use in a wide range of healthcare applications.

Still, this technology isn't new, as other companies have developed similar systems and apps. For instance, apps like Calorie Mama Food AI and SnapCalorie use AI to determine calories and nutritional values from images of food items. Cal AI is another image-based calorie-counting app.

Moreover, MyNetDiary recently launched its version of an image-based food scanner called AI Meal Scan.

Pros of photo-based food tracking

Julie Stefanski, MEd, RDN, a spokesperson for the Academy of Nutrition and Dietetics, tells WellnessPulse that technological advancements like food scanning apps or programs that help people live healthier lives are a positive development.

"One of the most common uses of food tracking is for individuals with diabetes," Stefanski explains. "Especially for a person who must dose their insulin based on their dietary intake, accurate food-scanning and tracking can help improve blood sugar levels and precision in dosing."

Moreover, an image-based food tracker could be particularly beneficial for identifying the nutrient content of foods when traveling or eating out, since it's challenging to determine ingredients in food not prepared at home.

Other potential benefits of using a phone camera to capture and log food images may include:

• Convenience, as it may be easier for some to take a quick photo vs. scanning bar codes like other apps require, or jotting down individual food items in a notebook.
• Image-based AI food trackers may be more accurate than manually calculating nutrition information.
• It could help people pay more attention to what they eat and be more mindful of their eating habits.
• The ease, accuracy, and feedback might motivate people to achieve weight loss or blood sugar management goals.

Potential cons of tracking food intake

Stefanski says that while there is an abundance of tools available to track nutritional intake, it's not foolproof, as people can misinterpret the data or make choices that don't support their personal health goals.

For example, a person could misjudge serving sizes or forget to log small quantities of high-calorie condiments like mayo or dip.

Misuse of food tracking methods, image-based or otherwise, is another potential issue.

As a registered dietitian nutritionist, I've observed that it's a fairly common occurrence for someone to set their calorie goals too low, track until they are out of calories, and then stop tracking for the rest of the day while still eating. The promise that the next day they will 'do better' keeps people in a vicious cycle of hoping for change without making any real progress.

Julie Stefanski, MEd, RDN

Other potential cons with traditional and photo food scanning technology may include:

• Some individuals might become hyper-focused on food and eating, which could potentially put them at risk for conditions like disordered eating.
• Apps and website-based trackers may be challenging for non-tech-savvy individuals to use.
• Food tracking alone might not provide enough support for people who want to make radical changes to their diet for weight loss or other reasons.

Still, image-based food tracking can detect subtle differences in foods, which could solve accuracy issues.

"There are a lot of variations that can't be seen with the naked eye that impact calorie, sugar, and fat content of a dish," Stefanski says. "If someone used additional sugar, oil, or condiments like mayo in a recipe, the average person can't always see those differences in the appearance of a food."

The bottom line

If you are trying to lose weight, image-based food tracking technology could be an easier and more accurate way to monitor calorie intake. However, Stefanski says there's more to weight loss than just counting calories.

"Unfortunately, having a healthy body weight is not just about calories in versus calories out," Stefanski explains. "There are less cut and dry influencers on weight such as hormones, access to healthy foods, taste preferences, and variations in daily physical activity that may impact outcomes beyond just calorie counting."

Moreover, having access to accurate nutritional information in food and beverages is only one piece of the weight loss puzzle.

Once you have nutrition facts data about your food, you still must apply it to food purchasing, cooking, and your own personal calorie and macronutrient needs. How someone chooses to deal with their hunger when their appetite exceeds their allotted calorie goal is a choice that goes beyond a tracking device.

Julie Stefanski, MEd, RDN

While using image-based or traditional food tracking apps or programs can help you be more mindful of how many calories you're consuming, using them with the help of a registered dietitian nutritionist (RDN) may improve the chances of success.

"An RDN can help tailor calorie and macronutrient targets on an ongoing basis based on outcomes," Stefanski says. "An RDN can also guide someone in making changes to their recipes, portions, or meal timing to improve outcomes using the tools that are available."