Authors:
Paper:
https://arxiv.org/abs/2408.10532
Introduction
In recent years, the integration of artificial intelligence (AI) into various aspects of daily life has revolutionized how we interact with technology. One significant area of impact is health and nutrition, where AI-powered applications are becoming increasingly popular. Despite the surge in diet and nutrition apps, a common drawback is the manual entry of food data, which is both time-consuming and tedious. This paper introduces NutrifyAI, a comprehensive system designed to address this issue by leveraging advanced computer vision techniques and nutritional analysis APIs to provide real-time food detection, nutritional analysis, and personalized meal recommendations.
NutrifyAI is built on three key components:
1. Food Detection: Utilizing the YOLOv8 model for accurate and rapid food item recognition.
2. Nutrient Analysis: Employing the Edamam Nutrition Analysis API to retrieve detailed nutritional information.
3. Personalized Meal Recommendations: Using the Edamam Meal Planning and Recipe Search APIs to offer tailored dietary suggestions.
The system is designed for both mobile and web platforms, ensuring fast processing times and an intuitive user interface. Preliminary results indicate the system’s effectiveness, making it a valuable tool for users to make informed dietary decisions.
Related Work
The development of NutrifyAI builds on a foundation of existing research in food detection, nutritional analysis, and AI-driven meal recommendations. Key studies in these areas include:
- YOLO Models in Food Detection: Studies have demonstrated the high accuracy and efficiency of YOLO models in real-time food detection applications. YOLOv8, in particular, offers optimized performance for rapid and precise object detection
. - Integration of Nutritional APIs: Research has highlighted the benefits of integrating nutritional APIs, such as Edamam, for real-time nutrient analysis, enhancing user experience by providing comprehensive nutrient breakdowns
. - AI-Driven Meal Recommendations: AI-powered systems for personalized meal recommendations have shown promise in supporting users’ dietary goals through high customization based on user data
.
Research Methodology
Overview
NutrifyAI aims to enhance daily food tracking through AI technologies by developing a web application that highlights its practicality and usability. The application allows users to capture an image of their meal, which is then processed to detect and identify the food items present. The system leverages the YOLOv8 model for food detection and the Edamam API for nutritional analysis, providing users with a comprehensive nutritional breakdown. Additionally, the app offers personalized meal recommendations based on the user’s nutritional goals.
Data Collection and Preprocessing
The training of the food detection model was based on multiple datasets aggregated and preprocessed from publicly accessible repositories. The datasets used include:
- Open Images V6-Food Dataset: Extracted relevant food-related images, focusing on 18 specific food labels, resulting in over 20,000 images with detailed annotations.
- School Lunch Dataset: Included 3,940 images of Japanese high school students’ lunches, categorized into 21 distinct classes.
- Vietnamese Food Dataset: Self-collected dataset containing images of 10 traditional Vietnamese dishes.
- MAFood-121 Dataset: Contains 21,175 images of 121 different dishes from the top 11 most popular cuisines worldwide.
- Food-101 Dataset: Includes 101,000 images of 101 different dish types, ensuring a wide variety of food items for training.
Model Architecture
The YOLOv8 model was trained using the aforementioned datasets, with data augmentation techniques applied to enhance generalization. The training process utilized Stochastic Gradient Descent with a momentum optimizer, starting with an initial learning rate of 0.01. The model was trained for 50 epochs, with early stopping to prevent overfitting. Additionally, EfficientNet-B4 was fine-tuned on the dataset for further classification.
Model Evaluation
The evaluation of the food detection model was conducted using the Food Recognition 2022 dataset, which includes 43,962 images with 95,009 labeled objects. The model’s performance was assessed using metrics such as precision, recall, F1 score, and overall accuracy. The highest mAP of 0.963 was achieved with YOLOv8s at 0.5 IoU 
.
Experimental Design
API Integration
NutrifyAI integrates several APIs to enhance its functionality:
- Edamam Nutrient Analysis API: Retrieves detailed nutritional information for detected food items, visualized using Chart.js.
- Edamam Recipe and Meal Planning API: Provides personalized meal recommendations based on the user’s nutritional goals.
- Google Sheets API: Stores and manages nutritional data and meal recommendations.
- Server-Side Implementation: Managed by a Flask server, handling communication between the client-side web application, the YOLOv8 model, and various APIs.
- Client-Server Implementation: The web client interacts with the server by sending images for food detection, with results displayed through a user-friendly interface
.
Results and Analysis
The evaluation of NutrifyAI’s food detection model revealed the following performance metrics:
- Accuracy: 75.4%
- Precision: 78.5%
- Recall: 72.8%
- F1 Score: 75.5%
The model performed exceptionally well on distinct-looking food items but struggled with visually similar items due to class imbalance in the training data. The detection confidence scores and detection speeds indicated that the model is efficient enough for real-time applications 
.
Overall Conclusion
NutrifyAI demonstrates the potential of integrating computer vision models with nutritional APIs to create an effective tool for food recognition and nutritional analysis. While the system achieved promising results, several areas remain for further exploration and enhancement. Future research should focus on expanding the diversity of training datasets, refining the model’s confidence calibration, and incorporating advanced evaluation metrics such as mAP. Developing mechanisms for user feedback will enable the system to adapt to real-world usage, offering more personalized and accurate meal recommendations.
As AI-powered nutrition technology continues to evolve, there is significant potential for its application in various domains, including healthcare and personalized diet planning, making the pursuit of these improvements highly worthwhile.
Acknowledgment
This project would not have been possible without the support and guidance of Junyao Chen and Professor Zhengyuan Zhou from the NYU Stern School of Business, Department of Technology, Operations, and Statistics. Their mentorship and insights were crucial in shaping the direction of this research. Furthermore, special thanks to the Winston Data Foundation for their financial support through their merit scholarship for the NYU GSTEM Program.