Artificially Intelligent Food Assistant for the Visually Impaired

Abstract

Vision disability is a prevalent condition that affects the lives of many adults and children. Previous research has established that it is harder for the visually impaired to evaluate the nutritional value of food. Therefore, by applying concepts used in agriculture and optical character recognition, we engineered a system that can make these perceptual evaluations on a variety of fresh and packaged foods and linguistically relay that info to a visually impaired user. We utilized some of the most memory-cheap and accurate object detection models to evaluate and then detect lesions in peaches, apples, tomatoes, and strawberries. Table 3 shows that our models performed with high accuracy as EffecientDet d0, SSD Lite Mobilenet, and Faster RCNN MobileNet all had higher than 40% mAP and 50% mAR. Figure 2 portrays how our interface was able to detect, determine, and relay surface spoilage percentage(s) to a user. Figure 3 shows that our OCR integration successfully was able to gather nutritional data on packaged goods and relay nutritional information to a user. Our system provides the brain for future applications that plan to deploy our code to devices like smart glasses or other hardware. We have made our source code available on GitHub through this link: https://github.com/SarthakJaingit/Artificially-Intelligent-Food-Assistant-for-the-Visually-Impaired. Our repository provides instructions about running our system through the command line and also a notebook demo that a less technical person can run to see how one of our models performs on a computer webcam with no video optimization.