BOWL: A rewards app for healthy eating
The objective of BOWL was to create a food-tracking app that allows you take pictures of your meals and rewards you with badges & relevant restaurant deals for healthy eating.
Using computer vision to identify meals based on images and supervised and unsupervised models to predict relevant restaurant deals, BOWL was able to come to life.
Using CNN for Computer Vision
The first component of my app was using computer vision to automatically recognize meals so that users did not have to manually input what they ate everyday - which can become quite cumbersome and an easy task to give up on. Rather, I wanted users to be able to quickly snap a picture of their food and for the app to understand what it was. To do this, I used transfer learning with InceptionV3 as the base model.
I was able to get a train accuracy of 86% and a test accuracy of 75%. I used image augmentation as well to create new images from the 1000 images / dish that I had from the food-101 data set.
train_datagen = ImageDataGenerator(featurewise_center=False, samplewise_center=False, featurewise_std_normalization=False, samplewise_std_normalization=False, zca_whitening=False, rotation_range=10, width_shift_range=0.05, height_shift_range=0.05, shear_range=0.1, zoom_range=0.2, channel_shift_range=0., fill_mode='nearest', cval=0., horizontal_flip=True, vertical_flip=False, rescale=1/255)) test_datagen = ImageDataGenerator(rescale=1. / 255) train_generator = train_datagen.flow_from_directory( "food-101/train", target_size=(299, 299), batch_size=64, class_mode='categorical') test_generator = test_datagen.flow_from_directory( "food-101/test", target_size=(299, 299), batch_size=64, class_mode='categorical') inception = InceptionV3(weights='imagenet', include_top=False) x = inception.output x = GlobalAveragePooling2D()(x) x = Dense(202,activation='relu')(x) x = Dropout(0.2)(x) predictions = Dense(101, kernel_regularizer=regularizers.l2(0.005), activation='softmax')(x) model = Model(inputs=inception.input, outputs=predictions) model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) checkpointer = ModelCheckpoint(filepath='best_model_class.h5', verbose=1, save_best_only=True) model.summary() history = model.fit_generator(train_generator, steps_per_epoch = 75750/64, validation_data=test_generator, validation_steps=25250/64, epochs=40, verbose=1, callbacks=[checkpointer]) model.save_weights('model_weights.h5') model.save('model_keras.h5')
Rewarding users for healthy eating
The second component of BOWL is the idea of rewards. After each day, a user is allowed to mix everything in their food log together. What this essentially means is that the nutrition facts of all their meals are aggregated together.
A user then receives a badge based on his/her aggregate nutrition facts for the day. If the user eats a lot of potassium, he/she may receive a “good” badge as a reward for high potassium intake. However, if the user eats too many carbs, he/she may receive a “needs improvement” badge reminding them of their carb intake.
This is a fun way to remind users where they may be doing well and where they may need improvement in their diet.
*Nutrition data was scraped from Nutrionix.com
After a user receives a certain amount of good badges, BOWL will reward the user with an additional restaurant deal.
In order to do this, I used the raw ingredients of the meals in a user’s food log and tried to predict a cuisine that encompassed similar ingredients.
For a user who was more “habitual” in eating habits, I used a logistic regression with count vectorizer to predict a cuisine (82% R2 score), and I used an unsupervised k-means clustering model with tf-idf weights (predicted the cluster based on ingredients, and randomly selected a nearby ingredient-cuisine in the same cluster) for a more “exploratory” eater who was down to try new things.
The predicted cuisine, along with the user’s location, was then put into the yelp api to return a restaurant that a user may like along with a restaurant deal from that restaurant.
*Cuisine data can be found here: Kaggle Recipe Ingredient Dataset
