Under the Hood: Prompt tuning Shake to Summarize for recipes

Following the rollout of Firefox’s Shake to Summarize feature to Android devices this May, we wanted to take a closer look at the modeling work behind the feature. In a previous blog post, we discussed our model selection process. Now that Shake to Summarize has been available on both iOS and Android for a few months, we wanted to take it a step further and outline our approach to prompt development. 

This is the story of a Shake to Summarize use case that required a little extra prompt tuning: online recipes.

The first step in developing a useful prompt is clearly describing what you want the LLM to do. LLMs thrive on specificity, so the more sharply you can define your task, the better your results are likely to be. 

This is especially important when using smaller LLMs (like we are here), since these little models are not as good at reading between the lines and intuiting unstated intentions as their more powerful cousins are. 

For this application, we were looking to create summaries. On the face of it, this seems pretty straightforward. However, as we iterated on prompts, we quickly discovered that what constitutes a “good” summary depends largely on what one is summarizing. 

For example, a useful summary of a novel should provide us with a quick overview of the plot without getting into too many specific details; we wouldn’t expect the summary to contain anything from the text verbatim. 

In contrast, a summary of a recipe website should include the recipe essentially as written. If the recipe says “cook lovingly” we might be OK with shortening it to “cook,” but if it calls for 4 cups of vegetable broth, a Tbsp of oregano and a tsp of thyme, we want these details relayed to us exactly. Merely stating, “this recipe calls for some broth and some spices” would not be adequate.