From b5ad5d57836ff3c4a308ab4b93ad5b15891f57c8 Mon Sep 17 00:00:00 2001
From: Christopher Creutzig <89011131+ccreutzi@users.noreply.github.com>
Date: Tue, 6 Aug 2024 12:06:46 +0200
Subject: [PATCH] Allow "pepper" or "vegetable"
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The moondream model is unreliable in reporting peppers or even reporting anything at all, notice the empty responses in several of these calls:

```
>> disp(generate(chat,messages))

 The image features a bunch of fruits and vegetables, including several bell peppers and chilies in various colors such as red, green, yellow, purple, and white. There is also an onion and some garlic on the table along with other food items that could be related to a meal or ingredient preparation.
>> disp(generate(chat,messages))

 The image displays a group of fresh vegetables on top of purple fabric. Among the vegetables are several peppers and garlic, some lying flat while others stand tall or in various positions.
>> disp(generate(chat,messages))
urn of different colored vegetables such as peppers and onions are placed on a table.
>> disp(generate(chat,messages))

 The image shows an assortment of various colorful vegetables, primarily focused on the peppers. There is a wide range of vegetables displayed in different shapes and sizes, creating a visually appealing arrangement that highlights their unique characteristics.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))

 The image displays a table with an arrangement of colorful, whole vegetables such as peppers and onions. These ingredients are spread across the table, showcasing a variety of colors including reds, oranges, greens, yellows, and purples. There is also an onion among these vegetables, further adding to their vibrant appearance. Overall, the scene captures a visually appealing display of fresh produce.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))
 The image shows a variety of brightly colored vegetables sitting on a table, including red bell pepper and green bell pepper. The vibrant hues and diverse shapes of the peppers create an eye-catching display against the purple background.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))
xtracted image of many different kinds of vegetables including red bell peppers, onions and garlic, as well as carrots.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))

The image features a variety of colorful vegetables, including bell peppers and garlic, all stacked up in an appealing manner. There are also several carrots mixed throughout the pile, contributing to the vibrant display.
>> disp(generate(chat,messages))
 The image features a pile of fresh vegetables, including numerous bell peppers in various shades. This vibrant collection includes both red and green varieties that create a visually appealing display on the table or surface they are placed on.
>> disp(generate(chat,messages))

 The image displays a variety of fresh vegetables, including bell peppers and carrots.
```

Since we are not interested in testing the model, but we do want to run an end-to-end test to ensure that we pass images in the data format required by Ollama, generate multiple responses and make sure that at least one of them mentions `"pepper"` or `"vegetable"`.
---
 tests/tollamaChat.m | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/tests/tollamaChat.m b/tests/tollamaChat.m
index 52d1376..4320774 100644
--- a/tests/tollamaChat.m
+++ b/tests/tollamaChat.m
@@ -99,13 +99,20 @@ function seedFixesResult(testCase)
         end
 
         function generateWithImages(testCase)
+            import matlab.unittest.constraints.ContainsSubstring
             chat = ollamaChat("moondream");
             image_path = "peppers.png";
             emptyMessages = messageHistory;
             messages = addUserMessageWithImages(emptyMessages,"What is in the image?",image_path);
 
-            text = generate(chat,messages);
-            testCase.verifyThat(text,matlab.unittest.constraints.ContainsSubstring("pepper"));
+            % The moondream model is small and unreliable. We are not
+            % testing the model, we are testing that we send images to
+            % Ollama in the right way. So we just ask several times and
+            % are happy when  one of the responses mentions "pepper" or 
+            % "vegetable".
+            text = arrayfun(@(~) generate(chat,messages), 1:5, UniformOutput=false);
+            text = join([text{:}],newline+"-----"+newline);
+            testCase.verifyThat(text,ContainsSubstring("pepper") | ContainsSubstring("vegetable"));
         end
 
         function streamFunc(testCase)