Skip to content

Commit

Permalink
Allow "pepper" or "vegetable"
Browse files Browse the repository at this point in the history
The moondream model is unreliable in reporting peppers or even reporting anything at all, notice the empty responses in several of these calls:

```
>> disp(generate(chat,messages))

 The image features a bunch of fruits and vegetables, including several bell peppers and chilies in various colors such as red, green, yellow, purple, and white. There is also an onion and some garlic on the table along with other food items that could be related to a meal or ingredient preparation.
>> disp(generate(chat,messages))

 The image displays a group of fresh vegetables on top of purple fabric. Among the vegetables are several peppers and garlic, some lying flat while others stand tall or in various positions.
>> disp(generate(chat,messages))
urn of different colored vegetables such as peppers and onions are placed on a table.
>> disp(generate(chat,messages))

 The image shows an assortment of various colorful vegetables, primarily focused on the peppers. There is a wide range of vegetables displayed in different shapes and sizes, creating a visually appealing arrangement that highlights their unique characteristics.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))

 The image displays a table with an arrangement of colorful, whole vegetables such as peppers and onions. These ingredients are spread across the table, showcasing a variety of colors including reds, oranges, greens, yellows, and purples. There is also an onion among these vegetables, further adding to their vibrant appearance. Overall, the scene captures a visually appealing display of fresh produce.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))
 The image shows a variety of brightly colored vegetables sitting on a table, including red bell pepper and green bell pepper. The vibrant hues and diverse shapes of the peppers create an eye-catching display against the purple background.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))
xtracted image of many different kinds of vegetables including red bell peppers, onions and garlic, as well as carrots.
>> disp(generate(chat,messages))
>> disp(generate(chat,messages))

The image features a variety of colorful vegetables, including bell peppers and garlic, all stacked up in an appealing manner. There are also several carrots mixed throughout the pile, contributing to the vibrant display.
>> disp(generate(chat,messages))
 The image features a pile of fresh vegetables, including numerous bell peppers in various shades. This vibrant collection includes both red and green varieties that create a visually appealing display on the table or surface they are placed on.
>> disp(generate(chat,messages))

 The image displays a variety of fresh vegetables, including bell peppers and carrots.
```

Since we are not interested in testing the model, but we do want to run an end-to-end test to ensure that we pass images in the data format required by Ollama, generate multiple responses and make sure that at least one of them mentions `"pepper"` or `"vegetable"`.
  • Loading branch information
ccreutzi committed Aug 6, 2024
1 parent 2cbda4d commit b5ad5d5
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions tests/tollamaChat.m
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,20 @@ function seedFixesResult(testCase)
end

function generateWithImages(testCase)
import matlab.unittest.constraints.ContainsSubstring
chat = ollamaChat("moondream");
image_path = "peppers.png";
emptyMessages = messageHistory;
messages = addUserMessageWithImages(emptyMessages,"What is in the image?",image_path);

text = generate(chat,messages);
testCase.verifyThat(text,matlab.unittest.constraints.ContainsSubstring("pepper"));
% The moondream model is small and unreliable. We are not
% testing the model, we are testing that we send images to
% Ollama in the right way. So we just ask several times and
% are happy when one of the responses mentions "pepper" or
% "vegetable".
text = arrayfun(@(~) generate(chat,messages), 1:5, UniformOutput=false);
text = join([text{:}],newline+"-----"+newline);
testCase.verifyThat(text,ContainsSubstring("pepper") | ContainsSubstring("vegetable"));
end

function streamFunc(testCase)
Expand Down

0 comments on commit b5ad5d5

Please sign in to comment.