Merge main into documentation. #74
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Refactor `messageHistory` to be agnostic of the image encoding. * Add backend encoding of images to `openAIChat`, `azureChat`, and `ollamaChat`. * Add image test points to the test files. Open question: Can we reliably detect which Ollama models support vision?
Preinstall bakllava, to have a vision model
Co-authored-by: MiriamScharnke <[email protected]>
Co-authored-by: Christopher Creutzig <[email protected]>
Co-authored-by: Christopher Creutzig <[email protected]>
Generating Markdown in pre-commit hook. The hook needs to be activated in the local repository, by including it in `.git/hooks/pre-commit`.
Co-authored-by: MiriamScharnke <[email protected]>
Future versions will get that automatically, from commit 570aaa1.
…g/llms-with-matlab into generate-md-from-mlx
Ollama images
Moved mlx into sub-directory, added Markdown export
Co-authored-by: Deborah Ferreira <[email protected]>
Co-authored-by: Christopher Creutzig <[email protected]>
…nly-edits Text only edits to error message catalog (doc review)
This file was not meant for this change.
Fix openai trademark
…nces Allow (long) char vectors for StopSequences
moondream is a much smaller vision model and perfectly suitable for the tests we want to run.
Replace bakllava by moondream
Create CODEOWNERS
To decouple `texampleTests.m` from availability and speed of external servers, record calls to `llms.internal.sendRequest` (on dev machine) and replay (during most test runs, including CI). See tests/recording/README.md for instructions.
The streaming interface sometimes responds with something like `json.choices = {"index":0,"delta":{},"logprobs":[],"finish_reason":"length"}`. Stop blindly assuming `json.choices.delta.content` exists.
Avoid bogus access to `json.choices.delta.content`
Trace/replay `llms.internal.sendRequest`
The moondream model is unreliable in reporting peppers or even reporting anything at all, notice the empty responses in several of these calls: ``` >> disp(generate(chat,messages)) The image features a bunch of fruits and vegetables, including several bell peppers and chilies in various colors such as red, green, yellow, purple, and white. There is also an onion and some garlic on the table along with other food items that could be related to a meal or ingredient preparation. >> disp(generate(chat,messages)) The image displays a group of fresh vegetables on top of purple fabric. Among the vegetables are several peppers and garlic, some lying flat while others stand tall or in various positions. >> disp(generate(chat,messages)) urn of different colored vegetables such as peppers and onions are placed on a table. >> disp(generate(chat,messages)) The image shows an assortment of various colorful vegetables, primarily focused on the peppers. There is a wide range of vegetables displayed in different shapes and sizes, creating a visually appealing arrangement that highlights their unique characteristics. >> disp(generate(chat,messages)) >> disp(generate(chat,messages)) The image displays a table with an arrangement of colorful, whole vegetables such as peppers and onions. These ingredients are spread across the table, showcasing a variety of colors including reds, oranges, greens, yellows, and purples. There is also an onion among these vegetables, further adding to their vibrant appearance. Overall, the scene captures a visually appealing display of fresh produce. >> disp(generate(chat,messages)) >> disp(generate(chat,messages)) >> disp(generate(chat,messages)) The image shows a variety of brightly colored vegetables sitting on a table, including red bell pepper and green bell pepper. The vibrant hues and diverse shapes of the peppers create an eye-catching display against the purple background. >> disp(generate(chat,messages)) >> disp(generate(chat,messages)) xtracted image of many different kinds of vegetables including red bell peppers, onions and garlic, as well as carrots. >> disp(generate(chat,messages)) >> disp(generate(chat,messages)) The image features a variety of colorful vegetables, including bell peppers and garlic, all stacked up in an appealing manner. There are also several carrots mixed throughout the pile, contributing to the vibrant display. >> disp(generate(chat,messages)) Â The image features a pile of fresh vegetables, including numerous bell peppers in various shades. This vibrant collection includes both red and green varieties that create a visually appealing display on the table or surface they are placed on. >> disp(generate(chat,messages)) The image displays a variety of fresh vegetables, including bell peppers and carrots. ``` Since we are not interested in testing the model, but we do want to run an end-to-end test to ensure that we pass images in the data format required by Ollama, generate multiple responses and make sure that at least one of them mentions `"pepper"` or `"vegetable"`.
Allow "pepper" or "vegetable"
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.