Release 0.0.14

0.0.14

New features

• use_existing_tensors flag, for add_documents (#335). Use existing Marqo tensors to autofill unchanged tensor fields, for existing documents. This lets you quickly add new metadata while minimising inference operations. See usage here.
• image_download_headers parameter for search and add_documents (#336). Index and search non-publicly available images. Add image download auth information to add_documents and search requests. See usage here.

Optimisations

• The index cache is now updated on intervals of 2 seconds (#333), rather than on every search. This reduces the pressure on Marqo-OS, allowing for greater search and indexing throughput.

Bug fixes

• Helpful validation errors for invalid index settings (#330). Helpful error messages allow for a smoother getting-started experience.
• Automatic conversion to fp32 when using fp16 models on CPU (#331). This allows Marqo running on a CPU-only machine to index and search a fp16 CLIP index (although we still recommend interacting with a fp16 index with device=CUDA for the best results).
• Broadening of the types of image download errors gracefully handled. (#321)

Release images can be found on Docker hub

Release 0.0.13

0.0.13

New features

• Support for custom CLIP models using the OpenAI and OpenCLIP architectures (#286). Read about usage here.
• Concurrency throttling (#304). Configure the number of allowed concurrent indexing and search threads. Read about usage here.
• Configurable logging levels (#314). Adjust log output for your debugging/log storage needs. See how to configure log level here.
• New array datatype (#312). You can use these arrays as a collection of tags to filter on! See usage here.
• Boost tensor fields during search (#300). Weight fields as higher and lower relative to each other during search. Use this to get a mix of results that suits your use case. See usage here.
• Weighted multimodal queries (#307). You can now search with a dictionary of weighted queries. If searching an image index, these queries can be a weighted mix of image URLs and text. See usage here.
• New GPT-Marqo integration example and article. Turn your boring user manual into a question-answering bot, with an optional persona, with GPT + Marqo!
• Added new OpenCLIP models to Marqo (#299)

Optimisations

• Concurrent image downloads (#281, #311)
• Blazingly fast fp16 ViT CLIP models (#286). See usage here
• Reduction of data transfer between Marqo and Marqo-os (#300)
• We see a 3.0x indexing speedup, and a 1.7x search speedup, using the new fp16/ViT-L/14 CLIP model, compared to the previous release using ViT-L/14.

Bug fixes

• Fixed 500 error when creating an index while only specifying number_of_shards(#293)
• Fixed model cache management no parsing reranker model properties properly (#308)

Contributor shout-outs

• Thank you to our 2.3k stargazers
• Thank you to @codebrain and others for raising issues.

Release images can be found on Docker hub

Release 0.0.12

0.0.12

New features

• Multilingual CLIP (#267). Search images in the language you want! Marqo now incorporates open source multilingual CLIP models. A list of available multilingual CLIP models are available here.
• Exact text matching (#243, #288). Search for specific words and phrases using double quotes (" ") in lexical search. See usage here.

Optimisations

• Search speed-up (#278). Latency reduction from Marqo-os indexes reconfigurations.

Contributor shout-outs

Thank you to our 2.2k stargazers and 80+ forkers!

Release images can be found on Docker hub

Release 0.0.11

0.0.11

New features

• Pagination (#251). Navigate through pages of results. Provide an extensive end-user search experience without having to keep results in memory! See usage here
• The /models endpoint (#239). View what models are loaded, and on what device. This lets Marqo admins examine loaded models and prune unneeded ones. See usage here
• The /device endpoint (#239). See resource usage for the machine Marqo is running on. This helps Marqo admins manage resources on remote Marqo instances. See usage here
• The index settings endpoint (/indexes/{index_name}/settings)(#248). See the model and parameters used by each index. See usage here.
• Latency log outputs (#242). Marqo admins have better transparency about the latencies for each step of the Marqo indexing and search request pipeline
• ONNX CLIP models are now available (#245). Index and search images in Marqo with CLIP models in the faster, and open, ONNX format - created by Marqo's machine learning team. These ONNX CLIP models give Marqo up to a 35% speedup over standard CLIP models. These ONNX CLIP models are open sourced by Marqo. Read about usage here.
• New simple image search guide (#253, #263).

Contributor shout-outs

• ⭐️ We've just hit over 2.1k GitHub stars! ⭐️ So an extra special thanks to our stargazers and contributors who make Marqo possible.

Release images can be found on Docker hub

Release 0.0.10

0.0.10

New features

• Generic model support (#179). Create an index with your favourite SBERT-type models from HuggingFace! Read about usage here
• Visual search update 2. (#214). Search-time image reranking and open-vocabulary localization, based on users' queries, is now available with the Owl-ViT model. Locate the part of the image corresponding to your query! Read about usage here
• Visual search update 1. (#214). Better image patching. In addition to faster-rcnn, you can now use yolox or attention based (DINO) region proposal as a patching method at indexing time. This allows localization as the sub patches of the image can be searched. Read about usage here.

Check out this article about how this update makes image search awesome.

Bug fixes

• Fixed imports and outdated Python client usage in Wikipedia demo (#216)

Contributor shout-outs

• Thank you to @georgewritescode for debugging and updating the Wikipedia demo
• Thank you to our 1.8k stargazers and 60+ forkers!

Release images can be found on Docker hub

Release 0.0.9

0.0.9

Optimisations

• Set k to limit to for Marqo-os search queries (#219)
• Reduced the amount of metadata returned from Marqo-os, on searches (#218)

Non-breaking data model changes

• Set default kNN m value to 16 (#222)

Bug fixes

• Better error messages when downloading an image fails (#198)
• Bug where filtering wouldn't work on fields with spaces (#213), resolving #115

Release images can be found on Docker hub

Release 0.0.8

0.0.8

New features

• Get indexes endpoint: GET /indexes (#181). Use this endpoint to inspect existing Marqo indexes. Read about usage here.
• Non-tensor fields(#161). During the indexing phase, mark fields as non-tensor to prevent tensors being created for them. This helps speed up indexing and reduce storage for fields where keyword search is good enough. For example: email, name and categorical fields. These fields can still be used for filtering. Read about usage here.
• Configurable preloaded models(#155). Specify which machine learning models to load as Marqo starts. This prevents a delay during initial search and index commands after Marqo starts. Read about usage here.
• New example and article: use Marqo to provide context for up-to-date GPT3 news summary generation (#171, #174). Special thanks to @iain-mackie for this example.

Bug fixes and minor changes

• Updated developer guide (#164)
• Updated requirements which prevented Marqo being built as an arm64 image (#173)
• Backend updated to use marqo-os:0.0.3 (#183)
• Default request timeout has been increased from 2 to 75 seconds (#184)

Contributor shout-outs

• For work on the GPT3 news summary generation example: @iain-mackie
• For contributing the non-tensor fields feature: @jeadie
• Thank you to our users who raise issues and give us valuable feeback
• Thank you to our 1.4k+ star gazers and 50+ forkers!

Release images can be found on Docker hub

Release 0.0.7

0.0.7

Bug fixes and minor changes

• 429 (too many request errors) are propagated from Marqo-os to the user properly (#150)

0.0.6

Also included in this release

New features

• Health check endpoint: GET /health. An endpoint that can be used to inspect the status of Marqo and Marqo's backend (Marqo-os)
  (#128). Read about usage here.
• Marqo can be launched with environment variables that define limits around maximum number of fields per index, maximum document size and the maximum number of documents that can be retrieved
  (#135). Read about usage here.
• README translations:
  • Chinese 🇨🇳 (by @wanliAlex, #133)
  • Polish 🇵🇱 (by @MichalLuck, #136)
  • Ukrainian 🇺🇦 (by @dmyzlata, #138)
  • French 🇫🇷 (by @rym-oualha, #147)

Breaking API changes

• The home / json response has been updated. If you have logic that reads the endpoint root, please update it (#128).
• The Python client's add_documents() and update_documents() batch_size parameter has been replaced by server_batch_size and client_batch_size parameters
  (py-marqo#27), (py-marqo#28)

Non-breaking data model changes

• Each text field just creates a top level Marqo-os text field, without any keywords
  (#135)
• Very large fields get their tensor_facet keywords ignored, rather than Marqo-OS preventing the doc being indexed
  (#135)
• Tensor facets can no longer have _id as a filtering field
  (#135)

Bug fixes and minor changes

• FastAPI runs with better concurrency (#128)
• Get documents by IDs and lexical search and no longer returns vectors if expose_facets isn't specified
• Fixed batching bug in Python client
  (py-marqo#28)

Caveats

• If a large request to add_documents or update_documents results in a document adding fields such that the index field limit is exceeded, the entire operation will fail (without resilience). Mitigate this sending add_documents and update_documents requests with smaller batches of documents.
• For optimal indexing of large volumes of images, we recommend that the images are hosted on the same region and cloud provider as Marqo.

Contributor shout-outs

• For their translation work: @rym-oualha, @dmyzlata, @wanliAlex, @dmyzlata, @MichalLuck
• For raising issues and helping with READMEs: @kdewald, @llermaly, @namit343
• Thank you to our 900+ star gazers and 30+ forkers

The image can be found on Docker

Release 0.0.5

0.0.5

Added Open CLIP models and added features to the get document endpoint.

New features

• Added Open CLIP models (#116).
  Read about usage here
• Added the ability to get multiple documents by ID
  (#122).
  Read about usage here
• Added the ability to get document tensor facets and embeddings through the get document endpoint
  (#122).
  Read about usage here

0.0.4

Release 0.0.4

Adding the attributesToRetrieve to the search endpoint and added the update documents endpoints

Image can be found at Docker.