Release 1.1.0

1.1.0

New features

• New field numberOfVectors in the get_stats response object (#553).
  This field counts all vectors from all documents in a given index. See here for detailed usage.

• New per-index health check endpoint GET /indexes/{index-name}/health (#552).
  This replaces the cluster-level health check endpoint, GET /health,
  which is deprecated and will be removed in Marqo 2.0.0. See here for detailed usage.

Bug fixes and minor changes

• Improved image download validation and resource management (#551). Image downloading in Marqo is more stable and resource-efficient now.

• Adding documents now returns an error when tensorFields is not specified explicitly (#554). This prevents users accidentally creating unwanted tensor fields.

Contributor shout-outs

• Thank you for the vibrant discussion in our forum. We love hearing your questions and about your use cases.

Release 1.0.0

1.0.0

Breaking Changes

• New parameter tensor_fields will replace non_tensor_fields in the add_documents endpoint (#538). Only fields in tensor_fields will have embeddings generated, offering more granular control over which fields are vectorised. See here for the full list of add_documents parameters and their usage. The non_tensor_fields parameter is deprecated and will be removed in a future release. Calls to add_documents with neither of these parameters specified will now fail.

• Multiple tensor field optimisation (#530). This optimisation results in faster and more stable searches across multiple tensor fields. Please note that indexed documents will now have a different internal document structure, so documents indexed with previous Marqo versions cannot be searched with this version, and vice versa.

• The add_documents endpoint's request body is now an object, with the list of documents under the documents key (#535). The query parameters use_existing_tensors, image_download_headers, model_auth, and mappings have been moved to the body as optional keys, and support for these parameters in the query string is deprecated. This change results in shorter URLs and better readability, as values for these parameters no longer need to be URL-encoded. See here for the new add_documents API usage. Backwards compatibility is supported at the moment but will be removed in a future release.

• Better validation for index creation with custom models (#530). When creating an index with a model not in the registry, Marqo will check if model_properties is specified with a proper dimension, and raise an error if not. See here for a guide on using custom models. This validation is now done at index creation time, rather than at add documents or search time.

• Stricter filter_string syntax for search (#530). The filter_string parameter must have special Lucene characters escaped with a backslash (\) to filter as expected. This will affect filtering on field names or content that contains special characters. See here for more information on special characters and see here for a guide on using Marqo filter strings.

• Removed server-side batching (batch_size parameter) for the add_documents endpoint (#527). Instead, client-side batching is encouraged (use client_batch_size instead of server_batch_size in the python client).

New Features

• Multi-field pagination (#530). The offset parameter in search can now be used to paginate through results spanning multiple searchable_attributes. This works for both TENSOR and LEXICAL search. See here for a guide on pagination.
• Optimised default index configuration (#540).

Bug Fixes & Minor Changes

• Removed or updated all references to outdated features in the examples and the README (#529).
• Enhanced bulk search test stability (#544).

Contributor shout-outs

• Thank you to our 3.2k stargazers!
• We've finally come to our first major release, Marqo 1.0.0! Thanks to all our users and contributors, new and old, for your feedback and support to help us reach this huge milestone. We're excited to continue building Marqo with you. Happy searching!

Release 0.1.0

0.1.0

New features

• Telemetry. Marqo now includes various timing metrics for the search, bulk_search and add_documents endpoints when the query parameter telemetry=True is specified (#506). The metrics will be returned in the response body and provide a breakdown of latencies for various stages of the API call.
• Consolidate default device to CUDA when available (#508). By default, Marqo now uses CUDA devices for search and indexing if available. See here for more information. This helps ensure you get the best indexing and search experience without having to explicitly add the device parameter to search and add_documents calls.
• Model download integrity verification (#502). Model files are validated and removed if corrupted during download. This helps ensure that models are not loaded if they are corrupted.

Breaking changes

• Remove deprecated add_or_update_documents endpoint (#517).
• Disable automatic index creation. Marqo will no longer automatically create an index if it does not exist (#516). Attempting to add documents to a non-existent index will now result in an error. This helps provide more certainty about the properties of the index you are adding documents to, and also helps prevent accidental indexing to the wrong index.
• Remove parallel indexing (#523). Marqo no longer supports server-side parallel indexing. This helps deliver a more stable and efficient indexing experience. Parallelisation can still be implemented by the user.

Bug fixes and minor changes

• Improve error messages (#494, #499).
• Improve API request validation (#495).
• Add new multimodal search example (#503).
• Remove autocast for CPU to speed up vectorisation on ARM64 machines (#491).
• Enhance test stability (#514).
• Ignore .kibana index (#512).
• Improve handling of whitespace when indexing documents (#521).
• Update CUDA version to 11.4.3 (#525).

Contributor shout-outs

• Thank you to our 3.1k stargazers!

Release 0.0.21

0.0.21

New features

• Load custom SBERT models from cloud storage with authentication (#474). Marqo now supports fetching your fine-tuned public and private SBERT models from Hugging Face and AWS s3. Learn more about using your own SBERT model here. For instructions on loading a private model using authentication, check [model auth during search(https://docs.marqo.ai/0.0.19/API-Reference/search/#model-auth) and model auth during add_documents.

• Bulk search score modifier and context vector support (#469). Support has been added for score modifiers and context vectors to our bulk search API. This can help enhance throughput and performance for certain workloads. Please see documentation for usage.

Bug fixes and minor changes

• README enhancements (#482, #481).

Contributor shout-outs

• A special thank you to our 3.0k stargazers!

Release 0.0.20

0.0.20

New features

• Custom model pre-loading (#475). Public CLIP and OpenCLIP models specified by URL can now be loaded on Marqo startup via the MARQO_MODELS_TO_PRELOAD environment variable. These must be formatted as JSON objects with model and model_properties.
  See here (configuring pre-loaded models) for usage.

Bug fixes and minor changes

• Fixed arm64 build issue caused by package version conflicts (#478)

Release 0.0.19

0.0.19

New features

• Model authorisation(#460). Non-public OpenCLIP and CLIP models can now be loaded from Hugging Face and AWS s3 via the model_location settings object and model_auth. See here (model auth during search) and here (model auth during add_documents) for usage.
• Max replicas configuration (#465). Marqo admins now have more control over the max number of replicas that can be set for indexes on the Marqo instance. See here for how to configure this.

Breaking changes

• Marqo now allows for a maximum of 1 replica per index by default (#465).

Bug fixes and minor changes

• README improvements (#468)
• OpenCLIP version bumped (#461)
• Added extra tests (#464)
• Unneeded files are now excluded in Docker builds (#448, #426)

Contributor shout-outs

• Thank you to our 2.9k stargazers!
• Thank you to community members for the increasingly exciting discussions on our Slack channel. Feedback, questions and hearing about use cases helps us build a great open source product.
• Thank you to @jalajk24 for the PR to exclude unneeded files from Docker builds!

Release images can be found on Docker hub

Release 0.0.18

0.0.18

New features

• New E5 model type is available (#419). E5 models are state of the art general-purpose text embedding models that obtained the best results on the MTEB benchmark when released in Dec 2022. Read more about these models here.
• Automatic model ejection (#372). Automatic model ejection helps prevent out-of-memory (OOM) errors on machines with a larger amount of CPU memory (16GB+) by ejecting the least recently used model.
• Speech processing article and example (#431). @OwenPendrighElliott demonstrates how you can build and query a Marqo index from audio clips.

Optimisations

• Delete optimisation (#436). The /delete endpoint can now handle a higher volume of requests.
• Inference calls can now execute in batches, with batch size configurable by an environment variable (#376).

Bug fixes and minor changes

• Configurable max value validation for HNSW graph parameters (#424). See here for how to configure.
• Configurable maximum number of tensor search attributes (#430). See here for how to configure.
• Unification of vectorise output type (#432)
• Improved test pipeline reliability (#438, #439)
• Additional image download tests (#402, #442)
• Minor fix in the Iron Manual example (#440)
• Refactored HTTP requests wrapper (#367)

Contributor shout-outs

• Thank you to our 2.8k stargazers!
• Thank you community members raising issues and discussions in our Slack channel.
• Thank you @jess-lord and others for raising issues

Release images can be found on Docker hub

Release 0.0.17

0.0.17

New features

• New parameters that allow tweaking of Marqo indexes' underlying HNSW graphs. ef_construction and m can be defined at index time (#386, #420, #421), giving you more control over the relevancy/speed tradeoff. See usage and more details here.
• Score modification fields (#414). Rank documents using knn similarity in addition to document metadata (#414). This allows integer or float fields from a document to bias a document's score during the knn search and allows additional ranking signals to be used. Use cases include giving more reputable documents higher weighting and de-duplicating search results. See usage here.

Bug fixes and minor changes

• Added validation for unknown parameters during bulk search (#413).
• Improved concurrency handling when adding documents to an index as it's being deleted (#407).
• Better error messages for multimodal combination fields (#395).
• Examples of recently added features added to README (#403).

Contributor shout-outs

• Thank you to our 2.6k stargazers.
• Thank you to @anlrde, @strich, @feature-hope, @bazuker for raising issues!

Release images can be found on Docker hub

Release 0.0.16

0.0.16

New features

• Bulk search (#363, #373).
  Conduct multiple searches with just one request. This improves search throughput in Marqo by parallelising multiple search queries in a single API call.
  The average search time can be decreased up to 30%, depending on your devices and models.
  Check out the usage guide here
• Configurable number of index replicas (#391).
  You can now configure how many replicas to make for an index in Marqo using the number_of_replicas parameter. Marqo makes 1 replica by default.
  We recommend having at least one replica to prevent data loss.
  See the usage guide here
• Use your own vectors during searches (#381). Use your own vectors as context for your queries.
  Your vectors will be incorporated into the query using a weighted sum approach,
  allowing you to reduce the number of inference requests for duplicated content.
  Check out the usage guide here

Bug fixes and minor changes

• Fixed a bug where some Open CLIP models were unable to load checkpoints from the cache (#387).
• Fixed a bug where multimodal search vectors are not combined based on expected weights (#383).
• Fixed a bug where multimodal document vectors are not combined in an expected way. numpy.sum was used rather than numpy.mean. (#384).
• Fixed a bug where an unexpected error is thrown when using_existing_tensor = True and documents are added with duplicate IDs (#390).
• Fixed a bug where the index settings validation did not catch the model field if it is in the incorrect part of the settings json (#365).
• Added missing descriptions and requirement files on our GPT-examples (#349).
• Updated the instructions to start Marqo-os (#371).
• Improved the Marqo start-up time by incorporating the downloading of the punkt tokenizer into the dockerfile (#346).

Contributor shout-outs

• Thank you to our 2.5k stargazers.
• Thank you to @ed-muthiah for submitting a PR (#349)
  that added missing descriptions and requirement files on our GPT-examples.

Release images can be found on Docker hub

Release 0.0.15

0.0.15

New features

• Multimodal tensor combination (#332, #355). Combine image and text data into a single vector! Multimodal combination objects can be added as Marqo document fields. This can be used to encode text metadata into image vectors. See usage here.

Bug fixes

• Fixed a bug that prevented CLIP's device check from behaving as expected (#337)
• CLIP utils is set to use the OpenCLIP default tokenizer so that long text inputs are truncated correctly (#351).

Contributor shout-outs:

• Thank you to our 2.4k stargazers
• Thank you to @ed-muthiah, @codebrain and others for raising issues.

Release images can be found on Docker hub