feat: vector support

.github/workflows/test.yml

@@ -64,7 +64,7 @@ jobs:
       #     key: ${{ runner.os }}-python-${{ env.pythonLocation }}-${{ matrix.pydantic-version }}-${{ hashFiles('pyproject.toml') }}-test-v06
       - name: Install OS Dependencies
         # if: steps.cache.outputs.cache-hit != 'true'
-        run: sudo apt-get install -y libsndfile1-dev libportaudio2 ffmpeg
+        run: sudo apt-get update && sudo apt-get install -y libsndfile1-dev libportaudio2 ffmpeg
       - name: Install Dependencies
         # if: steps.cache.outputs.cache-hit != 'true'
         run: make setup-test

.gitignore

@@ -146,6 +146,7 @@ tests/tests/
 # docs
 htmlcov/
 plugindocs/
+platformdocs/
 
 # nendo library
 nendo_library/

CHANGELOG.md

@@ -5,6 +5,42 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 <!-- insertion marker -->
+## [0.2.0](https://github.com/okio-ai/nendo/releases/tag/0.2.0) - 2024-02-19
+
+- Added the first extension for the Nendo Library: The `NendoLibraryVectorExtension` is a mix-in class that can be used by implementations of the Nendo Library to add support for saving and retrieving embedding vectors.
+- Added a new class `NendoEmbedding` that represents an embedding of a `NendoTrack` and stores the vector together with the ID of the track from which it was computed, the text representation of the track that was used to compute the embedding, as well as the embedding plugin's name and version that were used to compute the embedding.
+- Added a new subclass of the `NendoPlugin` called `EmbeddingPlugin` that accepts either a `NendoTrack`, a `NendoCollection`, a `signal` and a `text`, or only a `text` and computes corresponding `NendoEmbedding`(s), depending on whether a single object or a `NendoCollection` was provided. It also determines whether the currently used `NendoLibrary` implementation uses the `NendoLibraryVectorExtension` and, if so, saves the computed embeddings directly to the library. 
+- Fixed a problem with the way dictionaries (like `NendoTrack.meta` and `NendoTrack.resource`) were stored in the Nendo Library which resulted in them being stored as JSON strings instead of JSON dictionaries. **This means that upgrading Nendo will break any apps that use libraries that were created with previous versions of nendo.** Please make sure you understand the implications and either flush your DB or convert it to match the new data model.
+- Added the global configuration variable `replace_plugin_data`, that specifies whether new plugin data will overwrite existing plugin data for the specific plugin name and version used to generate the data.
+- Reworked the way relationships between tracks are managed. Previously, Nendo would create bidirectional relationships, which caused problems with the retrieval of related tracks, especially when using paging (i.e. `offset` and `limit`). The new approach is to only store relationships _from_ a derivative track _to_ the original track from which it was derived. Accordingly, all functions for retrieving related tracks accept a new parameter, `direction: str`, which can assume either one of the values `"to"`, `"from"` and `"both"` and will change their retrieval behavior accordingly. Also, the method names for some methods were changed to reflect the new semantics more accurrately: `NendoTrack.has_relationship_to()` was replaced by `NendoTrack.has_related_track(direction=...)`. `collection.has_relationship_to()` was replaced by `collection.has_related_collection()`.
+- Added a new shortcut function `NendoTrack.relate_to_track()` that creates a relationship of the specified `relationship_type` and with the specified metadata from the given track _to_ another track.
+- Fixed a bug where the length of a `NendoTrack` would not be correctly determined if the track had a mono signal.
+- Two new shortcut functions `NendoTrack.refresh()` and `collection.refresh()` were added that retrieve the latest version of the given object as it exists in the Nendo Library, effectively allowing to quickly pull the latest changes to an object from the database.
+- The `NendoTrack.get_plugin_data()` methods signature was extended to allow for filtering a track's plugin data also by `plugin_version` and `user_id` in addition to the existing filters.
+- Adjusted the `add_track()` function:
+    - Stores all the ID3 metadata in `NendoTrack.meta` instead of `NendoTrack.resource.meta`.
+    - If no ID3 metadata was found `NendoTrack.meta.title` will be set to the original name of the imported file.
+- Added the `load_related_tracks: bool = False` flag to the `NendoLibrary.get_tracks()` function that allows to enable/disable the populating of the `NendoTrack.related_tracks` field upon retrieval of many tracks from the Nendo Library. This optimizes the performance of the call and allows to retrieve smaller objects that are more suitable for network transmission in web applications that use Nendo Core.
+-  Add support for `NendoLibrary.filter_tracks(collection_id="...", order_by="collection")` that allows to order of the retrieved tracks by their `relationship_position` in the referenced collection.
+- Added the `NendoLibrary.filter_related_tracks()` method to allow for filtering of related tracks queries.
+- Changed the `NendoLibrary.filter_tracks()` method to search over `NendoTrack.meta` in addition to `NendoTrack.resource`. Changed the name of the `resource_filters` argument to `search_meta` to reflect the new behavior.
+- Added the `NendoLibrary.add_tracks_to_collection()` function to allow for multiple tracks given as a list of IDs to be added to the specified collection.
+- Added the `order` parameter to the `NendoLibrary.get_collection_tracks()` function to allow for control of the ordering. Defaults to `"asc"`, which will order the collection's tracks in ascending order of their `relationship_position`.
+- Changed the signature and behavior of the `get_collection()` function:
+    - Renamed the `details` parameter to `get_related_tracks` to better reflect it's effect on the function. The parameter controls whether the `NendoCollection.related_tracks` field will be populated.
+    - The function now always returns a `NendoCollection` instead of returning a `NendoCollectionSlim` if `details=False` as is was previously.
+- Added the `user_id` parameter to the following functions:
+    - `NendoTrack.local()`
+    - `NendoTrack.get_plugin_value()`
+    - `NendoLibrary.get_track()`
+    - `NendoLibrary.remove_collection()`
+- Changed the signature of the `NendoLibrary.add_plugin_data()` function to make specifying of the `plugin_version` optional. If none is given, version will be inferred from the currently registered version of the plugin.
+- Extended the signature of the `NendoLibrary.filter_collections()` function by the `collection_types` parameter to also allow for filtering by a list of collection types.
+- Added the `NendoLibrary.library_size()` and `NendoLirary.collection_size()` functions to get the number of tracks in the library per user and the number of tracks in a collection, respectively.
+- A bunch of small bugfixes, too many to mention them all here.
+
+<small>[Compare with 0.1.3](https://github.com/okio-ai/nendo/compare/0.1.3...0.2.0)</small>
+
 ## [0.1.3](https://github.com/okio-ai/nendo/releases/tag/0.1.3) - 2023-12-08
 
 <small>[Compare with 0.1.2](https://github.com/okio-ai/nendo/compare/0.1.2...0.1.3)</small>

CONTRIBUTING.md

@@ -10,7 +10,7 @@ It's super easy to get started with the nendo development environment. Just make
 make setup
 ```
 
-> **Note**
+> **Note**:
 > Developing for and with Nendo Core requires careful dependency management. The use of a [python virtual environment](https://docs.python.org/3/library/venv.html) like [pyenv](https://github.com/pyenv/pyenv) or [poetry](https://python-poetry.org/) is highly recommended.
 
 ## Development process

Makefile

@@ -8,14 +8,15 @@ FORMAT_CMD = python3 -m ruff --fix-only src tests  ; black ./src
 CHECK_CMD = python3 -m ruff check src tests
 TEST_CMD = python -m coverage run -m pytest tests/*.py && python -m coverage report && python -m coverage html
 ORM_CMD = alembic revision --autogenerate -m "Autogenerated model update"
-DOCS_CMD = mkdir -p htmlcov/ && touch htmlcov/index.html && mkdir -p docs/.overrides && python3 ./scripts/gen_plugin_docs.py && mkdocs build -vs
-DOCS_SERVE_CMD = mkdir -p htmlcov/ && touch htmlcov/index.html && mkdir -p docs/.overrides && python3 ./scripts/gen_plugin_docs.py && mkdocs serve -s
+ORM_UPGRADE_CMD = alembic upgrade heads
+DOCS_CMD = mkdir -p htmlcov/ && touch htmlcov/index.html && mkdir -p docs/.overrides && python3 ./scripts/gen_plugin_docs.py && python3 ./scripts/gen_platform_docs.py && mkdocs build -vs
+DOCS_SERVE_CMD = mkdir -p htmlcov/ && touch htmlcov/index.html && mkdir -p docs/.overrides && python3 ./scripts/gen_plugin_docs.py && python3 ./scripts/gen_platform_docs.py && mkdocs serve -s
 POETRY_SETUP_CMD = poetry install --all-extras --sync
 POETRY_PUBLISH_CMD = poetry build && poetry publish
 CHANGELOG_CMD = git-changelog -c angular -t keepachangelog -s build,deps,feat,fix,refactor,docs -i -o CHANGELOG.md -T --bump=auto .
 RELEASE_CMD = git add pyproject.toml CHANGELOG.md && git commit -m "chore: prepare release $(version)" && git tag $(version) && git push && git push --tags
 
-.PHONY: setup setup-test setup-poetry clean format check test docs orm changelog publish release
+.PHONY: setup setup-test setup-poetry clean format check test docs orm orm-upgrade changelog publish release
 
 # Default target
 all: format check test orm build-docs
@@ -57,6 +58,9 @@ orm:
 		exit 0; \
 	fi
 
+orm-upgrade:
+	@$(ORM_UPGRADE_CMD)
+
 changelog:
 	@$(CHANGELOG_CMD)
 
@@ -68,4 +72,4 @@ release:
 	@$(RELEASE_CMD)
 
 publish:
-	@$(POETRY_PUBLISH_CMD)
+	@$(POETRY_PUBLISH_CMD)

README.md

@@ -8,7 +8,7 @@
 
 <p align="center">
 <a href="https://okio.ai" target="_blank">
-    <img src="https://img.shields.io/website/https/nendo.ai" alt="Documentation">
+    <img src="https://img.shields.io/website/https/okio.ai" alt="Website">
 </a>
 <a href="https://github.com/okio-ai/nendo/actions/workflows/test.yml" target="_blank">
     <img src="https://github.com/okio-ai/nendo/actions/workflows/test.yml/badge.svg" alt="Test">
@@ -35,11 +35,11 @@
 
 ---
 
-**Website**: <a href="https://okio.ai/" target="_blank">https://okio.ai/</a>
+**Website**: <a href="https://okio.ai/" target="_blank">okio.ai</a>
 
-**Documentation**: <a href="https://okio.ai/docs" target="_blank">https://okio.ai/docs/</a>
+**Documentation**: <a href="https://okio.ai/docs" target="_blank">okio.ai/docs</a>
 
-**Source Code**: <a href="https://github.com/okio-ai/nendo" target="_blank">https://github.com/okio-ai/nendo</a>
+**Nendo Platform**: <a href="https://github.com/okio-ai/Nendo-Platform" target="_blank">Repository</a>
 
 ---
 
@@ -53,6 +53,7 @@ Nendo core is the AI audio tool suite allowing you to effortlessly develop audio
 - Integrated essentials for audio processing and library management.
 - An extensible plugin architecture and growing ecosystem of AI Audio plugins.
 - Easily combinable tools that together address a wide range of use cases.
+- Support for storing, managing and retrieving embedding vectors.
 
 ## Requirements
 
@@ -84,7 +85,7 @@ from nendo import Nendo
 nendo = Nendo()
 ```
 
-... and just like that, you're ready to go!
+... and just like that, you're ready to go! Now, there are multiple ways to configure Nendo, refer to the [relevant documentation pages](https://okio.ai/docs/usage/config/) for more information.
 
 ## Usage
 

alembic.ini

@@ -49,7 +49,9 @@ prepend_sys_path = .
 # version_path_separator = :
 # version_path_separator = ;
 # version_path_separator = space
-version_path_separator = os  # Use os.pathsep. Default configuration used for new projects.
+version_path_separator = os
+
+version_locations = %(here)s/alembic/versions
 
 # set to 'true' to search source files recursively
 # in each "version_locations" directory

docs/development/analysisplugindev.md

@@ -47,22 +47,17 @@ class DestructionAnalyzer(NendoAnalysisPlugin):
         return {"destroyed": "yes"}
 
     @NendoAnalysisPlugin.run_track
-    def is_fucked_up(self, track: NendoTrack, use_full_analysis: bool = True):
+    def is_destroyed(self, track: NendoTrack, use_full_analysis: bool = True):
         if use_full_analysis:
             self.analyze_devo_ness(track)
         self.analyze_destruction(track)
 ```
 
-The basics are very simple: Make sure to extend `NendoAnalysisPlugin`, and make sure that your extended class has at
-least
-one method that is decorated with `@NendoAnalysisPlugin.run_track` or `@NendoAnalysisPlugin.run_collection`.
-If you used the official setup script, these steps are already taken care of.
+The basics are very simple: Make sure to extend `NendoAnalysisPlugin`, and make sure that your extended class has at least one method that is decorated with `@NendoAnalysisPlugin.run_track` or `@NendoAnalysisPlugin.run_collection`. If you used the official setup script, these steps are already taken care of.
 
-Then your plugin can have a few different methods that do the concrete metadata extraction and addition to
-the `NendoTrack`.
-These need to be decorated with `@NendoAnalysisPlugin.plugin_data`.
+Then your plugin can have a few different methods that do the concrete metadata extraction and addition to the `NendoTrack`. These need to be decorated with `@NendoAnalysisPlugin.plugin_data`.
 
-#### NendoAnalysis.plugin_data
+#### NendoAnalysisPlugin.plugin_data
 
 This decorator automatically adds all calculated metadata to the `NendoTrack`'s `List[PluginData]`.
 To learn more about `NendoTrack` or `PluginData` check out the API reference.

docs/development/embeddingplugindev.md

@@ -0,0 +1,129 @@
+# Writing an `EmbeddingPlugin`
+
+We'll go step by step through all the files you need to create and what they'll do.
+
+Remember the directory structure from before:
+
+```shell
+├── README.md
+├── pyproject.toml
+├── setup.py
+├── src
+│   └── nendo_plugin_embed_text
+│       ├── __init__.py
+│       ├── config.py
+│       └── plugin.py
+```
+
+Let's look into the different files now.
+
+## Files
+
+### plugin.py
+
+The most important file in your plugin. It is where the main plugin call code resides Let's go through it step by step.
+
+```python
+from logging import Logger
+from my_embedding_model import MyModel
+from nendo import Nendo, NendoConfig, NendoEmbeddingPlugin, NendoTrack
+import numpy.typing as npt
+
+from .config import EmbedTextConfig
+
+settings = EmbedTextConfig()
+
+class TextEmbedder(NendoEmbeddingPlugin):
+    """Find out how weird your music is."""
+    nendo_instance: Nendo = None
+    config: NendoConfig = None
+    logger: Logger = None
+    model: MyModel = None
+    device: str = None
+
+    def __init__(self, **data: Any):
+        """Initialize the plugin."""
+        super().__init__(**data)
+        self.model = MyModel.from_pretrained(settings.model).to(self.device)
+
+    def track_to_text(self, track: NendoTrack) -> str:
+        """Convert track to text."""
+        text = ""
+        for pd in track.get_plugin_data():
+            text += f"{pd.key}: {pd.value}; "
+        return text
+
+    @NendoEmbeddingPlugin.run_track
+    def embed_track(
+        self,
+        track: NendoTrack,
+    ) -> Tuple[str, npt.NDArray]:
+        text_representation = self.track_to_text(track)
+        return (
+            text_representation,
+            self.model.embed_text(text_representation)
+        )
+```
+
+The basics are very simple: Extend `NendoEmbeddingPlugin`, and make sure that your class has at least one method that is decorated with `@NendoEmbeddingPlugin.run_track`, `@NendoEmbeddingPlugin.run_collection`, `@NendoEmbeddingPlugin.run_signal_and_text` or `@NendoEmbeddingPlugin.run_text`.
+
+#### NendoEmbeddingPlugin.run_track
+
+The function receives the argument `track: NendoTrack` and returns a `Tuple` containing the embedded text as a string and the embedding as a `numpy.types.NDArray`.
+
+#### NendoEmbeddingPlugin.run_collection
+
+The function receives the argument `collection: NendoCollection` and returns a `Tuple` containing the embedded text as a string and the embedding as a `numpy.types.NDArray`.
+
+#### NendoEmbeddingPlugin.run_signal_and_text
+
+The function receives the arguments `signal: str` and `text: str` and returns a `Tuple` containing the embedded text as a string and the embedding as a `numpy.types.NDArray`.
+
+#### NendoEmbeddingPlugin.run_text
+
+The function receives the argument `text: str` and returns a `Tuple` containing the embedded text as a string and the embedding as a `numpy.types.NDArray`.
+
+### config.py
+
+```python
+from nendo import NendoConfig
+
+
+class EmbeddingTextConfig(NendoConfig):
+    """Configuration defaults for the text embedding plugin."""
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    my_default_param: bool = False
+```
+
+This class extends the base `NendoConfig` and allows you to define some default and overridable parameters for your plugin.
+It behaves just like `NendoConfig`, read up more on basic `nendo` configuration [here](../usage/config.md).
+
+### setup.py
+
+```python
+from distutils.core import setup
+
+if __name__ == "__main__":
+    setup(
+        name="nendo-plugin-embed-text",
+        version="0.1.0",
+        description="Nendo text embedding plugin",
+        author="Author Name <[email protected]>",
+    )
+```
+
+This is a standard `setup.py` file. You can read up more on how to configure it [in the python packaging docs](https://packaging.python.org/tutorials/packaging-projects/). You just need to define some basics like the name of your plugin, a version number and a description.
+Make sure to specify the same version in `src/nendo_plugin_embed_text/__init__.py` and in `pyproject.toml`.
+
+### pyproject.toml
+
+All nendo plugins use `pyproject.toml` to manage their dependencies.
+You can read up more on how to configure it [here](https://packaging.python.org/tutorials/packaging-projects/).
+
+## Finished plugin
+
+That's it!
+
+!!! success
+    When you're finished go back to the plugin development [overview](plugindev.md#running-a-plugin) and learn how to test and publish your plugin.

docs/development/plugindev.md

@@ -40,6 +40,15 @@ In nendo we currently differentiate between four types of plugins.
 
     - [nendo_plugin_fx_core](https://github.com/okio-ai/nendo_plugin_fx_core)
 
+=== "Embedding Plugins"
+
+    Plugins that compute vector embeddings of `NendoTrack`s are called embedding plugins.
+    They take either a track's signal or first create a text representation from it or take both and then embed it into a vector space. The resulting embeddings can be used for further processing, sampling, similarity search, etc.
+
+    Examples:
+
+    - [nendo_plugin_embed_clap](https://github.com/okio-ai/nendo_plugin_embed_clap)
+
 === "Library Plugins"
 
     !!! warning

docs/platform.md

@@ -0,0 +1,21 @@
+# Nendo Platform
+
+Nendo Platform is the feature-rich web application stack to develop and run new GUI-enabled tools that are based on Nendo Core and it's plugin ecosystem.
+
+## Requirements
+
+To run Nendo Platform, make sure you have `docker` and `docker-compose` (`>=1.28.0`) installed.
+
+!!! info "Operating System Requirements"
+    Currently, Linux is the only operating system that is officially supported by Nendo Platform. Feel free to try and run it on Mac OSX or Windows but expect certain Features to fail without significant modifications to its codebase.
+
+!!! info "GPU Requirements"
+    Nendo Platform needs a GPU with at least 24 GB of VRAM for all of its features to work properly. If your system does not have a GPU available, you can still run Nendo Platform in [CPU mode](platform/usage.md#cpu-only-mode) but expect certain tools to fail.
+
+Visit the following pages to learn more about how to run and use Nendo Platform:
+
+- [:octicons-arrow-right-24: Get Started](platform/usage.md)
+- [:octicons-arrow-right-24: Configuration Guide](platform/config.md)
+- [:octicons-arrow-right-24: Development Guide](platform/development.md)
+- [:octicons-arrow-right-24: Learn about the Nendo Platform API Server](platform/server/index.md)
+- [:octicons-arrow-right-24: Learn about the Nendo Platform Web Frontend](platform/web/index.md)