BREAKING CHANGE: add OpenAI files and vector store API (including database and model)

[RobinBially]

related to #81

Pull Request Checklist

Describe your changes before submitting a pull request.

Before submitting, make sure you've checked the following:

• Description: Provide a concise description of the changes made in this pull request.
• Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
• Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
• Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
• Testing: Have you written and run sufficient tests for validating the changes?
• Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
• Prefix: To cleary categorize this pull request, prefix the pull request title, using one of the following:
  • BREAKING CHANGE: Significant changes that may affect compatibility
  • build: Changes that affect the build system or external dependencies
  • ci: Changes to our continuous integration processes or workflows
  • chore: Refactor, cleanup, or other non-functional code changes
  • docs: Documentation update or addition
  • feat: Introduces a new feature or enhancement to the codebase
  • fix: Bug fix or error correction
  • i18n: Internationalization or localization changes
  • perf: Performance improvement
  • refactor: Code restructuring for better maintainability, readability, or scalability
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
  • test: Adding missing tests or correcting existing tests
  • WIP: Work in progress, a temporary label for incomplete or ongoing work

Changelog Entry

Description

Possibility to create and remove knowledge bases for pipelines via open ai files and vector store api

Changed/Added

• add Database and SQLAlchemy orm
• add Alembic migrations
• add 2 sub applications for files and vector stores
• add all endpoints from this API ref:
  • https://platform.openai.com/docs/api-reference/files
  • https://platform.openai.com/docs/api-reference/vector-stores
  • https://platform.openai.com/docs/api-reference/vector-stores-files
• added http client files and API tests

---

Additional Information

How to add a file to a pipeline vector store:

• Create a file (see example in upload-file.http)
• Create a vector store and add a pipeline id to the metadata field (example in create_vector_store.http)
• implement the add_vector_store_file and remove_vector_store_file functions in your pipeline
• Attach the file to the vector store by creating a vector store file (example in create_vector_store_file.http)

Screenshots or Videos