diff --git a/.gitignore b/.gitignore index 3bf780b6..c256e842 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,5 @@ .idea -.env \ No newline at end of file +.env + +redis-data +venv \ No newline at end of file diff --git a/README.md b/README.md index 14f33bb5..9f29f5d2 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ + +

TiDB.AI

@@ -6,26 +8,66 @@
## Introduction + A conversational search tool based on GraphRAG (Knowledge Graph) that built on top of [TiDB Vector](https://tidb.cloud/ai) and [LlamaIndex](https://github.com/run-llama/llama_index) and [DSPy](https://github.com/stanfordnlp/dspy). -* **Live Demo**: [TiDB.AI](https://tidb.cloud/?utm_source=github&utm_medium=tidb.ai) -* **Documentation**: [Docs](https://tidb.ai/docs/?utm_source=github&utm_medium=tidb.ai) +- **Live Demo**: [TiDB.AI](https://tidb.cloud/?utm_source=github&utm_medium=tidb.ai) +- **Documentation**: [Docs](https://tidb.ai/docs/?utm_source=github&utm_medium=tidb.ai) ## Features + 1. **Perplexity-style Conversational Search page**: Our platform features an advanced built-in website crawler, designed to elevate your browsing experience. This crawler effortlessly navigates official and documentation sites, ensuring comprehensive coverage and streamlined search processes through sitemap URL scraping. -![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/9cc87d32-14ac-47c6-b664-efa7ec53e751 "Image Title") + ![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/9cc87d32-14ac-47c6-b664-efa7ec53e751 "Image Title") -You can even edit the Knowledge Graph to add more information or correct any inaccuracies. This feature is particularly useful for enhancing the search experience and ensuring that the information provided is accurate and up-to-date. + You can even edit the Knowledge Graph to add more information or correct any inaccuracies. This feature is particularly useful for enhancing the search experience and ensuring that the information provided is accurate and up-to-date. -![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/7bc57b34-99b7-4c4b-a098-9ad33dd0dfdc "Image Title") + ![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/7bc57b34-99b7-4c4b-a098-9ad33dd0dfdc "Image Title") 2. **Embeddable JavaScript Snippet**: Integrate our conversational search window effortlessly into your website by copying and embedding a simple JavaScript code snippet. This widget, typically placed at the bottom right corner of your site, facilitates instant responses to product-related queries. ![embeddable-javascript-snippet](https://github.com/pingcap/tidb.ai/assets/1237528/5a445231-a27a-4ae6-8287-a4f8cf7b64d0 "Image Title") +## Deploy + +> **Prerequisites:** +> +> 1. Set up a [TiDB Serverless cluster](https://docs.pingcap.com/tidbcloud/tidb-cloud-quickstart). +> 2. Install [Docker Compose](https://docs.docker.com/compose/install/). +> 3. Install [Python 3.11](https://www.python.org/downloads/release/python-3110/). +> 4. Install [Rye](https://rye.astral.sh/guide/installation/). + +1. Clone the repository: + + ```bash + git clone https://github.com/pingcap/tidb.ai.git + cd tidb.ai + ``` + +2. Copy and edit the `.env` file in the `backend` directory: + + ```bash + cp ./backend/.env.example ./backend/.env + vim ./backend/.env # or use another text editor to edit this file + ``` + +3. Migrate the database schema: + + ```bash + cd backend + rye sync + make migrate + cd .. + ``` + +4. Start the services using Docker Compose: + + ```bash + docker compose up + ``` ## Tech Stack + - [TiDB](https://pingcap.com/ai/?utm_source=github&utm_medium=tidb.ai) – Database to store chat history, vector, json, and analytic - [LlamaIndex](https://www.llamaindex.ai/) - RAG framework - [DSPy](https://github.com/stanfordnlp/dspy) - The framework for programming—not prompting—foundation models @@ -33,7 +75,9 @@ You can even edit the Knowledge Graph to add more information or correct any ina - [shadcn/ui](https://ui.shadcn.com/) - Design ## Contect Us + You can post topics on our [TiDB Community](https://ask.pingcap.com/) page. ## License + TiDB.AI is open-source under the Apache License, Version 2.0. You can [find it here](/LICENSE.txt). diff --git a/backend/.env.example b/backend/.env.example index 9dce7cfc..15281cb1 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -1,16 +1,19 @@ DEBUG=true +SECRET_KEY=******************************** + TIDB_DATABASE=test -TIDB_HOST=gateway01.eu-central-1.prod.aws.tidbcloud.com -TIDB_USER=*****.root +TIDB_HOST='gateway**.**-****-*.prod.aws.tidbcloud.com' +TIDB_USER='*****.root' TIDB_PASSWORD=******* -OPENAI_API_KEY=sk-******** -GOOGLE_API_KEY=******* +OPENAI_API_KEY=sk-******************************** +# Optional +GOOGLE_API_KEY=******************************** -LANGFUSE_SECRET_KEY=sk-***** -LANGFUSE_PUBLIC_KEY=pk-lf-***** +LANGFUSE_SECRET_KEY=sk-lf-******************************** +LANGFUSE_PUBLIC_KEY=pk-lf-******************************** LANGFUSE_HOST=https://us.cloud.langfuse.com -CELERY_BROKER_URL=redis://127.0.0.1:6379/0 -CELERY_RESULT_BACKEND=redis://127.0.0.1:6379/0 +CELERY_BROKER_URL=redis://redis:6379/0 +CELERY_RESULT_BACKEND=redis://redis:6379/0 diff --git a/backend/requirements-dev.lock b/backend/requirements-dev.lock index f57b6026..d07cd493 100644 --- a/backend/requirements-dev.lock +++ b/backend/requirements-dev.lock @@ -7,6 +7,7 @@ # all-features: false # with-sources: false # generate-hashes: false +# universal: false aiohttp==3.9.5 # via datasets @@ -33,6 +34,8 @@ anyio==4.4.0 # via watchfiles appdirs==1.4.4 # via ragas +appnope==0.1.4 + # via ipykernel argon2-cffi==23.1.0 # via pwdlib argon2-cffi-bindings==21.2.0 diff --git a/backend/requirements.lock b/backend/requirements.lock index df3a6351..02e325f1 100644 --- a/backend/requirements.lock +++ b/backend/requirements.lock @@ -7,6 +7,7 @@ # all-features: false # with-sources: false # generate-hashes: false +# universal: false aiohttp==3.9.5 # via datasets diff --git a/docker-compose.yml b/docker-compose.yml index 9b5decd1..c82ceac0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,11 +1,19 @@ name: tidb-ai services: + redis: + image: redis:6.0.16 + restart: always + volumes: + - ./redis-data:/data + backend: image: tidb.ai/backend:local build: context: ./backend restart: always + depends_on: + - redis ports: - "5000:80" env_file: @@ -22,6 +30,8 @@ services: context: . dockerfile: frontend/Dockerfile restart: always + depends_on: + - backend ports: - 3000:3000 environment: @@ -32,12 +42,6 @@ services: max-size: "50m" max-file: "6" - redis: - image: redis:6.0.16 - restart: always - volumes: - - ./redis-data:/data - background: image: tidb.ai/backend:local build: @@ -45,6 +49,7 @@ services: restart: always depends_on: - redis + - backend ports: - "5555:5555" env_file: diff --git a/frontend/app/src/pages/docs/_meta.json b/frontend/app/src/pages/docs/_meta.json index 21e2e743..d8ed51fc 100644 --- a/frontend/app/src/pages/docs/_meta.json +++ b/frontend/app/src/pages/docs/_meta.json @@ -15,6 +15,7 @@ "title": "Deployment" }, "requirements": "Requirements", + "quickstart": "Quick Start", "deploy-with-docker": { "title": "Deploy with Docker", "display": "hidden" diff --git a/frontend/app/src/pages/docs/quickstart.mdx b/frontend/app/src/pages/docs/quickstart.mdx new file mode 100644 index 00000000..fab1b878 --- /dev/null +++ b/frontend/app/src/pages/docs/quickstart.mdx @@ -0,0 +1,43 @@ +# Quick Start + +This document provides instructions for deploying the entire RAG using Docker Compose. + +## Prerequisites + +1. Set up a [TiDB Serverless cluster](https://docs.pingcap.com/tidbcloud/tidb-cloud-quickstart). +2. Install [Docker Compose](https://docs.docker.com/compose/install/). +3. Install [Python 3.11](https://www.python.org/downloads/release/python-3110/). +4. Install [Rye](https://rye.astral.sh/guide/installation/). + +## Deployment Steps + +1. Clone the repository: + + ```bash + git clone https://github.com/pingcap/tidb.ai.git + cd tidb.ai + ``` + +2. Copy and edit the `.env` file in the `backend` directory: + + ```bash + cp ./backend/.env.example ./backend/.env + vim ./backend/.env # or use another text editor to edit this file + ``` + +3. Migrate the database schema: + + ```bash + cd backend + rye sync + make migrate + cd .. + ``` + +4. Start the services using Docker Compose: + + ```bash + docker compose up + ``` + +That's it! You can now use TiDB.AI locally. You can also go to https://tidb.ai to experience the live demo. \ No newline at end of file