diff --git a/.gitignore b/.gitignore
index 3bf780b6..c256e842 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,5 @@
.idea
-.env
\ No newline at end of file
+.env
+
+redis-data
+venv
\ No newline at end of file
diff --git a/README.md b/README.md
index 14f33bb5..9f29f5d2 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,5 @@
+
+
## Introduction
+
A conversational search tool based on GraphRAG (Knowledge Graph) that built on top of [TiDB Vector](https://tidb.cloud/ai) and [LlamaIndex](https://github.com/run-llama/llama_index) and [DSPy](https://github.com/stanfordnlp/dspy).
-* **Live Demo**: [TiDB.AI](https://tidb.cloud/?utm_source=github&utm_medium=tidb.ai)
-* **Documentation**: [Docs](https://tidb.ai/docs/?utm_source=github&utm_medium=tidb.ai)
+- **Live Demo**: [TiDB.AI](https://tidb.cloud/?utm_source=github&utm_medium=tidb.ai)
+- **Documentation**: [Docs](https://tidb.ai/docs/?utm_source=github&utm_medium=tidb.ai)
## Features
+
1. **Perplexity-style Conversational Search page**: Our platform features an advanced built-in website crawler, designed to elevate your browsing experience. This crawler effortlessly navigates official and documentation sites, ensuring comprehensive coverage and streamlined search processes through sitemap URL scraping.
-![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/9cc87d32-14ac-47c6-b664-efa7ec53e751 "Image Title")
+ ![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/9cc87d32-14ac-47c6-b664-efa7ec53e751 "Image Title")
-You can even edit the Knowledge Graph to add more information or correct any inaccuracies. This feature is particularly useful for enhancing the search experience and ensuring that the information provided is accurate and up-to-date.
+ You can even edit the Knowledge Graph to add more information or correct any inaccuracies. This feature is particularly useful for enhancing the search experience and ensuring that the information provided is accurate and up-to-date.
-![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/7bc57b34-99b7-4c4b-a098-9ad33dd0dfdc "Image Title")
+ ![out-of-box-conversational-search](https://github.com/pingcap/tidb.ai/assets/1237528/7bc57b34-99b7-4c4b-a098-9ad33dd0dfdc "Image Title")
2. **Embeddable JavaScript Snippet**: Integrate our conversational search window effortlessly into your website by copying and embedding a simple JavaScript code snippet. This widget, typically placed at the bottom right corner of your site, facilitates instant responses to product-related queries.
![embeddable-javascript-snippet](https://github.com/pingcap/tidb.ai/assets/1237528/5a445231-a27a-4ae6-8287-a4f8cf7b64d0 "Image Title")
+## Deploy
+
+> **Prerequisites:**
+>
+> 1. Set up a [TiDB Serverless cluster](https://docs.pingcap.com/tidbcloud/tidb-cloud-quickstart).
+> 2. Install [Docker Compose](https://docs.docker.com/compose/install/).
+> 3. Install [Python 3.11](https://www.python.org/downloads/release/python-3110/).
+> 4. Install [Rye](https://rye.astral.sh/guide/installation/).
+
+1. Clone the repository:
+
+ ```bash
+ git clone https://github.com/pingcap/tidb.ai.git
+ cd tidb.ai
+ ```
+
+2. Copy and edit the `.env` file in the `backend` directory:
+
+ ```bash
+ cp ./backend/.env.example ./backend/.env
+ vim ./backend/.env # or use another text editor to edit this file
+ ```
+
+3. Migrate the database schema:
+
+ ```bash
+ cd backend
+ rye sync
+ make migrate
+ cd ..
+ ```
+
+4. Start the services using Docker Compose:
+
+ ```bash
+ docker compose up
+ ```
## Tech Stack
+
- [TiDB](https://pingcap.com/ai/?utm_source=github&utm_medium=tidb.ai) – Database to store chat history, vector, json, and analytic
- [LlamaIndex](https://www.llamaindex.ai/) - RAG framework
- [DSPy](https://github.com/stanfordnlp/dspy) - The framework for programming—not prompting—foundation models
@@ -33,7 +75,9 @@ You can even edit the Knowledge Graph to add more information or correct any ina
- [shadcn/ui](https://ui.shadcn.com/) - Design
## Contect Us
+
You can post topics on our [TiDB Community](https://ask.pingcap.com/) page.
## License
+
TiDB.AI is open-source under the Apache License, Version 2.0. You can [find it here](/LICENSE.txt).
diff --git a/backend/.env.example b/backend/.env.example
index 9dce7cfc..15281cb1 100644
--- a/backend/.env.example
+++ b/backend/.env.example
@@ -1,16 +1,19 @@
DEBUG=true
+SECRET_KEY=********************************
+
TIDB_DATABASE=test
-TIDB_HOST=gateway01.eu-central-1.prod.aws.tidbcloud.com
-TIDB_USER=*****.root
+TIDB_HOST='gateway**.**-****-*.prod.aws.tidbcloud.com'
+TIDB_USER='*****.root'
TIDB_PASSWORD=*******
-OPENAI_API_KEY=sk-********
-GOOGLE_API_KEY=*******
+OPENAI_API_KEY=sk-********************************
+# Optional
+GOOGLE_API_KEY=********************************
-LANGFUSE_SECRET_KEY=sk-*****
-LANGFUSE_PUBLIC_KEY=pk-lf-*****
+LANGFUSE_SECRET_KEY=sk-lf-********************************
+LANGFUSE_PUBLIC_KEY=pk-lf-********************************
LANGFUSE_HOST=https://us.cloud.langfuse.com
-CELERY_BROKER_URL=redis://127.0.0.1:6379/0
-CELERY_RESULT_BACKEND=redis://127.0.0.1:6379/0
+CELERY_BROKER_URL=redis://redis:6379/0
+CELERY_RESULT_BACKEND=redis://redis:6379/0
diff --git a/backend/requirements-dev.lock b/backend/requirements-dev.lock
index f57b6026..d07cd493 100644
--- a/backend/requirements-dev.lock
+++ b/backend/requirements-dev.lock
@@ -7,6 +7,7 @@
# all-features: false
# with-sources: false
# generate-hashes: false
+# universal: false
aiohttp==3.9.5
# via datasets
@@ -33,6 +34,8 @@ anyio==4.4.0
# via watchfiles
appdirs==1.4.4
# via ragas
+appnope==0.1.4
+ # via ipykernel
argon2-cffi==23.1.0
# via pwdlib
argon2-cffi-bindings==21.2.0
diff --git a/backend/requirements.lock b/backend/requirements.lock
index df3a6351..02e325f1 100644
--- a/backend/requirements.lock
+++ b/backend/requirements.lock
@@ -7,6 +7,7 @@
# all-features: false
# with-sources: false
# generate-hashes: false
+# universal: false
aiohttp==3.9.5
# via datasets
diff --git a/docker-compose.yml b/docker-compose.yml
index 9b5decd1..c82ceac0 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,11 +1,19 @@
name: tidb-ai
services:
+ redis:
+ image: redis:6.0.16
+ restart: always
+ volumes:
+ - ./redis-data:/data
+
backend:
image: tidb.ai/backend:local
build:
context: ./backend
restart: always
+ depends_on:
+ - redis
ports:
- "5000:80"
env_file:
@@ -22,6 +30,8 @@ services:
context: .
dockerfile: frontend/Dockerfile
restart: always
+ depends_on:
+ - backend
ports:
- 3000:3000
environment:
@@ -32,12 +42,6 @@ services:
max-size: "50m"
max-file: "6"
- redis:
- image: redis:6.0.16
- restart: always
- volumes:
- - ./redis-data:/data
-
background:
image: tidb.ai/backend:local
build:
@@ -45,6 +49,7 @@ services:
restart: always
depends_on:
- redis
+ - backend
ports:
- "5555:5555"
env_file:
diff --git a/frontend/app/src/pages/docs/_meta.json b/frontend/app/src/pages/docs/_meta.json
index 21e2e743..d8ed51fc 100644
--- a/frontend/app/src/pages/docs/_meta.json
+++ b/frontend/app/src/pages/docs/_meta.json
@@ -15,6 +15,7 @@
"title": "Deployment"
},
"requirements": "Requirements",
+ "quickstart": "Quick Start",
"deploy-with-docker": {
"title": "Deploy with Docker",
"display": "hidden"
diff --git a/frontend/app/src/pages/docs/quickstart.mdx b/frontend/app/src/pages/docs/quickstart.mdx
new file mode 100644
index 00000000..fab1b878
--- /dev/null
+++ b/frontend/app/src/pages/docs/quickstart.mdx
@@ -0,0 +1,43 @@
+# Quick Start
+
+This document provides instructions for deploying the entire RAG using Docker Compose.
+
+## Prerequisites
+
+1. Set up a [TiDB Serverless cluster](https://docs.pingcap.com/tidbcloud/tidb-cloud-quickstart).
+2. Install [Docker Compose](https://docs.docker.com/compose/install/).
+3. Install [Python 3.11](https://www.python.org/downloads/release/python-3110/).
+4. Install [Rye](https://rye.astral.sh/guide/installation/).
+
+## Deployment Steps
+
+1. Clone the repository:
+
+ ```bash
+ git clone https://github.com/pingcap/tidb.ai.git
+ cd tidb.ai
+ ```
+
+2. Copy and edit the `.env` file in the `backend` directory:
+
+ ```bash
+ cp ./backend/.env.example ./backend/.env
+ vim ./backend/.env # or use another text editor to edit this file
+ ```
+
+3. Migrate the database schema:
+
+ ```bash
+ cd backend
+ rye sync
+ make migrate
+ cd ..
+ ```
+
+4. Start the services using Docker Compose:
+
+ ```bash
+ docker compose up
+ ```
+
+That's it! You can now use TiDB.AI locally. You can also go to https://tidb.ai to experience the live demo.
\ No newline at end of file