Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: hug llamaindex #91

Merged
merged 137 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
137 commits
Select commit Hold shift + click to select a range
ec859d3
use binary as id
634750802 Mar 18, 2024
2833b50
add llamaindex
634750802 Mar 18, 2024
aa83e51
add document store
634750802 Mar 18, 2024
d6cb0be
update
634750802 Mar 19, 2024
40da60d
update
634750802 Mar 19, 2024
f87acf1
refine flow
634750802 Mar 19, 2024
343cd3d
feat: add tidb vector store
Mini256 Mar 19, 2024
d7c9937
fix: align table schema to llamaindex node definition
Mini256 Mar 20, 2024
695b782
refactor: align document store to llamaindex dir
Mini256 Mar 20, 2024
f73cced
llamdaindex pipeline example
634750802 Mar 20, 2024
d17f9bc
update
634750802 Mar 20, 2024
529710c
fix
634750802 Mar 20, 2024
d1afa5a
add index store
634750802 Mar 20, 2024
9da356e
fix: refine TiDBVectorStore
Mini256 Mar 21, 2024
08270c7
fix: should process with uuid_to_bin when insert
Mini256 Mar 25, 2024
a986b4c
refactor: update schema
Mini256 Mar 25, 2024
0959490
Merge remote-tracking branch 'origin/main' into refactor
Mini256 Mar 25, 2024
58a56f2
fix sql
634750802 Mar 25, 2024
05992d4
Merge branch 'main' into refactor
Mini256 Mar 25, 2024
0758626
update
634750802 Mar 26, 2024
9ba0f36
implement importing and indexing tasks logic
634750802 Mar 26, 2024
c0c29e2
revert storage extension
634750802 Mar 26, 2024
5be8499
Merge remote-tracking branch 'origin/main' into refactor
Mini256 Apr 2, 2024
b190b92
update pnpm lock
Mini256 Apr 2, 2024
dcc44d3
fix PdfLoader type
Mini256 Apr 2, 2024
6ed955a
update
634750802 Apr 2, 2024
81606eb
wrapper API error
Mini256 Apr 2, 2024
42aa128
refine document_task schedule API and errors handle
Mini256 Apr 2, 2024
8bbd2cf
update
634750802 Apr 2, 2024
19b06d4
config: update initial config
Mini256 Apr 3, 2024
655d02f
fix: fix get index bug
Mini256 Apr 3, 2024
e4fd08c
update
634750802 Apr 3, 2024
3e4d97a
update
634750802 Apr 3, 2024
8542dc9
update
634750802 Apr 3, 2024
7485565
update
634750802 Apr 7, 2024
acec974
fix
634750802 Apr 7, 2024
2a70a25
using BLOB_READ_WRITE_TOKEN first if it exists
Mini256 Apr 7, 2024
760e0ba
fix build ci
Mini256 Apr 7, 2024
3cfa988
exclude pdfjs canvas dep
Mini256 Apr 7, 2024
50a3198
try to fix exceeded the unzipped maximum size of 250 MB
Mini256 Apr 7, 2024
a517ce5
fix
Mini256 Apr 7, 2024
992917c
fix
Mini256 Apr 7, 2024
71ee027
update
634750802 Apr 7, 2024
63c2c80
remove splitter abstraction
634750802 Apr 7, 2024
0e24e4b
remove prompting abstraction
634750802 Apr 7, 2024
b36e3ca
fix status check
634750802 Apr 7, 2024
c363b1c
todo: fix list source API
Mini256 Apr 7, 2024
5f7710e
fix user not found error message
Mini256 Apr 7, 2024
a619e3f
refine document import task error handle
Mini256 Apr 7, 2024
633255f
update
634750802 Apr 7, 2024
a134f6f
fix ui
634750802 Apr 7, 2024
38a90c3
fix ui
634750802 Apr 7, 2024
95b8539
fix
634750802 Apr 7, 2024
20441f7
fix
634750802 Apr 7, 2024
326a4dc
fix
634750802 Apr 7, 2024
d34e4aa
update cronjob
634750802 Apr 7, 2024
b939beb
update cronjob maxDuration
634750802 Apr 7, 2024
f2098bb
update speed
634750802 Apr 7, 2024
029702f
fix
634750802 Apr 7, 2024
a7c4caf
fix
634750802 Apr 7, 2024
d279a38
update
634750802 Apr 7, 2024
1f9f347
update index process duration limit
634750802 Apr 8, 2024
ef47cd1
fix
634750802 Apr 8, 2024
86f06ba
fix
634750802 Apr 8, 2024
8f13ffc
fix log
634750802 Apr 8, 2024
f3e24f5
fix sql
634750802 Apr 8, 2024
4c35fc4
update
634750802 Apr 8, 2024
84d5309
fix retrieve
634750802 Apr 8, 2024
a3343d9
update docs
634750802 Apr 8, 2024
4e4c555
fix
634750802 Apr 8, 2024
6022c82
fix
634750802 Apr 8, 2024
0eb3401
update prompt
634750802 Apr 9, 2024
529ff09
fix context retrieving
634750802 Apr 9, 2024
5bd78e2
remove log
634750802 Apr 9, 2024
b480478
speed up!
634750802 Apr 10, 2024
8e75c51
fix error messages length
634750802 Apr 10, 2024
3b6ad7c
fill document names
634750802 Apr 10, 2024
6532ab2
Merge branch 'main' into refactor
Mini256 Apr 10, 2024
890dcba
fix old api
634750802 Apr 10, 2024
4f7b9c6
add url_key to chat
634750802 Apr 10, 2024
3001d2d
fix delete chat
634750802 Apr 10, 2024
dc3b05f
feat: add llm reranker (#96)
Mini256 Apr 10, 2024
2ad7981
feat: support select chat engine
634750802 Apr 10, 2024
2b11309
fix
634750802 Apr 10, 2024
51ffbf8
reorder fields
634750802 Apr 10, 2024
af831ea
fix source title
634750802 Apr 11, 2024
0841b3f
feat: add bitdeer llm (#97)
Mini256 Apr 11, 2024
7f239b6
add index config page
634750802 Apr 11, 2024
9b75c24
fix dashboard page
634750802 Apr 11, 2024
f1697ed
fix
634750802 Apr 11, 2024
731c4de
fix
634750802 Apr 11, 2024
f1e025f
feat: refine ann index preload (#99)
Mini256 Apr 11, 2024
d251637
refactor authentication provider
Mini256 Apr 11, 2024
2f395fc
migrate settings
Mini256 Apr 12, 2024
f14c095
migrate extensions
Mini256 Apr 12, 2024
45f8fce
add dockerfile
634750802 Apr 12, 2024
ea44805
add source operations
634750802 Apr 12, 2024
db09627
refactor: refine v1 APIs (#101)
Mini256 Apr 12, 2024
54df890
protect settings API
Mini256 Apr 12, 2024
2b72f43
feat: impl readonly index config page
634750802 Apr 12, 2024
20b3b81
fix: chat session id
634750802 Apr 12, 2024
23ae988
fix: add public dir in dockerfile
634750802 Apr 12, 2024
4790055
refactor: remove v1 dir in core (#102)
Mini256 Apr 12, 2024
2c87567
chore: move operations to client dir
634750802 Apr 12, 2024
196426b
chore: remove docs
634750802 Apr 12, 2024
7994250
chore: remove unused
634750802 Apr 12, 2024
d1f3862
feat: add index summary stats
634750802 Apr 15, 2024
e567d56
fix initial table schema (#103)
wd0517 Apr 15, 2024
3019cba
fix ci
Mini256 Apr 15, 2024
3de7788
fix settings api
634750802 Apr 15, 2024
77458e9
update index editor pages
634750802 Apr 15, 2024
d2b7ec3
fix settings page refresh
634750802 Apr 15, 2024
38fb538
feat: enable llm provider as default reranker
Mini256 Apr 15, 2024
8fde44b
fix ci
Mini256 Apr 15, 2024
4001b13
refactor: use services replacing jobs (#104)
634750802 Apr 16, 2024
d5e8a16
refine chats API
Mini256 Apr 16, 2024
e6e04bf
add github action to build docker image (#106)
wd0517 Apr 16, 2024
e777ee0
refactor: optimize settings
634750802 Apr 16, 2024
e77856c
fix: flatten empty array settings
634750802 Apr 16, 2024
5fb38b8
fix: upload image with local fs storage
634750802 Apr 16, 2024
cb8e24f
try to fix Dockerfile
634750802 Apr 16, 2024
c667ff0
try to fix Dockerfile
634750802 Apr 16, 2024
03c15fb
prevent re-fetch chats
634750802 Apr 16, 2024
f68726a
ui: update side nav
634750802 Apr 16, 2024
ab7bfe6
ui: support enable index
634750802 Apr 16, 2024
96c47f1
fix settings
634750802 Apr 16, 2024
bb6e464
update github action
wd0517 Apr 16, 2024
0fad67d
support update chat engine
634750802 Apr 16, 2024
2de4fa4
try fix chat stream
634750802 Apr 16, 2024
7e8ab60
fix: settings schema
634750802 Apr 16, 2024
50b8a0e
ui: remove hard coded tidb.ai
634750802 Apr 16, 2024
b22deab
ui: support update chat engine's llm and reranker
634750802 Apr 16, 2024
744d90e
feat: support bitdeer embedding and chat api (#107)
Mini256 Apr 16, 2024
d7b6f01
ui: update form default values
634750802 Apr 16, 2024
d1454e1
ui: add wizard page
634750802 Apr 16, 2024
678c3aa
prepare merge to main
634750802 Apr 16, 2024
c53836c
remove import
634750802 Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules
.next
.store
34 changes: 34 additions & 0 deletions .github/workflows/build-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Build and Push Docker Image

on:
push:
branches:
- main
workflow_dispatch:

jobs:
build:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v4
-
name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
-
name: Build and push
uses: docker/build-push-action@v5
with:
context: .
file: ./Dockerfile
push: true
tags: |
${{ secrets.DOCKERHUB_USERNAME }}/tidb.ai:latest
${{ secrets.DOCKERHUB_USERNAME }}/tidb.ai:${{ github.sha }}
36 changes: 36 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
FROM node:20-alpine AS base
# 1. Install dependencies only when needed
FROM base AS deps
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
WORKDIR /app
# Install dependencies based on the preferred package manager
COPY . .
RUN corepack enable
RUN pnpm i

# 2. Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
# This will do the trick, use the corresponding env file for each environment.
# COPY .env.production.sample .env.production
RUN corepack enable
RUN STANDALONE_BUILD=1 pnpm run build

# 3. Production image, copy all the files and run next
FROM base AS runner
WORKDIR /app
ENV NODE_ENV=production
RUN corepack enable
RUN addgroup -g 1001 -S nodejs
RUN adduser -S nextjs -u 1001
# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
COPY public ./public
USER nextjs
EXPOSE 3000
CMD ["node", "server.js"]
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,6 @@ For deploying the application to Vercel, you can use the 'Deploy to Vercel' butt
* Adapt the system for AWS Redrock Claude3, with an anticipated completion date of April 15th.
* Finalize the "How It Works" series of documentation by April 30th.


## License

TiDB.AI is open-source under the Apache License, Version 2.0. You can [find it here](/LICENSE.txt).
279 changes: 0 additions & 279 deletions ddl/0-initial-ddl.sql

This file was deleted.

Loading