Skip to content

Commit

Permalink
Update openai
Browse files Browse the repository at this point in the history
  • Loading branch information
@kesmit13
kesmit13 committed Nov 21, 2023
1 parent 47df567 commit b945c84
Showing 1 changed file with 40 additions and 27 deletions.
67 changes: 40 additions & 27 deletions notebooks/evaluating-llms-with-uptrain/notebook.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "7e46bdec-aeb7-4d3f-aea6-9aefd9e834cb",
"id": "391591bd-e6b0-4b53-84d1-5e579bf77e2a",
"metadata": {},
"source": [
"<div id=\"singlestore-header\" style=\"display: flex; background-color: rgba(209, 153, 255, 0.25); padding: 5px;\">\n",
Expand Down Expand Up @@ -47,15 +47,16 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Create a workspace in your workspace group\n",
"### Create a workspace in your workspace group\n",
"\n",
"S-00 is sufficient."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2. Create a Database named evaluate_llm"
"### Create a Database named evaluate_llm"
]
},
{
Expand Down Expand Up @@ -84,7 +85,7 @@
"outputs": [],
"source": [
"!pip install uptrain --quiet\n",
"!pip install openai==0.28.1 --quiet\n",
"!pip install openai==1.3.3 --quiet\n",
"!pip install langchain --quiet\n",
"!pip install singlestoredb --quiet\n",
"!pip install tiktoken --quiet"
Expand All @@ -105,7 +106,12 @@
"source": [
"import getpass\n",
"import os\n",
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass('OpenAI API Key: ')"
"\n",
"os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key: ')\n",
"\n",
"import openai\n",
"\n",
"client = openai.OpenAI()"
]
},
{
Expand Down Expand Up @@ -151,7 +157,7 @@
"source": [
"from langchain.document_loaders import WebBaseLoader\n",
"\n",
"loader = WebBaseLoader(\"https://cloud.google.com/vertex-ai/docs/generative-ai/learn/generative-ai-studio\")\n",
"loader = WebBaseLoader('https://cloud.google.com/vertex-ai/docs/generative-ai/learn/generative-ai-studio')\n",
"data = loader.load()"
]
},
Expand All @@ -170,7 +176,7 @@
"source": [
"from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
"\n",
"text_splitter = RecursiveCharacterTextSplitter(chunk_size = 200, chunk_overlap = 0)\n",
"text_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=0)\n",
"all_splits = text_splitter.split_documents(data)"
]
},
Expand All @@ -187,21 +193,23 @@
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from langchain.vectorstores import SingleStoreDB\n",
"from langchain.embeddings import OpenAIEmbeddings\n",
"from singlestoredb import create_engine\n",
"import os\n",
"\n",
"conn = create_engine().connect()\n",
"\n",
"vectorstore = SingleStoreDB.from_documents(documents=all_splits, embedding= OpenAIEmbeddings(), table_name = \"vertex_ai_docs_chunk_size_200\")"
"vectorstore = SingleStoreDB.from_documents(documents=all_splits,\n",
" embedding=OpenAIEmbeddings(),\n",
" table_name='vertex_ai_docs_chunk_size_200')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Setting Up the QA Prompt**: Once our data is processed and stored, we can use it to answer queries. The following cell defines a \"generate_llm_response\" which finds the document closest to the given question via vector similarity search and uses OpenAI's GPT-3.5-Turbo to generate the response."
"**Setting Up the QA Prompt**: Once our data is processed and stored, we can use it to answer queries. The following cell defines a `generate_llm_response` which finds the document closest to the given question via vector similarity search and uses OpenAI's GPT-3.5-Turbo to generate the response."
]
},
{
Expand All @@ -210,9 +218,8 @@
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"def generate_llm_response(question, vectorstore):\n",
" documents = vectorstore.similarity_search(question, k= 1)\n",
" documents = vectorstore.similarity_search(question, k=1)\n",
" context = \" , \".join([x.page_content for x in documents])\n",
"\n",
" prompt = f\"\"\"\n",
Expand All @@ -224,9 +231,9 @@
" Answer:\n",
" \"\"\"\n",
"\n",
" response = openai.ChatCompletion.create(\n",
" response = client.chat.completions.create(\n",
" model=\"gpt-3.5-turbo\", messages=[{\"role\": \"system\", \"content\": prompt}], temperature=0.1\n",
" ).choices[0][\"message\"][\"content\"]\n",
" ).choices[0].message.content\n",
"\n",
" return [{'question': question, 'context': context, 'response': response}]"
]
Expand Down Expand Up @@ -292,7 +299,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Running Evaluations using UpTrain**: We now define a set of questions to test our bot upon and evaluate the quality of responses. UpTrain provides an APIClient that can be initialized with UPTRAIN_API_KEY. It provides a \"log_and_evaluate\" method which takes the input data to be evaluated along with the list of checks to be run. It returns the scores along with explanations."
"**Running Evaluations using UpTrain**: We now define a set of questions to test our bot upon and evaluate the quality of responses. UpTrain provides an APIClient that can be initialized with `UPTRAIN_API_KEY`. It provides a `log_and_evaluate` method which takes the input data to be evaluated along with the list of checks to be run. It returns the scores along with explanations."
]
},
{
Expand All @@ -302,13 +309,14 @@
"outputs": [],
"source": [
"from uptrain import APIClient\n",
"eval_client = APIClient(uptrain_api_key = UPTRAIN_API_KEY)\n",
"\n",
"_ = eval_client.log_and_evaluate(\n",
" project_name=\"VertexAI-QnA-Bot-Evals\",\n",
"eval_client = APIClient(uptrain_api_key=UPTRAIN_API_KEY)\n",
"\n",
"eval_client.log_and_evaluate(\n",
" project_name='VertexAI-QnA-Bot-Evals',\n",
" data=results,\n",
" checks=[Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY]\n",
")"
");"
]
},
{
Expand Down Expand Up @@ -345,9 +353,11 @@
"metadata": {},
"outputs": [],
"source": [
"text_splitter = RecursiveCharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)\n",
"text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
"all_splits = text_splitter.split_documents(data)\n",
"vectorstore_new = SingleStoreDB.from_documents(documents=all_splits, embedding= OpenAIEmbeddings(), table_name = \"vertex_ai_docs_chunk_size_1000\")"
"vectorstore_new = SingleStoreDB.from_documents(documents=all_splits,\n",
" embedding=OpenAIEmbeddings(),\n",
" table_name='vertex_ai_docs_chunk_size_1000')"
]
},
{
Expand Down Expand Up @@ -381,8 +391,11 @@
"metadata": {},
"outputs": [],
"source": [
"_ = [x.update({\"chunk_size\": 200}) for x in results]\n",
"_ = [x.update({\"chunk_size\": 1000}) for x in results_larger_chunk]"
"for x in results:\n",
" x.update({'chunk_size': 200})\n",
"\n",
"for x in results_larger_chunk:\n",
" x.update({'chunk_size': 1000})"
]
},
{
Expand All @@ -398,12 +411,12 @@
"metadata": {},
"outputs": [],
"source": [
"_ = eval_client.evaluate_experiments(\n",
" project_name=\"VertexAI-QnA-Bot-Chunk-Size-Experiments\",\n",
"eval_client.evaluate_experiments(\n",
" project_name='VertexAI-QnA-Bot-Chunk-Size-Experiments',\n",
" data=results + results_larger_chunk,\n",
" checks=[Evals.CONTEXT_RELEVANCE],\n",
" exp_columns=['chunk_size']\n",
")"
");"
]
},
{
Expand All @@ -420,7 +433,7 @@
},
{
"cell_type": "markdown",
"id": "c2bb28aa-3def-49e8-8a57-1be7f4c1c4d1",
"id": "62f5d31e-637c-4e97-8eee-34c529d21e8a",
"metadata": {},
"source": [
"<div id=\"singlestore-footer\" style=\"background-color: rgba(194, 193, 199, 0.25); height:2px; margin-bottom:10px\"></div>\n",
Expand Down

0 comments on commit b945c84

Please sign in to comment.