[Bug]: No result with assistant file search

[patrickkasl]

Description

I have been able to get creating and using an assistant working. I also have the code to create a vector database and up load a file and relate it the the vector db and assistant.
Sadly i get no result from the file search whatever i try, and i tried a lot.
I am really hoping you can push me in the right direction or maybe its a known bug.

Important: When i create the assistant with my code it works perfect in the playground! Just not when i try it from my code.

this is my chat function:

   public function chat(Request $request)
{
    $messages = $request->input('messages');

    if (is_null($messages) || !is_array($messages)) {
        return response()->json(['error' => 'Invalid messages format.'], 400);
    }

    try {
        Log::info("Input messages: " . json_encode($messages));
        // dd('Step 1: Input messages', $messages);

        // Stap 1: Haal beschikbare bestanden op
        $filesResponse = $this->client->files()->list();
        $filesData = $filesResponse->toArray();
        Log::info("Available files: " . json_encode($filesData));
        // dd('Step 2: Available files', $filesData);

        // Controleer of het juiste bestand aanwezig is
        // $fileId = 'file-BXlBwaQuHXe452eIcRVus8iT';
        // $fileExists = false;
        // foreach ($filesData['data'] as $file) {
        //     if ($file['id'] === $fileId) {
        //         $fileExists = true;
        //         break;
        //     }
        // }

        // if (!$fileExists) {
        //     return response()->json(['error' => 'Specified file not found.'], 400);
        // }

        // Stap 2: Maak een Thread aan met berichten
        $threadResponse = $this->client->threads()->create([
            'messages' => $messages,
            'tool_resources' => [
                'file_search' => [
                    'vector_store_ids' => [$this->vectorStoreId],
                ],
            ],
        ]);

        $threadId = $threadResponse['id'];
        Log::info("Thread created: {$threadId}");
        // dd('Step 3: Thread created', $threadResponse);

        // Stap 3: Creëer een Run om een antwoord te krijgen
        $runResponse = $this->client->threads()->runs()->create($threadId, [
            'assistant_id' => $this->assistantId,
            'tools' => [
                [
                    'type' => 'file_search',
                    'file_search' => [
                        'max_num_results' => 50,
                    ],
                ],
            ],
        ]);

        $runId = $runResponse['id'];
        Log::info("Run created: {$runId}");
        // dd('Step 4: Run created', $runResponse);

        // Wacht tot de run is voltooid
        $completed = false;
        $maxAttempts = 30;
        $attempts = 0;

        while (!$completed && $attempts < $maxAttempts) {
            sleep(1);
            $runStatusResponse = $this->client->threads()->runs()->retrieve($threadId, $runId);
            $runStatusData = $runStatusResponse->toArray();
            Log::info("Run status: " . json_encode($runStatusData));
            // dd('Step 5: Run status', $runStatusData);

            if ($runStatusData['status'] === 'completed') {
                $completed = true;
            } elseif ($runStatusData['status'] === 'failed') {
                Log::error("Run failed: " . json_encode($runStatusData['last_error']));
                return response()->json(['error' => 'Run failed: ' . $runStatusData['last_error']['message']], 500);
            }

            $attempts++;
        }

        if (!$completed) {
            return response()->json(['error' => 'Run timed out.'], 500);
        }

        // Haal het laatste bericht op van de Run stappen
        $stepsResponse = $this->client->threads()->runs()->steps()->list($threadId, $runId);
        $steps = $stepsResponse->toArray();
        Log::info("Run steps: " . json_encode($steps));
        // dd('Step 6: Run steps', $steps);

        if (empty($steps['data'])) {
            return response()->json(['error' => 'No steps found in the run.'], 500);
        }

        $lastStep = end($steps['data']);
        Log::info("Last step: " . json_encode($lastStep));
        // dd('Step 7: Last step', $steps['data']);

        if ($lastStep['type'] === 'message_creation' && isset($lastStep['step_details']['message_creation']['message_id'])) {
            $messageId = $lastStep['step_details']['message_creation']['message_id'];

            // Haal het bericht op
            $messageResponse = $this->client->threads()->messages()->retrieve($threadId, $messageId);
            $messageData = $messageResponse->toArray();
            Log::info("Message response: " . json_encode($messageData));
            // dd('Step 8: Message response', $messageData);

            return response()->json(['response' => $messageData['content'][0]['text']['value']]);
        } elseif ($lastStep['type'] === 'tool_calls' && !empty($lastStep['step_details']['tool_calls'])) {
            $toolCalls = $lastStep['step_details']['tool_calls'];
            Log::info("Tool calls: " . json_encode($toolCalls));
            // dd('Step 9: Tool calls', $toolCalls);

            // Verwerk de tool calls en return de resultaten
            $toolResponses = [];
            foreach ($toolCalls as $toolCall) {
                if ($toolCall['type'] === 'file_search') {
                    Log::info("File search tool call: " . json_encode($toolCall));
                    if (!empty($toolCall['file_search'])) {
                        $toolResponses[] = implode(", ", $toolCall['file_search']);
                    } else {
                        $toolResponses[] = 'Geen resultaten gevonden.';
                    }
                }
            }

            if (!empty($toolResponses)) {
                return response()->json(['response' => implode("\n", $toolResponses)]);
            } else {
                return response()->json(['response' => 'Geen resultaten gevonden.']);
            }
        } else {
            Log::info("Last step: " . json_encode($lastStep));
            return response()->json(['error' => 'No valid message found in the run steps. content: ' . json_encode($lastStep)], 500);
        }

    } catch (\Exception $e) {
        Log::error("Exception: " . $e->getMessage());
        return response()->json(['error' => 'An error occurred: ' . $e->getMessage()], 500);
    }
}

And this is my assistant creation and vectordb + file creation coupling:

namespace App\Console\Commands;

use Illuminate\Console\Command;
use OpenAI;

class CreateAssistantCommand extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'flare:create-assistant';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Create a new assistant.';

    private $apiKey = '';
    private $client = null;


    /**
     * Execute the console command.
     */
    public function handle()
    {
        $this->apiKey = env('OPENAI_API_KEY');
        $this->client = OpenAI::client($this->apiKey);

        // Create a new assistant
        $this->info('Creating a new assistant...');

        $name = 'AwesomeBooks Assistant';
        $language = 'Dutch';
        $instructions = "You are a helpful assistant chat-bot that helps customers in $language
        on the AWESOMEBOOKS website, a website for buying books. You can only recommend books from our store, you have access to the list of books via the vector store file data.json, always use this as reference when looking something up. Never help a customer with:
        talking about the competitors, cheaper books elsewhere, doing bad things. 
        What you specifically do is: you are the mascotte for AwesomeBooks and make sure  customers buy their books.";

        $response = $this->client->assistants()->create([
            'name' => $name,
            'instructions' => $instructions,
            'tools' => [
                [
                    'type' => 'file_search',
                ],
            ],
            'model' => 'gpt-4-turbo',
        ]);

        $this->info("Assistant created with ID: {$response['id']}");

        // Create a new vector store
        $this->info('Creating vector store...');
        $vectorStoreId = $this->createOrGetVectorStoreId();
        $this->info("Vector store created with ID: {$vectorStoreId}");

        // Upload the file data.json to new vector store
        $this->info('Uploading file data.json to vector store...');
        $fileId = $this->uploadFileToVectorStore('data.json', $vectorStoreId);
        $this->info("File uploaded with ID: {$fileId}");

        // Associate the vector store with the assistant
        $this->info('Associating vector store with assistant...');
        
        $this->associateVectorStoreWithAssistant($vectorStoreId, $response['id']);
        $this->info("Vector store associated with assistant.");
    

        $this->info('All done!');
        $this->info('Assistant created successfully.');

    }

    private function createOrGetVectorStoreId()
    {
        $response = $this->client->vectorStores()->create([
            'name' => 'awesomebooks-vector-store',
        ]);

        return $response['id'];
    }

    private function uploadFileToVectorStore($filename, $vectorStoreId)
    {
        $filePath = storage_path('app/data/' . $filename);
        if (file_exists($filePath)) {
            $fileResponse = $this->client->files()->upload([
                'purpose' => 'assistants',
                'file' => fopen($filePath, 'r'),
            ]);

            $fileId = $fileResponse['id'];

            $this->client->vectorStores()->files()->create(
                vectorStoreId:$vectorStoreId, 
                parameters: [
                    'file_id' => $fileId,
                ]
            );

            $this->waitForFileProcessing($fileId);

            return $fileId;
        }

        throw new \Exception("File {$filename} not found and could not be uploaded.");
    }

    private function waitForFileProcessing($fileId)
    {
        $maxAttempts = 30; // Maximaal aantal pogingen om te voorkomen dat we vastlopen
        $attempts = 0;

        while ($attempts < $maxAttempts) {
            sleep(1); // Wacht een seconde voordat je opnieuw controleert
            $fileStatusResponse = $this->client->files()->retrieve($fileId);
            $fileStatus = $fileStatusResponse['status'];

            if ($fileStatus === 'processed') {
                return;
            }

            $attempts++;
        }

        throw new \Exception("File processing timed out.");
    }

    private function associateVectorStoreWithAssistant($vectorStoreId, $assistantId)
    {
        $this->client->assistants()->modify($assistantId, [
            'tool_resources' => [
                'file_search' => [
                    'vector_store_ids' => [$vectorStoreId],
                ],
            ],
        ]);
    }
}

Log:

[2024-06-23 16:17:49] local.INFO: Input messages: [{"role":"user","content":"geef me een boek en de prijs uit je data"}]  
[2024-06-23 16:17:50] local.INFO: Available files: {"object":"list","data":[{"id":"file-PzHboorZZkAj2dKSe1yiuhvt","object":"file","bytes":6134,"created_at":1719159384,"filename":"data.json","purpose":"assistants","status":"processed","status_details":null}]}  
[2024-06-23 16:17:50] local.INFO: Thread created: thread_S2TA14hQz6vSd6qufs0eE3Rm  
[2024-06-23 16:17:50] local.INFO: Run created: run_s08Efn4lDR378YLXZudCrjn6  
[2024-06-23 16:17:51] local.INFO: Run status: {"id":"run_s08Efn4lDR378YLXZudCrjn6","object":"thread.run","created_at":1719159470,"assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","status":"in_progress","started_at":1719159470,"expires_at":1719160070,"cancelled_at":null,"failed_at":null,"completed_at":null,"incomplete_details":null,"last_error":null,"model":"gpt-4-turbo","instructions":"You are a helpful assistant chat-bot that helps customers in Dutch\n        on the AWESOMEBOOKS website, a website for buying books. You can only recommend books from our store, you have access to the list of books via the vector store file data.json, always use this as reference when looking something up. Never help a customer with:\n        talking about the competitors, cheaper books elsewhere, doing bad things. \n        What you specifically do is: you are the mascotte for AwesomeBooks and make sure  customers buy their books.","tools":[{"type":"file_search"}],"metadata":[],"temperature":1,"top_p":1,"max_prompt_tokens":null,"max_completion_tokens":null,"truncation_strategy":{"type":"auto","last_messages":null},"tool_choice":"auto","response_format":"auto"}  
[2024-06-23 16:17:53] local.INFO: Run status: {"id":"run_s08Efn4lDR378YLXZudCrjn6","object":"thread.run","created_at":1719159470,"assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","status":"in_progress","started_at":1719159470,"expires_at":1719160070,"cancelled_at":null,"failed_at":null,"completed_at":null,"incomplete_details":null,"last_error":null,"model":"gpt-4-turbo","instructions":"You are a helpful assistant chat-bot that helps customers in Dutch\n        on the AWESOMEBOOKS website, a website for buying books. You can only recommend books from our store, you have access to the list of books via the vector store file data.json, always use this as reference when looking something up. Never help a customer with:\n        talking about the competitors, cheaper books elsewhere, doing bad things. \n        What you specifically do is: you are the mascotte for AwesomeBooks and make sure  customers buy their books.","tools":[{"type":"file_search"}],"metadata":[],"temperature":1,"top_p":1,"max_prompt_tokens":null,"max_completion_tokens":null,"truncation_strategy":{"type":"auto","last_messages":null},"tool_choice":"auto","response_format":"auto"}  
[2024-06-23 16:17:54] local.INFO: Run status: {"id":"run_s08Efn4lDR378YLXZudCrjn6","object":"thread.run","created_at":1719159470,"assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","status":"in_progress","started_at":1719159470,"expires_at":1719160070,"cancelled_at":null,"failed_at":null,"completed_at":null,"incomplete_details":null,"last_error":null,"model":"gpt-4-turbo","instructions":"You are a helpful assistant chat-bot that helps customers in Dutch\n        on the AWESOMEBOOKS website, a website for buying books. You can only recommend books from our store, you have access to the list of books via the vector store file data.json, always use this as reference when looking something up. Never help a customer with:\n        talking about the competitors, cheaper books elsewhere, doing bad things. \n        What you specifically do is: you are the mascotte for AwesomeBooks and make sure  customers buy their books.","tools":[{"type":"file_search"}],"metadata":[],"temperature":1,"top_p":1,"max_prompt_tokens":null,"max_completion_tokens":null,"truncation_strategy":{"type":"auto","last_messages":null},"tool_choice":"auto","response_format":"auto"}  
[2024-06-23 16:17:55] local.INFO: Run status: {"id":"run_s08Efn4lDR378YLXZudCrjn6","object":"thread.run","created_at":1719159470,"assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","status":"in_progress","started_at":1719159470,"expires_at":1719160070,"cancelled_at":null,"failed_at":null,"completed_at":null,"incomplete_details":null,"last_error":null,"model":"gpt-4-turbo","instructions":"You are a helpful assistant chat-bot that helps customers in Dutch\n        on the AWESOMEBOOKS website, a website for buying books. You can only recommend books from our store, you have access to the list of books via the vector store file data.json, always use this as reference when looking something up. Never help a customer with:\n        talking about the competitors, cheaper books elsewhere, doing bad things. \n        What you specifically do is: you are the mascotte for AwesomeBooks and make sure  customers buy their books.","tools":[{"type":"file_search"}],"metadata":[],"temperature":1,"top_p":1,"max_prompt_tokens":null,"max_completion_tokens":null,"truncation_strategy":{"type":"auto","last_messages":null},"tool_choice":"auto","response_format":"auto"}  
[2024-06-23 16:17:56] local.INFO: Run status: {"id":"run_s08Efn4lDR378YLXZudCrjn6","object":"thread.run","created_at":1719159470,"assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","status":"completed","started_at":1719159470,"expires_at":null,"cancelled_at":null,"failed_at":null,"completed_at":1719159475,"incomplete_details":null,"last_error":null,"model":"gpt-4-turbo","instructions":"You are a helpful assistant chat-bot that helps customers in Dutch\n        on the AWESOMEBOOKS website, a website for buying books. You can only recommend books from our store, you have access to the list of books via the vector store file data.json, always use this as reference when looking something up. Never help a customer with:\n        talking about the competitors, cheaper books elsewhere, doing bad things. \n        What you specifically do is: you are the mascotte for AwesomeBooks and make sure  customers buy their books.","tools":[{"type":"file_search"}],"metadata":[],"usage":{"prompt_tokens":4885,"completion_tokens":117,"total_tokens":5002},"temperature":1,"top_p":1,"max_prompt_tokens":null,"max_completion_tokens":null,"truncation_strategy":{"type":"auto","last_messages":null},"tool_choice":"auto","response_format":"auto"}  
[2024-06-23 16:17:57] local.INFO: Run steps: {"object":"list","data":[{"id":"step_HekPTK5reRfD1lrSMBqG32gn","object":"thread.run.step","created_at":1719159473,"run_id":"run_s08Efn4lDR378YLXZudCrjn6","assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","type":"message_creation","status":"completed","cancelled_at":null,"completed_at":1719159475,"expires_at":null,"failed_at":null,"last_error":null,"step_details":{"type":"message_creation","message_creation":{"message_id":"msg_ez5eYuze9jOdKElLP19nxrC5"}},"usage":{"prompt_tokens":4151,"completion_tokens":100,"total_tokens":4251}},{"id":"step_k3veQi1bm9aMvzWmG1yaK4A5","object":"thread.run.step","created_at":1719159471,"run_id":"run_s08Efn4lDR378YLXZudCrjn6","assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","type":"tool_calls","status":"completed","cancelled_at":null,"completed_at":1719159473,"expires_at":null,"failed_at":null,"last_error":null,"step_details":{"type":"tool_calls","tool_calls":[{"id":"call_4fZ6qoi5zCIofaowHS02wgwu","type":"file_search","file_search":[]}]},"usage":{"prompt_tokens":734,"completion_tokens":17,"total_tokens":751}}],"first_id":"step_HekPTK5reRfD1lrSMBqG32gn","last_id":"step_k3veQi1bm9aMvzWmG1yaK4A5","has_more":false}  
[2024-06-23 16:17:57] local.INFO: Last step: {"id":"step_k3veQi1bm9aMvzWmG1yaK4A5","object":"thread.run.step","created_at":1719159471,"run_id":"run_s08Efn4lDR378YLXZudCrjn6","assistant_id":"asst_7iohF1DZrzFt5mg1fjis1cYc","thread_id":"thread_S2TA14hQz6vSd6qufs0eE3Rm","type":"tool_calls","status":"completed","cancelled_at":null,"completed_at":1719159473,"expires_at":null,"failed_at":null,"last_error":null,"step_details":{"type":"tool_calls","tool_calls":[{"id":"call_4fZ6qoi5zCIofaowHS02wgwu","type":"file_search","file_search":[]}]},"usage":{"prompt_tokens":734,"completion_tokens":17,"total_tokens":751}}  
[2024-06-23 16:17:57] local.INFO: Tool calls: [{"id":"call_4fZ6qoi5zCIofaowHS02wgwu","type":"file_search","file_search":[]}]  
[2024-06-23 16:17:57] local.INFO: File search tool call: {"id":"call_4fZ6qoi5zCIofaowHS02wgwu","type":"file_search","file_search":[]}  

Hopefully you can get me out of this headbreaker, thanks!

Steps To Reproduce

1. Create assistant with file_search tool
2. Create vector store
3. Add file to vector store
4. Associate vector store with assistant
5. Chat with assistant to activate file_search
6. Vector store is found but result is always empty without any errors

OpenAI PHP Client Version

V0.10.1

PHP Version

8.2

Notes

No response

[MichaelMagomedov]

  
// create thread with message
$thread = $client->threads()->create([]);
$client->threads()->messages()->create($thread->id, [
    'role' => 'user',
    'content' => $prompt,
]);
// run search by vector store
$runParmse = $client->threads()->runs()->create($thread->id, [
    'model' => $modelType->value,
    'assistant_id' => $assistantId,
    'tool_choice' => [
        "type" => "file_search"
    ]
]);
do {
    $runStatus = $client->threads()->runs()->retrieve($thread->id, $runParmse->id);
    if ($runStatus->status === 'failed') {
        throw new RuntimeException('request error');
    }
    sleep(1);
} while ($runStatus->status !== 'completed');

// get response custom format
$runResponse = $client->threads()->runs()->create($thread->id, [
    'model' => $modelType->value,
    'assistant_id' => $assistantId,
    'tools' => [
        [
            'type' => 'function',
            'function' => [
                'name' => 'simple_function',
                'description' => ' ',
                'parameters' => $properties,
            ],
        ],
    ]
]);

do {
    $runStatus = $client->threads()->runs()->retrieve($thread->id, $runResponse->id);
    if ($runStatus->status === 'failed') {
        throw new RuntimeException('error');
    }
    sleep(1);
} while ($runStatus->status !== 'requires_action');

return json_decode($runStatus->requiredAction->submitToolOutputs->toolCalls[0]->function->arguments, true);

tthis is code work for me. Steps:

1. Create thread
2. Crate message
3. Run search with vector store (file_search)
4. Run any custom function for response json format