gpt4v.md

Using GPT-4 Turbo with Vision

This repository now includes an example of integrating GPT-4 Turbo with Vision with Azure AI Search. This feature enables indexing and searching images and graphs, such as financial documents, in addition to text-based content.

Feature Overview

• Document Handling: Source documents are split into pages and saved as PNG files in blob storage. Each file's name and page number are embedded for reference.
• Data Extraction: Text data is extracted using OCR.
• Data Indexing: Text and image embeddings, generated using Azure AI Vision (Azure AI Vision Embeddings), are indexed in Azure AI Search along with the raw text.
• Search and Response: Searches can be conducted using vectors or hybrid methods. Responses are generated by GPT-4 Turbo with Vision based on the retrieved content.

Getting Started

Prerequisites

• Create a Computer Vision account in Azure Portal first, so that you can agree to the Responsible AI terms for that resource. You can delete that account after agreeing.

• The ability to deploy a GPT-4 Turbo with Vision model in the supported regions. If you're not sure, try to create a deployment from your Azure OpenAI deployments page. You should be able to select:

  Model	Version
  gpt-4	vision-preview

• Ensure that you can deploy the Azure OpenAI resource group in a region where all required components are available:

  • Azure Open AI models
    • gpt-35-turbo
    • text-embedding-ada-002
    • gpt-4v
  • Azure AI Vision

Setup and Usage

1. Update repository: Pull the latest changes.

2. Enable GPT-4 Turbo with Vision:

   First, make sure you do not have integrated vectorization enabled, since that is currently incompatible:

   azd env set USE_FEATURE_INT_VECTORIZATION false

   Then set the environment variable for enabling vision support:

   azd env set USE_GPT4V true

   When set, that flag will provision a Computer Vision resource and GPT-4-vision model, upload image versions of PDFs to Blob storage, upload embeddings of images in a new imageEmbedding field, and enable the vision approach in the UI.

3. Clean old deployments (optional): Run azd down --purge for a fresh setup.

4. Start the application: Execute azd up to build, provision, deploy, and initiate document preparation.

5. Web Application Usage:

   • Access the developer options in the web app and select "Use GPT-4 Turbo with Vision".
   • Sample questions will be updated for testing.
   • Interact with the questions to view responses.
   • The 'Thought Process' tab shows the retrieved data and its processing by GPT-4 Turbo with Vision.

Feel free to explore and contribute to enhancing this feature. For questions or feedback, use the repository's issue tracker.