Skip to content
/ JARVIB Public

Just A Rather Very Intelligent Bot that can create dynamic 3D worlds using Teams AI Library

License

Notifications You must be signed in to change notification settings

davrous/JARVIB

Repository files navigation

AI in Microsoft Teams: J.A.R.V.I.B.

Your custom Microsoft Teams AI Copilot to build interactive dynamic 3D worlds

Logo of JARVIB

JARVIB (Just A Rather Very Intelligent Bot) is your personnal Copilot to help you designing 3D worlds in a collaborative way inside a personnal conversation or in a Teams meeting. Ask him to create various Babylon.js primitive or to search for specific models to load them into the scene. You can also ask to the AI to visually describe the scene for accessibility reasons. Look at possible prompts in promptsExamples.txt

View it in action

Image alt text

JARVIB is using the Teams AI library to map users' intent to 3 different type of actions:

  • codeToExecute which will take the Babylon.js code generated by the LLM and send it to the web page containing the 3D canvas to execute it via WebSocket
  • listAvailableModel which will use the same API as PowerPoint to find for existing 3D models and return a list of available models to load through an Adaptive Card list.
  • loadThisModel which will load one of the models exposed in the list displayed before

You have also 3 commands available:

  • /reset will clear the conversation state, clean the various objects states and reload the web page containing the 3D canvas to start from scratch
  • /describe will ask to the AI, via the text-davinci-003 LLM model to visually describe the scene
  • /fullcode will return all Babylon.js code executed so far so you can copy/paste it in your project or in the Babylon.js Playground

Look at the /src/prompts in /ChatGPT/skprompt.txt to check how the AI is prompted and check the /ChatGPT/actions.json file to understand how the various actions are described for the 3 Teams AI actions.

The prompt configuration:

Pretend you're an expert in Babylon.js, the JavaScript WebGL 3D engine. 

Assume there is already an existing Babylon.js scene and engine so you don't have to create them, just generate the code to add into an existing program.
Use the scene and engine objects directly. 

Pay attention when trying to access previously created Meshes by getting access to them via their name rather than assuming the associated variable is already created.
When writing a new code, consider all the previous one you've generated to be sure the new code will be consistent with the previous one.
Remember about the already created meshes, animations or any other specific ressources before trying to create them or reuse them.

The JSON schema to help the LLM to map to the defined actions:

[
    {
        "name": "codeToExecute",
        "description": "Returns the Babylon.js JavaScript code matching the user intent, building the next code working in a consistent way with the previous code already executed",
        "parameters": {
            "type": "object",
            "properties": {
                "code": {
                    "type": "string",
                    "description": "The JavaScript code to execute next"
                }
            },
            "required": [
                "code"
            ]
        }
    },
    {
        "name": "listAvailableModel",
        "description": "List the available 3D models we can load from the library",
        "parameters": {
            "type": "object",
            "properties": {
                "nameOfTheModel": {
                    "type": "string",
                    "description": "The name of the model to search inside the library"
                }
            },
            "required": [
                "nameOfTheModel"
            ]
        }
    },
    {
        "name": "loadThisModel",
        "description": "Load the 3D model specified by the user",
        "parameters": {
            "type": "object",
            "properties": {
                "nameOfTheModel": {
                    "type": "string",
                    "description": "The name of the model to load from the library"
                }
            },
            "required": [
                "nameOfTheModel"
            ]
        }
    }
]

Setting up the sample

  1. Clone the repository

    git clone https://github.com/davrous/JARVIB.git
  2. Install the Teams Toolkit: https://learn.microsoft.com/en-us/microsoftteams/platform/toolkit/install-teams-toolkit

  3. Set your Azure OpenAI key, endpoints and model names or your OpenAI key in the .env file.

    For Azure OpenAI, pay attention to use the "Deployment name" and not the "Model name" as the value for the properties in the .env file. If you've got an error 404 during execution, it's because you either entered the wrong endpoint or the wrong deployment name.

  4. Press F5!

    The Team Toolkit will do a lot of magic for you by:

    • installing all dependencies and building the TypeScript code
    • registering the bot in the Azure Active Directory
    • building the Teams manifest file, zipping the resources to upload them in the Teams Developer Portal
    • creating 2 tunnels to expose your localhost on public URLs using VS Dev Tunnels, you can find the details in /.vscode/tasks.json
      • 1 for the bot itself mapped on port 3978
      • 1 for the tab to be used inside a Teams meeting mapped on port 3000
    • launching the Teams web app to let you install the bot / meeting extension
  5. If everything goes well, you should first be able that the 2 tunnels have been created:

Terminal showing the VS Dev Tunnels running fine

  1. Then, the webserver will be launched and you'll be asked to install JARVIB in your developer tenant:

Terminal showing the VS Dev Tunnels running fine

You can either simply press the "Add" button and this will install the app to let you discuss with it directly as a classical bot.

You can also add it to an existing meeting to be able to replicate the full experience showcased in the video.

Interacting with the bot

The easiest way to try it is to install it using the "Add" button as a bot, then open http://localhost:3000/debug.html on the side open a simple canvas 3D websocket client. Then ask to the bot to create 3D content, load some models and so on:

screenshot of the Teams AI bot with the canvas scene on its right

The other option to replicate the full experience is to load the app inside an existing meeting:

screenshot the Teams app installation in an existing meeting found

You need to accept connecting to the VS Dev Tunnel redirecting to your localhost developer machine:

screenshot VS Dev Tunnels warning message

Finally, save JARVIB as an app part of this chosen Teams meeting:

screenshot VS Dev Tunnels warning message

Now, launch the Teams meeting alone or with someone else to collaborate with:

Teams meeting launched with the JARVIB app visible in the bar

Click on the JARVIB icon and then click on the "Share to meeting" button:

Teams meeting with the sidebar window and the meeting stage showing the 3D canvas

Finally, call the AI bot via @JARVIB-local following your order:

JARVIB fully working in a Teams meeting

Further reading

About

Just A Rather Very Intelligent Bot that can create dynamic 3D worlds using Teams AI Library

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published