ai2SpeckleApp:Texture Search_v0

Documentation

In the above diagram, There are three components to our "ai2SpeckleApp:matgen Web Application"

1. A Front-end React Page that loads a Speckle Stream Viewer(3D Object Loader) via an authentication page through your Speckle account.

   • AI Image Analyzer Application GUI Window: A Button that takes a snapshot of the stream and uses AI Render Image API to render the stream and feeds it to the AI Image Analyzer Application based on whether the user accepts it.
   • A Button that calls the AI image Analyzer Application that then applies material to the elements in the Speckle Stream.

2. An AI Image Renderer Application GUI Window This window does have a text prompt for the user to enter the Text Prompt which opens an "AI Image Rendered" window. The user has an option to either continue,re-render, or change the prompt for re-rendering. For the AI Image Rendering, when we click the AI Render or Rerender button, the button makes an API call to the free available state-of-the-art Image to Image AI Generators in the Generative AI community.

   We explored several Free Image to Image AI Renderers that offer free or paid pricing for API usage outlined below:

• Monster API using ZeroGPU from HuggingFace

• Automatic 111 Web UI using Google Colab

• Limewire Developer API

• Deep AI (Ai Image Generator API)-

  For this hackathon, the most promising solution for the Image-to-Image Generation API for our "AI Image Rendering Application" is:Stable Diffusion Hugging Face API

3. AI Image Analyzer Application GUI Window(AI Workflow diagram)

   When the user is satisfied with 2 and proceeds with the "Continue" button, it sends the Rendered Image to the AI Image Analyzer Application GUI Window. This GUI window has a Machine Learning pipeline that generates Material as JPG/PNG which are hosted on PolyHaven so the user can download with a downloadable link Under the hood of this machine learning pipeline, there are several steps like Image Segmentation, Image Recognition from the Hugging Face dataset, and Image Similarity search to identify the materials based on labels inside the datasets and lastly post-processing the segmented labeled image from AI render Image to new materials.

   This technique expands the material database for users to select and apply later either to their BIM Applications or CAD software like Unreal,Unity3D,etc.

   All with the Power of the Speckle !!

INSTALLATION

Please follow the steps on the authentication for installing this repository

FUTURE IMPROVEMENTS

Credits | Teammates

Josie Harrison (Machine Learning Developer) Marwan Elzainy(Fullstack Developer) Jordana Rosa(Idea Conception) Abhishek Shinde(Researcher|Documentation)