Open Data Analysis

Reverse engineering OpenAI's Advanced Data Analysis feature, formerly Code Interpreter. The steps, learnings and mistakes along the way.

Table of Contents

1. Introduction
2. The Journey: Building "Advanced Data Analysis"
   • 2.1 The Quick and Dirty
   • 2.2 Core Limitations
   • 2.3 Solution 1 - Prompt Engineering
     • 2.3.1 Pseudo-Prompts
   • 2.4 The Problems with Prompt Engineering
     • 2.4.1 Context Window Challenges
   • 2.5 The Reversal
     • 2.5.1 Limitations with Alignment
3. Key Investigation
   • 3.1 Solution 2 - Jupyter Server
   • 3.2 Key Limitations with Solution 2
     • 3.2.1 Serverless Scalability
   • 3.3 Solution 3 - JupyterHub
4. Configuring a Jupyter Server
   • 4.1 Selecting a Jupyter Server image
   • 4.2 Creating our custom Dockerfile
   • 4.3 Building our custom Dockerfile
   • 4.4 Running and Testing the Container Locally
   • 4.5 Running the Server Example
5. Configuring a JupyterHub
   • 5.1 Configuration
   • 5.2 Creating the infrastructure
   • 5.3 Deploying JupyterHub
   • 5.4 Configuring storage
   • 5.5 Running the hub example
   • 5.6 Managing user servers
6. Conclusion
   • 6.1 Key Takeaways
   • 6.2 Future Directions
   • 6.3 Acknowledgments
   • 6.4 Final Thoughts
7. References

---