- [8.4.24] Model weights are out! You can download them here!
- [8.4.24] After a long wait, we released our initial codebase!
Official Implementation of "MineObserver 2.0: A Deep Learning & In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery" by Jay Mahajan , Samuel Hum , Jack Henhapl , Diya Yunus, Matthew Gadbury , Emi Brown, Jeff Ginger, and H. Chad Lane
MineObserver 2.0 is an AI framework that uses Computer Vision and Natural Language Processing for assessing the accuracy of learner-generated descriptions of Minecraft images that include some scientifically relevant content. The system automatically assesses the accuracy of participant observations, written in natural language, made during science learning activities that take place in Minecraft. We demonstrate our system working in real-time and describe a teacher dashboard to showcase observations, both of which advance our previous work. We present the results of a study showing that MineObserver 2.0 improves over its predecessor both in perceived accuracy of the system's generated descriptions as well as in usefulness of the system's feedback. In future work, we intend improve system generated descriptions to give more teacher control and shift the system to perform continuous learning to more rapidly respond to novel observations made by learners.
-
Started with the API folder. Get the API up and running. Test using postman if needed
-
Build the Photographer plugin and deploy this into a Minecraft Java Server.
-
Run the plugin's client on another computer with Minecraft & Python running. This is will control the computer and run the client automatically.
-
(Optional) Run the dashboard to collect your observations.
-
Use any account and call the photographer via the commands!
- Release ML Solution
- Release Dashboard Code
- Release Best Trained Weights
- Release Safe Plugin Code
- Release ML Training Code
- If possible, release dataset