CodeIt!

A standalone Blockly programming application, integrated with ROS. It allows you to use a graphical interface to generate code for a robot, and run it. You implement "primitives" for the robot, which are combined with a subset of JavaScript to form programs. From the interface, you can run programs and stop them mid-program.

CodeIt! is compatible with RWS.

Links

• Tutorial: How to use CodeIt!
• Developer guide: see the rest of the README below

How it works

CodeIt! provides a frontend and a backend. The frontend is a website that lets you create programs with the Blockly programming interface. It also has a simple interface for adding, updating, or deleting programs.

The backend provides a ROS actionlib server that runs JavaScript programs. The programs are run through a sandboxed interpreter. You must define the robot primitives that the interpreter will run. Most primitives do very little work and just call a ROS service. The backend also includes a database of programs that have been created.

Note that CodeIt! itself does not implement any robot functionality. Instead, it calls ROS services to do the actual work on the robot, assuming that some nodes exist which provide these services. It is up to you to implement these services to do things on your robot. We have some pre-made implementations of these services for the PR2, Fetch, and the Turtlebot. However, these are still in the early stages of development and are not documented. If you are actually interested in using these, please contact us by filing an issue and we'll be happy to help.

Getting started

Installing

Make sure you've already gone through 1) the HCR evironment set up that invovles sourcing your bashrc and 2) adding your SSH

• Set up your catkin workspace
• Clone: git clone --recursive [email protected]:hcrlab/code_it.git (Note that this project uses Git submodules hence the recursion) -[ ] Whenever there's a new git pull on the Block submod, run in root blockly folder to make sure you have the most updated. Do this if you see black shapes instead of certain newer blocks. python build.py
• Clone code_it_msgs: git clone [email protected]:hcrlab/code_it_msgs.git
• Install a robot implementation package like code_it_pr2, code_it_fetch, or code_it_turtlebot. These implement ROS services to actually control the robot, which CodeIt! calls while running programs. You can also implement these services yourself for another robot, see code_it_msgs to see services currently defined.
• Build your workspace: catkin build (reccomended) or catkin_make
• Install MongoDB /Install pymongo 3.4: sudo pip install pymongo==3.4
• Add all these other packages into your src folder

              git clone [email protected]:hrclab/blinky.git
              git clone [email protected]:jstnhuang/map_annotator.git
              git clone [email protected]:jstnhuang/rapid.git
              git clone [email protected]:jstnhuang/rapid_pbd.git (threw an error, don’t need?)
              git clone [email protected]:jstnhuang/rapid_pbd_msgs.git
              git clone [email protected]:jstnhuang/surface_perception.git
              git clone [email protected]:hcrlab/location_server.git
              git clone [email protected]:jstnhuang/mongodb_store.git
              git clone [email protected]:jstnhuang/mongo_msg_db.git
              git clone [email protected]:jstnhuang/mongo_msg_db_msgs.git
  

• Install Node.js using NVM. nvm allows you to install and switch between arbitrary versions of node easily.
• Install npm dependencies, including our fork of rosnodejs in the "code_it" folder:

  npm install
  

• Generate messages: node -e "rosnodejs = require('rosnodejs'); rosnodejs.loadAllPackages();"
• Build the frontend. From the frontend folder, run (note that you may have issues with certain versions of gulp, version 3.9.1 is known to work):

  npm install -g bower gulp
  npm install
  bower update
  gulp
  

  • Make sure git branch is pointing to master (* master (in green text)), if not git checkout master, git pull orgin master.

Running

It's handy to run these as separate tabs in your terminal window.

1. roscore
2. Run the robot simulation / robot startup (For Fetch)

  sudo apt-get update
  sudo apt-get install ros-indigo-fetch-gazebo-demo
  roslaunch fetch_gazebo playground.launch 

1. Run the appropriate package for your robot: code_it_pr2, code_it_fetch, code_it_turtlebot (rosrun code_it_fetch main_node )
2. roslaunch rosbridge_server rosbridge_websocket.launch
3. rosrun code_it programs.py - This is the backend that saves and loads the programs to and from MongoDB.
4. rosrun code_it program_server.js - This is the JavaScript interpreter that runs the programs.
5. From the frontend folder, run gulp serve - This is the frontend. Go to localhost:5000 to see the page.

Deploying

Build the frontend

1. If you are not going to host CodeIt! on RWS, then edit frontend/app/elements/code-it-app/code-it-app.html.

2. In the ready function, change:

   ready: function() {
     if (window.location.port === '') {
       this.baseUrl = '/'; // Was /a/code_it/index.html/
     }
   },
   

3. Make sure you are using a relatively new version of node: nvm use node

4. Go to the frontend folder and run gulp.

5. This should generate a www folder in the root of the repository, with the built frontend.

If you are not using RWS, then simply serve the www folder as static content and use roslaunch code_it pr2_app.launch to run the backend. Be sure to double-check the contents of pr2_app.launch or fetch_app.launch, which are designed for the PR2 and Fetch robots running an instance of RWS.

Deploy to RWS

If you are using RWS, then copy the repository (with the built frontend) to the RWS catkin workspace.

• Create a symlink for app.launch: cd launch ln -s fetch_app.launch app.launch
• Create a symlink for code-it-blockly-toolbox.html cd frontend/app/elements/code-it-blockly-toolbox/ ln -s code-it-blockly-toolbox-fetch.html code-it-blockly-toolbox.html CodeIt! should appear in the app list.

How to add / modify primitives

Implement the backend

1. ROS-level implementation

To start with, the primitive should be implemented and accessible through ROS, either as a node listening to a topic, a service, or an action.

2. Add your function to the interpreter

In src/robot.js, add a call to your primitive using rosnodejs to the Robot object, using existing methods as a template.

• Most primitives will call an asynchronous method (e.g., calling a ROS service). In that case, you need to pass a callback argument as the last argument to your function. To "return," call the callback with your return value as the only argument.
• If you would like the program to end due to an irrecoverable error, assign this.error an error message.

Next, register your primitive with the interpreter's sandbox. Edit src/interpreter_api.js using existing methods as a template.

• The line interpreter.setProperty(robotObj, 'myFunction', ...) means that your primitive will be available as the function robot.myFunction(...) in the interpreter.
• To create a global function, change the line to interpreter.setProperty(scope, 'myFunction', ...), which creates the global function myFunction(...).
• If your primitive uses an asynchronous method, then pass a callback as the last argument to the wrapper and to your robot.myFunction call. You will also use interpreter.createAsyncFunction(wrapper).
• If your primitive does not use any asynchronous methods, then do not pass a callback to the wrapper. You will use interpreter.createNativeFunction(wrapper) instead of createAsyncFunction.
• Refer to the JS-Interpreter docs for more details.

Using objects or arrays as arguments or return values

To return an object from a primitive, use interpreter.nativeToPseudo as shown below:

var wrapper = function(callback) {
  function nativeToPseudoCallback(result) {
    callback(interpreter.nativeToPseudo(result));
  }
  robot.findPeople(nativeToPseudoCallback);
}

To use an object passed into a primitive, use interpreter.pseudoToNative as shown below:

var wrapper = function(pseudoPerson, callback) {
  var person = interpreter.pseudoToNative(pseudoPerson);
  robot.recognizePerson(person, callback);
}

Implement the frontend

It's recommended that you read the Custom Blocks section of the Blockly documentation to learn how to make custom blocks in detail.

The general process is to create the code for your block in the Block Factory, which is a meta-programming interface that uses Blockly to generate the code for other Blockly blocks. The video linked above is helpful for understanding how to use the Block Factory.

Once you have the code generated, put the language code in frontend/app/blockly/blocks/robot.js, and the generator stub in frontend/app/blockly/generators/javascript/robot.js. You will need to implement the code generation for your block, such that it calls robot.myFunction() with the appropriate arguments.

Build your blocks using python build.py inside of the blockly folder.

Finally, add the block to the Blockly toolbox, so that users can see the block and drag it into the program. To do that, edit frontend/app/elements/code-it-blockly-toolbox/code-it-blockly-toolbox.html and add your block to the toolbox according to the toolbox documentation. Because toolbox configurations can vary from robot to robot, we have created multiple toolbox files for each robot (the PR2 and Fetch). code-it-blockly-toolbox.html is a symlink that points to one of the robot-specific toolboxes. You can tell Git not to track your toolbox changes using git update-index --assume-unchanged frontend/app/elements/code-it-blockly-toolbox/code-it-blockly-toolbox.html.

About

This project is based on Custom Programs, as described in Design and Evaluation of a Rapid Programming System for Service Robots. If you use this work in your research, we would appreciate you citing it:

@inproceedings{huang2016design,
  title={Design and Evaluation of a Rapid Programming System for Service Robots},
  author={Huang, Justin and Lau, Tessa and Cakmak, Maya},
  booktitle={Proceedings of the 2016 ACM/IEEE international conference on Human-robot interaction (HRI)},
  pages={295--302},
  year={2016},
  organization={ACM}
}