Skip to content

Running SLOOP object search on AirSIm UE4

Jump to bottom
Kaiyu Zheng edited this page Jul 7, 2021 · 1 revision

Overview

This guides setting up running spatial language object search in AirSim running on an environment in Unreal Engine 4 (UE4). In general this process consists of:

  1. Install UE4 and AirSim following AirSim's documentation. (Linux, Windows)

  2. Set up an UE4 project with AirSim plugin and desired environment

  3. Annotate environment map into the SL_OSM_Dataset/mapinfo system

  4. Configure the controller.py to run object search

Set up an UE4 project with AirSim plugin and desired environment

  1. Download assets for the UE4 environment. For example Neighborhoods and Urban City

  2. Create UE4 project. Place the assets under this project. Sometimes this is done through Epic launcher's "Add to Project" button, sometimes Epic launcher provides a "Download" button and you need to merge it into the project manually. Epic launcher (or Epic Games launcher) is the software that manages UE4 itself, assets and contents and games in this community.

  3. Place AirSim Plugin under the UE4 project. Create a Plugins directory, copy and paste the compiled AirSim plugin (available in the AirSim cloned repo, under Unreal/Plugins). See official documentation for installing and building AirSim. The installation process took me about 3 hours. Note if your project is on Windows, then you should copy the windows-built AirSim to the project. If on Linux, then copy the Linux-built AirSim.

    Many UE4 plugins can be installed this way. For example, the StreetMap plugin.

  4. Now, open the project. The first time is slow because it requires compiling shaders. The copied-AirSim package may need to be rebuilt.

  5. Then, File->"Open Level" to open the Map for your environment. This is typically a .umap file.

  6. Actually, the shader compilation may happen here.

  7. Now you should see the beautiful environment in UE4. You can play around in it.

  8. To test if AirSim works, make sure you check if the Level contains a "Player Start" (under World Outliner). Then, under "World Settings" -> "Game Mode" -> "Game Mode Override", choose "AirSimGameMode". Then hit "Play". Choose "No" for drone.

  9. If works, then great. Note that pressing "1", "2", "3" turns on the depth, semantic, and RGB camera views at the screen bottom. Now it's time to make a connection between this UE4 environment and the POMDP side.

Annotate environment map into the SL_OSM_Dataset/mapinfo system

  1. The first thing to do is to take a screenshot of the Top-down view of the environment. For example topdown-neighborhoods Save this screenshot.

  2. Then, annotate this map. The output of this step are .json files of the same format as other city maps under the SL_OSM_Dataset/mapinfo.

Run

cd spatial_foref/demo/maps`
python map_annotator.py

This will start an interactive landmark annotator. Use option 1 to add landmark. Click on the map to select grid cells for that landmark. When finished, click on the red "X" and enter landmark symbol and name separated by comma (e.g. MainSt, Main Street) on the terminal. Once annotated all, save by choosing option 2 on the command line. Note that this tool is basic. Try to annotate correctly in one shot. If you mess up, you could overwrite earlier annotations by re-annotating for the same landmark symbol and name.

*Save a screenshot of the colored annotations you drew. It's very useful. And, if the map is not too big, you can sketch it with the landmark names. It's handy.

Landmark annotations

landmark annotations

Sketch

sketch

  1. When the above step is done, you should have a folder for the mapinfo for this new city/area. Copy that under the SL_OSM_Dataset/mapinfo directory.

  2. You need to copy the content in {your_map_name}/symbol_to_synonym.json to SL_OSM_Dataset/symbol_to_synonyms.json under a key "landmark_{your_map_name}". This file (SL_OSM_Dataset/symbol_to_synonyms.json) is where the landmark synonyms are loaded. By default each landmark only has its full name under the list of synonyms. You can add additional ones as you please.

Remember to add an entry to the list of map_names in SL_OSM_Dataset/mapinfo/constants.py, otherwise MapInfoDataset won't be able to load your map data.

  1. Now, add lines similar to below to the main program of map_info_datasets.py. Replace "neighborhoods" and the landmark symbols with your new additions.
if __name__ == "__main__":
    ...
    mapinfo.load_by_name("neighborhoods")
    img = mapinfo.visualize("neighborhoods", "HZ1A", bg_path=FILEPATHS["neighborhoods"]["map_png"], color=(205, 105, 100))
    ...
    img = mapinfo.visualize("neighborhoods", "RiverDr", bg_path=FILEPATHS["neighborhoods"]["map_png"], img=img, color=(23, 200, 99))
    mapinfo.visualize("neighborhoods", "MainSt", bg_path=FILEPATHS["neighborhoods"]["map_png"], color=(30, 120, 100), img=img, display=True)
    cv2.destroyAllWindows()
  1. If this works correctly, then great. You have successfully added a new map to our system.

Configure the controller.py to run object search

  1. The painstaking process of writing POMDP controller that syncs multi-object search in pomdp_py with UE4 and AirSim is done. So you should be able to directly use it - just modify a few configurations in demo/controller.py, including

  • map_name

  • "init_point_pair": [(pomdp_x, pomdp_y), (ue4_x, ue4_y)]. This is the initial position of the drone when AirSim starts. This corresponds to the "Player Start"'s (x, y) location. You can read off the UE4 x, y coordinates for "Player Start" in the "Details" window in UE4. You should eyeball, based on your colorful, annotated map screenshot, which pomdp location it should correspond to. Remember that in the POMDP visualization (supposedly) the x axis is horizontal, +x to the right, and y axis is vertical, +y down.

  • "second_point_pair": [(pomdp_x, pomdp_y), (ue4_x, ue4_y)]. This is another reference point. This and the one above are used to determine the transformation from POMDP coordinates to UE4 coordinates, and thus to the NED (AirSim) coordinate frame. You can obtain this by simply adding a "Sphere", read off its UE4 x, y coordinates in the "Details" window and eyeball the corresponding POMDP grid using your screenshot.

  • "flight_height" This determines how high the drone should fly (in meters). The drone (as of now) won't change its height during flight. But this number is needed to determine whether the drone will collide with landmarks which have heights that can be specified using the field below.

  • "landmark_heights" This is the landmark height (e.g. a building). If unspecified, all landmarks will be of height 5 meters; Note that streets are automatically considered to have 0 height.

  1. Now you are reading to try it out. Specify a target location using the target_poses field in run_instance() in the controller.py script. Change the config["language"] to be an appropriate language. Start the AirSim simulation. Then run python controller.py. The drone should first hover at specified height, then a pygame window should pop up with belief overlay and you should see the drone flying to search for the targets.

airsim view

Updating AirSim

If you installed AirSim following their documentation, then you likely cloned its repository. The master branch has the latest updates. To get those updates, do inside the AirSim directory:

git pull origin master

Then, follow these instructions, essentially

./setup.sh
./build.sh

Then the new AirSim plugin is built. Replace the existing plugin with this one in your UE4 project folder.

Clone this wiki locally