Skip to content

Experiment Configuration

Jump to bottom Edit New page
Radu Apsan edited this page Jul 15, 2024 · 3 revisions

devices.json

A JSON config that maps devices names to their adb ids for easy reference in config files.

config.json

Below is a reference to the fields for the experiment configuration. It is not always updated.

adb_path

adb_path string Path to adb. Normally don't need to include. Example path: /opt/platform-tools/adb

monkeyrunner_path

monkeyrunner_path string Path to Monkeyrunner. Example path: /opt/platform-tools/bin/monkeyrunner

type

type string Type of the experiment. Can be web, native or plugintest

devices_spec

devices_spec string Specify this property inside of your config to specify a devices.json outside of the Android Runner repository. For example:

{
  // ....
  "type": "native",
  "devices_spec": "/home/user/experiments/devices.json",
  "devices": {
    "nexus6p": {}
  },
  // ...
}

repetitions

repetitions positive integer Number of times each application is profiled in an experiment.

clear_cache

clear_cache boolean Clears the cache before every run for both web and native experiments. Default is false.

randomization

randomization boolean Random order of run execution. Default is false.

duration

duration positive integer The duration of each run in milliseconds, default is 0. Setting a too short duration may lead to missing results when running native experiments, it is advised to set a higher duration time if unexpected results appear.

reset_adb_among_runs

reset_adb_among_runs boolean Restarts the adb connection after each run. Default is false.

time_between_run

time_between_run positive integer The time that the framework waits between 2 successive experiment runs. Default is 0. The usb_handler option enables Android Runner to disable the USB connection during each run (while AR is profiling) and enables it after the run. This allows devices to charge inbetween runs. In addition, some (energy) profilers can only provide accurate measurements when there is no charge flowing into the battery during profiling. To use this option the device(s) should be connected to ADB using WiFi. The option expects a JSON object with "enable_command" and "disable_command" as keys and the corresponding commands as values.

"usb_handler" : {
                "enable_command"  : "uhubctl -l 2 -a 1",
                "disable_command" : "uhubctl -l 2 -a 0" 
                }

The example above uses uhubtl, an utilitiy that makes it possible to programmatically enable and disable USB port(s). Please note uhubctl is only compatible with a selection of devices (see the full list on GitHub). We have used and tested uhubctl primarily on a Raspberry PI 4B. There are a few caveats when using uhubctl on the Raspberry PI 4B:

  • You cannot specify which USB port to enable or disable. Either ALL USB ports are enabled or ALL USB ports are disabled.
  • When a USB block/mass storage device is connected, in addition to your device(s) under test, it is likely that the power will turn back on after a few seconds. This is caused by the fact that some device drivers in the kernel are surprised by USB device being turned off and automatically try to power it back on. The easiest way to fix this issue to disconnect the USB block/mass storage. An alternative is to use the udiskctl, see here for more info.

run_stopping_condition

The run_stopping_condition makes it possible to stop the current run when a specific event is triggered. If no event is triggered the run will continue as usual. Right now there are 3 supported events. When the logcat_regex or post_request conditions are used users can also stop the run by using the stop() function call.

  1. A regex in the logcat is matched.

    With the configuration below AR continuously checks if the device's logcat contains an entry matching the "<expr>" where "<expr>" is a regular expression. If this is the case the run will be stopped. Please note that the regex option is required to specify the regex.

    "run_stopping_condition" : {"logcat_regex" : {"regex" : "<expr>"}}

  2. The reception of an HTTP POST request.

    With the configuration below AR will start a local webserver on port 2222 which accepts HTTP POST requests to stop the run. The payload of these HTTP POST requests are saved to the output directory. The file format will be .json if the Content-type header is application/json and .txt otherwise.

    "run_stopping_condition" : {"post_request" : {"server_port" : 2222}}

    The server_port option is optional. If it is not provided the local webserver will be started on port 8000.

    For tips on how to use this option in practice consider checking out this guide.

  3. A direct call of the stop() function on an Experiment instance.

    With the configuration below AR allows one to call the stop() method on an AndroidRunner.Experiment instance to stop the current run.

    "run_stopping_condition" : {"function" : {}}

    This can be useful in an interaction script when we want to stop the run if some condition holds. The current Experiment instance can be accessed by args[0]. An example which stops the run if "certain app" is installed on the device.

    def main(device, *args, **kwargs):
  if device.is_installed("certain app"):
    args[0].stop()

devices

devices JSON A JSON object to describe the devices to be used and their arguments. Below are several examples:

  "devices": {
    "nexus6p": {
      "root_disable_charging": "True",
      "logcat_buffer_size" : 64,
      "charging_disabled_value": 0,
      "usb_charging_disabled_file": "/sys/class/power_supply/usb/device/charge",
      "device_settings_reqs": {"e.www.gyroscopetest": ["location_high_accuracy", ...], ...}
      }
    }
  }
  "devices": {
    "nexus6p": {
      "root_disable_charging": "False"
    }
  }
  "devices": {
    "nexus6p": {}
  }

logcat_buffer_size

The logcat_buffer_size option specifies the size of the logcat buffer in KB (1KB = 1024 Bytes) for the given device. This must be an integer between 64 and 262144 (which is 256MB in KB). If this value is not set a default value of 131072KB (128MB) is used.

Note that the last two examples result in the same behaviour.

root_disable_charging

The root_disable_charging option specifies if the devices needs to be root charging disabled by writing the charging_disabled_value to the usb_charging_disabled_file. Different devices have different values for the charging_disabled_value and usb_charging_disabled_file, so be careful when using this feature. Also keep an eye out on the battery percentage when using this feature. If the battery dies when the charging is root disabled, it becomes impossible to charge the device via USB.

device_settings_reqs

device_settings_reqs can be set to programmatically enable and disable settings on the test device for native apps. It was added to automate the process of turning on and off location services in a randomized experiment where some applications required it and others that didn't. Two options available currently: location services with the help of Google and one without.

location_high_accuracy

location_high_accuracy is the option for location services with Google; location_gps_only is the other. More adb commands are likely to be added in the future that work for other sensors and settings. Turn off location services before the experiment starts.

WARNING: Always check the battery settings of the device for the charging status of the device after using root disable charging. If the device isn't charging after the experiment is finished, reset the charging file yourself via adb su command line using:

adb su -c 'echo <charging enabled value> > <usb_charging_disabled_file>'

paths

paths Array<String> The paths to the APKs/URLs to test with. In case of the APKs, this is the path on the local file system.

apps

apps Array<String> The package names of the apps to test when the apps are already installed on the device. For example:

  "apps": [
    "org.mozilla.firefox",
    "com.quicinc.trepn"
  ]

browsers

browsers Array<String> Dependent on type = web The names of browser(s) to use. Currently supported values are chrome, firefox and opera.

profilers

profilers JSON A JSON object to describe the profiler plugins to be used and their arguments. Check the Plugin Profilers page for more detailed explanations on the profilers. Below, an example is found:

  "profilers": {
    "trepn": {
      "sample_interval": 100,
      "data_points": ["battery_power", "mem_usage"]
    },
    "android": {
      "sample_interval": 100,
      "data_points": ["cpu", "mem"],
      "subject_aggregation": "user_subject_aggregation.py",
      "experiment_aggregation": "user_experiment_aggregation.py"
    }
  }

Did you develop a plugin for Android Runner? You can create a pull request in this repository and we will include it!

The profiler section may accept custom aggregation scripts. If a user specified aggregation script is used then the script should contain a bash main(dummy, data_dir, result_file) method, as this method is used as the entry point to the script. The aggregation options are as follows:

subject_aggregation

subject_aggregation string Specify which subject aggregation to use. The default is the subject aggregation provided by the profiler.

experiment_aggregation

experiment_aggregation string Specify which experiment aggregation to use. The default is the experiment aggregation provided by the profiler.

scripts

scripts JSON A JSON list of types and paths of scripts to run. Below is an example:

"scripts": {
  "before_experiment": "before_experiment.py",
  "before_run": "before_run.py",
  "interaction": "interaction.py",
  "after_run": "after_run.py",
  "after_experiment": "after_experiment.py"
}

Below are the supported types:

  • before_experiment executes once before the first run
  • before_run executes before every run
  • after_launch executes after the target app/website is launched, but before profiling starts
  • interaction executes between the start and end of a run
  • before_close executes before the target app/website is closed
  • after_run executes after a run completes
  • after_experiment executes once after the last run

Instead of a path to string it is also possible to provide a JSON object in the following form:

    "interaction": [
      {
        "type": "python3",
        "path": "Scripts/interaction.py",
        "timeout": 500,
        "logcat_regex": "<expr>"
      }
   ]
  • Within the JSON object you can use "type" to "python3", "monkeyrunner" or, "monkeyreplay" depending on the type of script.
    • "python3" can be used for a standard python script,
    • "monkeyreplay" for running a Monkeyrunner script with the use of the Monkeyrunner framework and
    • "monkeyrunner" can be used to run a Monkeyrunner directly without the entire Monkeyrunner framework.
  • The "timeout" option is to set a maximum run time in miliseconds for the specified script.
  • The optional option "logcat_regex" filters the logcat messages such that it only keeps lines where the log message matches "<expr>" where "<expr>" is a regular expression.

Resuming Interrupted Experiments

In case of an error or a user abort during experiment execution, it is possible to continue the experiment if desired. This is possible by using a --progress tag with the starting command. For example:

python3 android_runner your_config.json --progress path/to/progress.xml

Add a custom footer
Add a custom sidebar
Clone this wiki locally