Skip to content

[en] Farm Testing

Jump to bottom Edit New page
Timo Gühring edited this page Jan 30, 2019 · 11 revisions

Farm Testing

Farm Testing Demo

The testing extension provides the possibility to execute comfortably and arbitrary tools for desired Android devices. By leveraging Docker the desired tool can be executed in fully customized environments in isolated containers. Emulators with arbitrary configuration can be spawned on demand. The farm manages the distribution of available devices for the configured test in order to parallelize the execution. The standard output and standard error is displayed to the user in real-time to provide immediate feedback and the possibility to follow the test. After the execution finished succesfully the results can be downloaded. For long running tasks and a better summary there is a test job overview.

Setup and prerequisites

The setup for the STF application itself needs to be taken. Furthermore, there must be a running Docker service (tested with Docker version 17.03.2-ce, build f5ec1e2). Ports must be opened for Docker to fetch repositories. As building and running multiple containers simultaneously is storage and computing expensive, make sure you have enough of both.

Docker needs to run as non-root user. To do that, execute the next commands:

sudo groupadd docker
sudo usermod -aG docker $ USER

Restart the computer so that your group membership is re-evaluated, and be sure you can run docker commands without sudo. You can test this by executing:

docker run hello-world

Running a test job

In order to run a test job, navigate to Testing -> Run. Additionally, a certain setup is necessary:

  • If you would like to use physical devices, select the desired devices in the Devices menu bar. Make sure all physical devices are listed under Devices.
  • If additionally or alternatively you would like to use emulators, see Adding custom emulators.
  • Select a testing tool configuration under Tools. See Adding a custom testing tool for adding one.
  • Under Configuration you can specify the number of total test runs and the number of retries on failure. You can also pass additional tool parameters in the text area. Note: You should not configure here the output or input directory or device serial number. This highly depends on your overall setup, but these are parameters which should be managed by the Docker setup.
  • Optionally, you can upload a single Android .apk file or a .zip archive containing multiple Android .apk files under APK. APKs might be already existing or downloaded in the container, therefore it is optional.
  • If all the former mentioned conditions are satisfied, you can run the test.
  • In the Results section a tab for each single test run, specified by the number of tests, will appear. They will update the stdout and stderr of the running container in real-time up to a certain limit of overall characters for each run. When all runs are finished the Download test results button appears. Once a test job is executed, it is listed in the Testing -> Jobs overview.

Adding a custom testing tool

When navigating to Testing -> Run you can add over the +-Button a custom testing tool with the following properties:

  • Name: A name of the length 3 to 20 alphanumeric lowercase characters. The alphanumeric lowercase character constraint is necessary, because the name is used to tag the Docker image, which will be created. Therefore, the name has to conform the Docker tagging policy.
  • Tool git repository: The git URL for your testing tool. You can also specify private repositories by using token authentication and adjusting the URL. See here for how to generate a personal access token for GitLab. A URL with token authentication in GitLab most likely looks like https://oauth2:<TOKEN>@<YOUR-REPOSITORY>.git, but this may vary from GitLab version to version. See here for how to generate a personal access token for GitHub. A URL with token authentication in GitHub looks like https://<USERNAME>:<TOKEN>@github.com/<YOUR-REPOSITORY>.git.
  • Git commit or branch: The desired commit or branch for the specified testing tool.
  • Setup parameters: Optional setup parameters you would like to pass to your Docker environment. This may be useful if you want to have certain additional dependencies installed in advance.
  • Docker repository for the execution environment: The URL for the Docker repository specifying the environment and setup in which the testing tool will be executed. This can be a git URL or URL to a Docker Hub repository, everything that conforms to Docker building with a URL.

Adding custom emulators

In the Emulators menu you can add custom emulators for the current test job. The following properties are supported:

  • System image: The system image determines the android sdk, see Android Developers. This property is equal to avdmanager's -k "sdk_id". Avoid CPU architectures other than x86, because they are really slow. Make sure your executing machine allows hardware accelaration.
  • SD card size (MB): The size of the SD card image in MB. See Android Developers. This property is equal to avdmanager's -c size.
  • Startup parameters: The startup parameters for the emulator, which will be passed when starting the emulator. See Android Developers for command line parameters.
  • Number of emulators: The number of emulators that will be started with this configuration.

Test job overview

Farm job overview

The test job overview shows all executed test jobs with the corresponding id, start date, time, test configuration name, number of succesful runs / number of total runs and status. If a test job finished succesfully the test results can be downloaded. If a test job failed (e.g. in the Docker building phase) no test results can be downloaded, but the error message can be seen.

Software Architecture

Test job execution

The following depicts the test job execution in detail. It is assumed that the user executed a test like described in Running a test with n devices, number of total runs k and the overall number of retries on failure r.

  1. If a .zip archive was uploaded, it will be unzipped.

  2. Building the Docker image: The Docker image will be built based on the provided Docker repository. The (1) tool git repository, (2) tool git commit or branch, (3) tool name and (4) tool setup parameters will be passed as build arguments:

    docker build \
--build-arg GIT_REPOSITORY=${TESTING_TOOL_GIT_REPOSITORY} \
--build-arg TOOL_COMMIT_DEF=${TESTING_TOOL_GIT_COMMIT} \
--build-arg TOOL_FOLDERNAME=${TESTING_TOOL_NAME} \
--build-arg SETUP_PARAMETERS="[ ${TESTING_TOOL_SETUP_PARAMETERS} ]" \
-q -t "${DOCKER_IMAGE_TAG}" ${TESTING_TOOL_DOCKER_REPOSITORY}

    The intention is to do as much work as possible before creating and starting the container (see next steps) to save execution time, because the image is only built once and the container runs at least k times. Typically, in the building phase the testing tool is cloned and built.

  3. Prepare devices: For every selected physical device the remote connect URL will be requested. See the REST API for more details.

    For every configured emulator the emulator Docker container will be started:

    docker build \
--build-arg SYSTEM_IMAGE="${SYSTEM_IMAGE}" \
--build-arg SD_CARD_SIZE=${SD_CARD_SIZE} \
--build-arg START_UP_PARAMETERS="${STARTUP_PARAMETERS}" -t ${UNIQUE_NAME} ${REPOSITORY}
docker run \
-dt \ # detached mode to return from the run call
--privileged \ # privileged mode for necessary virtualization access, see https://github.com/thyrlian/AndroidSDK/
--network ${DOCKER_NETWORK} \ # set the network for the communication with testing containers
--rm \ # remove the container after it stopped
--name ${UNIQUE_NAME} # name the container so that the testing container can communicate via the name, Docker will resolve the name, because of the passed network

    The Docker container is created for this purpose and takes care of the argument parsing, starting the adb server and starting the emulator.

  4. Creating the Docker container: Based on the previously built Docker image for the test, the Docker container will be created by the following configuration:

    # Pseudo code
createParameters = ""
# The network enables communication between the emulator container and this container
if device is emulator then createParameters =+ "--network ${DOCKER_NETWORK} "
cmdParameters = ""
cmdParameters =+ "--toolParameters=${TOOL_PARAMETERS} "
cmdParameters =+ "--deviceSerialNumber=${device.getSerialNumber()} "
if device is emulator then cmdParameters =+ "--serverAddress=${device.getServerAddress()} "
if device is physical device then cmdParameters =+ "--serverAddress=${device.getRemoteConnectUrl()} "

docker create \
# Mount the adb keys for the remote connection to physical devices
-v ${ADB_KEY_DIR_PATH_HOST}/adbkey.pub:${ANDROID_DIR_CONTAINER}/adbkey.pub \
-v ${ADB_KEY_DIR_PATH_HOST}/adbkey:${ANDROID_DIR_CONTAINER}/adbkey \
--privileged
${createParameters}
${DOCKER_IMAGE_TAG} \
${cmdParameters}

  5. Copying resources into the container: Afterwards all Android .apk files are copied into the container, if any provided:

    docker cp ${APK_FILE} ${DOCKER_CONTAINER_ID}:${DESTINATION_DOCKER_CONTAINER}

    Whereas ${DESTINATION_DOCKER_CONTAINER}='/root/apks'. Files with different file extension as .apk will be ignored.

  6. Starting the Docker container: The container will be started:

    docker start -a ${DOCKER_CONTAINER_ID}

    Produced standard output and standard error from the running container will be communicated in real-time to the frontend, so the user can follow the execution in real-time (there is a maximum of total characters which are allowed to be sent to the frontend). Depending on n the test runs are executed in parallel, obviously only n container can run in parallel. A pending test run will be distributed the next available device. Failing test runs are repeated r times in total.

  7. Copying resources from the container: When the container finishes, the content of the defined output directory will be copied to the host machine:

    docker cp ${DOCKER_CONTAINER_ID}:${OUTPUT_DIR_IN_DOCKER_CONTAINER} ${DESTINATION_HOST}

    Whereas ${OUTPUT_DIR_IN_DOCKER_CONTAINER}='/root/output'.

  8. When all test runs finish, all results will be zipped and archived. The user can follow the execution status in real-time and sees, when the job finishes. The result is available as download.

  9. The Docker images and containers and results will be removed in a defined time interval to prevent consuming too much storage. The default values trigger the process once a day and removes the data older than 9 days.

Step 4, 5, 6 and 7 will be executed between k and k + r times, depending on failed runs.

Testing Setups

DroidMate-2 (A Platform for Android Test Generation)

DroidMate-2 is an automated execution generator for Android apps. Use the linked DroidMate GitHub repository https://github.com/uds-se/droidmate as testing tool URL and https://github.com/uds-se/droidmatedockerenv.git#farmtesting as Docker repository URL for a DroidMate setup.

DroidMate-2 Plug-In

Assuming you developed a Plug-In for DroidMate-2, which uses DroidMate as Gradle dependency, you provide the URL to your repository as testing tool URL and https://github.com/uds-se/droidmatedockerenv.git#farmtesting as Docker repository URL for a DroidMate setup. Additionally, use dependency=https://github.com/uds-se/droidmate as setup parameter. The Docker setup will install every dependency with Gradle.

Custom Setup

TODO

See as an example setup for DroidMate-2. The section Software Architecture gives an understanding of how the Docker container for the execution environment accepts arguments and is supposed to work. The environment should include all necessary setups e.g. Android SDK, Java and environment variables.

The (1) tool git repository, (2) tool git commit or branch, (3) tool name and (4) tool setup parameters will be passed as build arguments. Therefore, it is advised to clone and build the testing tool during the Docker image build phase. An example setup might look like:

# ARGs will be overridden, when build is called with '--build-arg <ARG_NAME>'
ARG GIT_REPOSITORY="https://github.com/uds-se/droidmate.git"
ARG TOOL_COMMIT_DEF="dev"
ARG TOOL_FOLDERNAME="droidmate"
ENV TOOL_FOLDERNAME_ENV=${TOOL_FOLDERNAME}
ENV TOOL_PATH="/root/${TOOL_FOLDERNAME_ENV}"

# Clone
RUN git clone ${GIT_REPOSITORY} ${TOOL_PATH} && \
    cd ${TOOL_PATH} && \
    git checkout ${TOOL_COMMIT_DEF}

# Build
RUN cd ${TOOL_PATH} && \
    chmod +x gradlew && \
    sync && \
    ./gradlew build -x test

# Process setup parameters
ARG SETUP_PARAMETERS="[ ]"
COPY ./processSetupParameters.sh /
RUN chmod +x ./processSetupParameters.sh
RUN ./processSetupParameters.sh ${SETUP_PARAMETERS} # processSetupParameters.sh will install further dependencies

The testing tool must cope with the defined constraints for the APKs directory and the tool output directory. An example setup might look like:

# Prepare resources
ENV TOOL_OUTPUT_FOLDER="/root/output"
RUN mkdir ${TOOL_OUTPUT_FOLDER}
ENV APK_FOLDER_CONTAINER="/root/apks"
RUN mkdir ${APK_FOLDER_CONTAINER}

As it can be seen from the container creation command the device serial number will be passed first then the tool parameters. The selection of an available device is managed by the farm. An example of calling the testing tool with all the necessary parameters might look like:

echo "All passed parameters: $@"
# TOOL_PATH, APK_FOLDER_CONTAINER and TOOL_OUTPUT_FOLDER were previously defined as environment variable in the Docker setup
cd ${TOOL_PATH}
./testtool -device $1 -input ${APK_FOLDER_CONTAINER} -output ${TOOL_OUTPUT_FOLDER} # might want to pass further tool parameters...
Add a custom footer
Add a custom sidebar
Clone this wiki locally