[en] Farm Testing

Farm Testing

Farm Testing Demo

In order to run a test, navigate to Testing -> Run. Additionally, a certain setup is necessary:

Select at least one device in the Devices menu bar. Make sure all devices are listed under Devices.
Select a testing tool configuration under Tools. See ... for how to add a custom testing tool.
Under Configuration you can specify the number of total tests and the number of retries on failure. You can also pass additional tool parameter in the text area. Note: You should not configure here the output or input directory or device serial number. This highly depends on your overall setup, but these are parameters which should be managed by the Docker setup. The device serial number will be passed as parameter by the farm.
Under APK you can upload a single Android .apk file or a .zip archive containing multiple Android .apk files.
If all the former mentioned conditions are satisfied, you can run the test.
In the Results section a tab for each single test run, specified by the number of tests, will appear. They will update the stdout and stderr of the running container in real-time up to a certain limit of overall characters for each run. When all runs are finished the Download test results button appears. Once a test is executed, you can see the test in the Testing -> Jobs overview.

When navigating to Testing -> Run you can add over the +-Button a custom testing tool with the following properties:

Name: A name of the length 3 to 20 alphanumeric lowercase characters. The alphanumeric lowercase character constraint is necessary, because the name is used to tag the Docker image, which will be created. Therefore, the name has to conform the Docker tagging policy.
Tool git repository: The git URL for your testing tool. You can also specify private repositories by using token authentication and adjusting the URL. See here for how to generate a personal access token for GitLab. A URL with token authentication in GitLab most likely looks like https://oauth2:<TOKEN>@<YOUR-REPOSITORY>.git, but this may vary from GitLab version to version. See here for how to generate a personal access token for GitHub. A URL with token authentication in GitHub looks like https://<USERNAME>:<TOKEN>@github.com/<YOUR-REPOSITORY>.git.
Git commit or branch: The desired commit or branch for the specified testing tool.
Additional parameters: Optional additional parameters you would like to pass to your Docker environment. This may be useful if you want to define certain properties like dependencies.
Docker repository for the execution environment: The URL for the Docker repository specifying the environment and setup in which the testing tool will be executed. This can be a git URL or URL to a Docker Hub repository, everything that conforms to Docker building with a URL.