Update uiauto library, extend framework to allow jank-testing and add new JetNews jank-testing workload #1268
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pport and into androidx Currently the uiauto library is built on top of the deprecated android.support libraries. Create an updated uiauto library (uiauto-androidx) using androidx and keep it alongside the old one for compatibility reasons. As we update workload-automation going forward, we should update the workloads to use the androidx version of uiauto. This has been tested with the latest gradle version available at the time (8.8).
Jank testing involves using a different library (JankTestBase) that our testing class needs to extend. Since we already make the base testing class UiAutomation extend BaseUiAutomation, we cannot use the UiAutomation class for our tests. The uiauto_class property is hardcoded to "UiAutomation" though, so this patch adds a uiauto_jank_class property for the Jank test class. While at it, we also make the uiauto_runner configurable (currently hardcoded to the old deprecated android.support version of it) so that our workloads can pass whatever runner they need to use. In terms of stages for testing, their current names are the following: stages = ['setup', 'runWorkload', 'extractResults', 'teardown'] Those won't work for jank tests. For some reason the jank test library wants to see the testing function name contain or start with a "test" string pattern on it. This behavior doesn't seem to be documented, but at least with the current version of the JankTestBase library we have this restriction. To make things work despite this restriction, I've added separate jank test stages that can be provided by the workload. The generic workload code will use these stages to generate the instrumentation invocation command, and the provided jank test stages (essentially jank test functions) will be invoked one by one. Finally, we make the UiAutomatorGUI's _execute method return the output of the execution so the workload can freely process and pattern-match the results, like the UiBenchJankTests workload.
This patch adds the JetNews jank-testing workload. This is accomplished
through the uiauto helper library and the jank test classes.
This workload requires a JetNews app APK to be available. We plan to
make it available through the workload-automation-assets repo.
At the end of the run, users should end up with a json file containing all
the frame/jank metrics.
There are 3 parameters for this workload:
- tests: Specifies which of the 3 available tests to run (default is to run
all of them)
- flingspeed: The speed of the fling interactions.
- repeat: How many times each of the selected tests is to be executed in a
single measuring session.
|
@marcbonnici this PR replaces the previous patch to add a JetNews workload. This approach leverages the existing uiauto library and seems cleaner. Could you please take a look at it and let me know what you think? |
marcbonnici
reviewed
Jul 24, 2024
wa/framework/uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/ActionLogger.java
Outdated
Show resolved
Hide resolved
wa/workloads/jetnews/uiauto/app/src/main/java/com/arm/wa/uiauto/jetnews/UiAutomation.java
Outdated
Show resolved
Hide resolved
We don't want to change the old uiauto library at the moment. We will update uiauto along with each workload using it.
- Add JetNews workload description. - Fixup copyright years. - Adjust formatting. - Add constraints to the workload's parameters. - Fix comment typos. - Remove unused commented-out code from app/build.gradle.
|
Updated the code based on the comments, pending the two comments above. |
Update timestamp to match date the file was created.
Update the copyright years in the headers of files to match the year they were created.
Restore UiAutomatorGUI back to what it was, as we don't want to change the existing generic code for implementing the jank test workloads.
Create two new classes to deal with workloads for which we want to collect frame metrics/jank percentages. ApkUiautoJankTestWorkload inherits from ApkUiautoWorkload and instantiates a UiAutomatorJankTestGUI gui member. UiAutomatorJankTestGUI (inherits from UiAutomatorGUI) handles most of the new logic for jank tests, including using a regular expression to parse the final output.
… in a simpler way Modify the JetNews workload to use the new jank testing classes. This allows simplification of the code and also makes it more straightforward to implement new workloads that also need to gather frame metrics.
|
These new 5 commits achieve the following: |
|
I'm not sure if the R0901 issue caught by linters/tests is of any concern. The workload classes seem to be split into a number of different layers already. |
|
Ping @marcbonnici. |
I agree this looks reasonable to me. Would it be possible to add a |
marcbonnici
reviewed
Sep 12, 2024
| // Scroll to the first postcard in the list. | ||
| mDevice.wait(Until.findObject(By.res(MAIN_VIEW)), DEFAULT_TIMEOUT); | ||
| UiObject2 articles = mDevice.findObject(By.res(MAIN_VIEW)); | ||
| ViewMatchers.assertThat(articles, CoreMatchers.notNullValue()); |
There was a problem hiding this comment.
Attempting to run this test on my emulator with the latest jetnews app results in the following crash.
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: class=com.arm.wa.uiauto.jetnews.UiAutomationJankTests
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: current=1
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: numtests=1
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: stream=
2024-09-12 17:29:36,628 ERROR signal: com.arm.wa.uiauto.jetnews.UiAutomationJankTests:
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: test=test1
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS_CODE: 1
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: class=com.arm.wa.uiauto.jetnews.UiAutomationJankTests
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: current=1
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: numtests=1
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: stack=java.lang.reflect.InvocationTargetException
2024-09-12 17:29:36,628 ERROR signal: at java.lang.reflect.Method.invoke(Native Method)
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.jank.JankTestBase.runTest(JankTestBase.java:174)
2024-09-12 17:29:36,628 ERROR signal: ... 26 trimmed
2024-09-12 17:29:36,628 ERROR signal: Caused by: junit.framework.AssertionFailedError:
2024-09-12 17:29:36,628 ERROR signal: Expected: not null
2024-09-12 17:29:36,628 ERROR signal: Got: was null
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:16)
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:1)
2024-09-12 17:29:36,628 ERROR signal: at com.arm.wa.uiauto.jetnews.UiAutomationJankTests.runPortraitVerticalTests(UiAutomationJankTests.java:116)
2024-09-12 17:29:36,628 ERROR signal: at com.arm.wa.uiauto.jetnews.UiAutomationJankTests.test1(UiAutomationJankTests.java:88)
2024-09-12 17:29:36,628 ERROR signal: ... 29 more
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: stream=
2024-09-12 17:29:36,628 ERROR signal: Error in test1(com.arm.wa.uiauto.jetnews.UiAutomationJankTests):
2024-09-12 17:29:36,628 ERROR signal: java.lang.reflect.InvocationTargetException
2024-09-12 17:29:36,628 ERROR signal: at java.lang.reflect.Method.invoke(Native Method)
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.jank.JankTestBase.runTest(JankTestBase.java:174)
2024-09-12 17:29:36,628 ERROR signal: ... 26 trimmed
2024-09-12 17:29:36,628 ERROR signal: Caused by: junit.framework.AssertionFailedError:
2024-09-12 17:29:36,628 ERROR signal: Expected: not null
2024-09-12 17:29:36,628 ERROR signal: Got: was null
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:16)
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:1)
2024-09-12 17:29:36,628 ERROR signal: at com.arm.wa.uiauto.jetnews.UiAutomationJankTests.runPortraitVerticalTests(UiAutomationJankTests.java:116)
2024-09-12 17:29:36,628 ERROR signal: at com.arm.wa.uiauto.jetnews.UiAutomationJankTests.test1(UiAutomationJankTests.java:88)
2024-09-12 17:29:36,628 ERROR signal: ... 29 more
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS: test=test1
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_STATUS_CODE: -2
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_RESULT: stream=
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: Time: 1.761
2024-09-12 17:29:36,628 ERROR signal: There was 1 failure:
2024-09-12 17:29:36,628 ERROR signal: 1) test1(com.arm.wa.uiauto.jetnews.UiAutomationJankTests)
2024-09-12 17:29:36,628 ERROR signal: java.lang.reflect.InvocationTargetException
2024-09-12 17:29:36,628 ERROR signal: at java.lang.reflect.Method.invoke(Native Method)
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.jank.JankTestBase.runTest(JankTestBase.java:174)
2024-09-12 17:29:36,628 ERROR signal: ... 26 trimmed
2024-09-12 17:29:36,628 ERROR signal: Caused by: junit.framework.AssertionFailedError:
2024-09-12 17:29:36,628 ERROR signal: Expected: not null
2024-09-12 17:29:36,628 ERROR signal: Got: was null
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:16)
2024-09-12 17:29:36,628 ERROR signal: at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:1)
2024-09-12 17:29:36,628 ERROR signal: at com.arm.wa.uiauto.jetnews.UiAutomationJankTests.runPortraitVerticalTests(UiAutomationJankTests.java:116)
2024-09-12 17:29:36,628 ERROR signal: at com.arm.wa.uiauto.jetnews.UiAutomationJankTests.test1(UiAutomationJankTests.java:88)
2024-09-12 17:29:36,628 ERROR signal: ... 29 more
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: FAILURES!!!
2024-09-12 17:29:36,628 ERROR signal: Tests run: 1, Failures: 1
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal:
2024-09-12 17:29:36,628 ERROR signal: INSTRUMENTATION_CODE: -1
Do you see the same behaviour?
There was a problem hiding this comment.
Did you use the stock jetnews app? If so, it doesn't have the required code, so I'd expect it to fail. You need a modified version of JetNews with test tags and additional articles.
I have shared a page that contains both the apk and the repository that can be used to build it from sources.
| @@ -0,0 +1 @@ | |||
| android.useAndroidX=true | |||
There was a problem hiding this comment.
Apologies for the late comment on this, but IIUC it seems that this is currently the main meaningful difference between the uiauto and uiauto-androidx directory trees?
I.e. If I do a diff on these these directories, this is my output:
Common subdirectories: uiauto/app and uiauto-androidx/app
diff --color uiauto/build.gradle uiauto-androidx/build.gradle
5d4
< jcenter()
6a6
> mavenCentral()
9c9
< classpath 'com.android.tools.build:gradle:7.2.1'
---
> classpath 'com.android.tools.build:gradle:8.5.0'
11d10
<
13c12
< // in the individual module build.gradle files
---
> // in the individual module build.gradle files
19d17
< jcenter()
20a19
> mavenCentral()
diff --color uiauto/build.sh uiauto-androidx/build.sh
2c2
< # Copyright 2013-2017 ARM Limited
---
> # Copyright 2013-2024 ARM Limited
Common subdirectories: uiauto/gradle and uiauto-androidx/gradle
Only in uiauto-androidx: gradle.properties
Binary files uiauto/uiauto.aar and uiauto-androidx/uiauto.aar differ
Therefore I'm wondering if we really need the two trees here currently as I'm concerned about the duplication while we transition over to the new libraries.
Do you think it would be possible to instead just have 2 gradle.properties files and 2 build scripts in the same tree?
One that sets this property to True and one that sets it to False? That way we could maintain the same source tree but would still be able to build both versions of the library?
Plmk if I'm missing something that would make this a non-starter.
There was a problem hiding this comment.
On my end, the differences between wa/framework/uiauto and wa/framework/uiauto-androidx are a bit more significant:
--- uiauto/app/build.gradle 2024-07-26 15:42:00.889125106 +0100
+++ uiauto-androidx/app/build.gradle 2024-07-26 15:40:54.948614211 +0100
@@ -1,18 +1,18 @@
apply plugin: 'com.android.library'
android {
+ namespace "com.arm.wa.uiauto"
compileSdkVersion 28
- buildToolsVersion '28.0.3'
defaultConfig {
minSdkVersion 18
targetSdkVersion 28
- testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
+ testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}
}
dependencies {
implementation fileTree(include: ['*.jar'], dir: 'libs')
- implementation 'com.android.support.test:runner:0.5'
- implementation 'com.android.support.test:rules:0.5'
- implementation 'com.android.support.test.uiautomator:uiautomator-v18:2.1.2'
+ implementation 'androidx.test:runner:1.6.1'
+ implementation 'androidx.test:rules:1.6.1'
+ implementation 'androidx.test.uiautomator:uiautomator-v18:2.2.0-alpha1'
}
diff -up -r uiauto/app/src/main/AndroidManifest.xml uiauto-androidx/app/src/main/AndroidManifest.xml
--- uiauto/app/src/main/AndroidManifest.xml 2024-07-26 15:40:54.948614211 +0100
+++ uiauto-androidx/app/src/main/AndroidManifest.xml 2024-07-26 15:40:54.948614211 +0100
@@ -1,5 +1,4 @@
-<manifest xmlns:android="http://schemas.android.com/apk/res/android"
- package="com.arm.wa.uiauto">
+<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.READ_LOGS"/>
diff -up -r uiauto/app/src/main/java/com/arm/wa/uiauto/ActionLogger.java uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/ActionLogger.java
--- uiauto/app/src/main/java/com/arm/wa/uiauto/ActionLogger.java 2024-07-26 15:40:54.948614211 +0100
+++ uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/ActionLogger.java 2024-08-29 11:09:59.184231510 +0100
@@ -1,4 +1,4 @@
-/* Copyright 2014-2018 ARM Limited
+/* Copyright 2014-2024 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
diff -up -r uiauto/app/src/main/java/com/arm/wa/uiauto/ApplaunchInterface.java uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/ApplaunchInterface.java
--- uiauto/app/src/main/java/com/arm/wa/uiauto/ApplaunchInterface.java 2024-07-26 15:40:54.948614211 +0100
+++ uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/ApplaunchInterface.java 2024-08-29 11:09:59.184231510 +0100
@@ -1,4 +1,4 @@
-/* Copyright 2013-2017 ARM Limited
+/* Copyright 2013-2024 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -16,7 +16,7 @@
package com.arm.wa.uiauto;
import android.os.Bundle;
-import android.support.test.uiautomator.UiObject;
+import androidx.test.uiautomator.UiObject;
/**
* ApplaunchInterface.java
diff -up -r uiauto/app/src/main/java/com/arm/wa/uiauto/BaseUiAutomation.java uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/BaseUiAutomation.java
--- uiauto/app/src/main/java/com/arm/wa/uiauto/BaseUiAutomation.java 2024-07-26 15:40:54.952614242 +0100
+++ uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/BaseUiAutomation.java 2024-08-29 11:09:59.188231543 +0100
@@ -1,4 +1,4 @@
-/* Copyright 2013-2018 ARM Limited
+/* Copyright 2013-2024 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -22,13 +22,13 @@ import android.content.Context;
import android.graphics.Point;
import android.graphics.Rect;
-import android.support.test.InstrumentationRegistry;
-import android.support.test.uiautomator.UiDevice;
-import android.support.test.uiautomator.UiObject;
-import android.support.test.uiautomator.UiObjectNotFoundException;
-import android.support.test.uiautomator.UiSelector;
-import android.support.test.uiautomator.UiWatcher;
-import android.support.test.uiautomator.UiScrollable;
+import androidx.test.InstrumentationRegistry;
+import androidx.test.uiautomator.UiDevice;
+import androidx.test.uiautomator.UiObject;
+import androidx.test.uiautomator.UiObjectNotFoundException;
+import androidx.test.uiautomator.UiSelector;
+import androidx.test.uiautomator.UiWatcher;
+import androidx.test.uiautomator.UiScrollable;
import org.junit.Before;
import org.junit.Test;
@@ -39,7 +39,7 @@ import java.io.InputStreamReader;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import static android.support.test.InstrumentationRegistry.getArguments;
+import static androidx.test.InstrumentationRegistry.getArguments;
public class BaseUiAutomation {
diff -up -r uiauto/app/src/main/java/com/arm/wa/uiauto/UiAutoUtils.java uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/UiAutoUtils.java
--- uiauto/app/src/main/java/com/arm/wa/uiauto/UiAutoUtils.java 2024-07-26 15:40:54.952614242 +0100
+++ uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/UiAutoUtils.java 2024-08-29 11:09:59.188231543 +0100
@@ -1,4 +1,4 @@
-/* Copyright 2013-2017 ARM Limited
+/* Copyright 2013-2024 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
diff -up -r uiauto/app/src/main/java/com/arm/wa/uiauto/UxPerfUiAutomation.java uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/UxPerfUiAutomation.java
--- uiauto/app/src/main/java/com/arm/wa/uiauto/UxPerfUiAutomation.java 2024-07-26 15:40:54.952614242 +0100
+++ uiauto-androidx/app/src/main/java/com/arm/wa/uiauto/UxPerfUiAutomation.java 2024-08-29 11:09:59.188231543 +0100
@@ -1,4 +1,4 @@
-/* Copyright 2013-2017 ARM Limited
+/* Copyright 2013-2024 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
diff -up -r uiauto/build.gradle uiauto-androidx/build.gradle
--- uiauto/build.gradle 2024-07-26 15:41:53.129065105 +0100
+++ uiauto-androidx/build.gradle 2024-07-26 15:40:54.948614211 +0100
@@ -2,22 +2,21 @@
buildscript {
repositories {
- jcenter()
google()
+ mavenCentral()
}
dependencies {
- classpath 'com.android.tools.build:gradle:7.2.1'
-
+ classpath 'com.android.tools.build:gradle:8.5.0'
// NOTE: Do not place your application dependencies here; they belong
- // in the individual module build.gradle files
+ // in the individual module build.gradle files
}
}
allprojects {
repositories {
- jcenter()
google()
+ mavenCentral()
}
}
diff -up -r uiauto/build.sh uiauto-androidx/build.sh
--- uiauto/build.sh 2024-07-26 15:40:54.952614242 +0100
+++ uiauto-androidx/build.sh 2024-08-29 11:09:59.188231543 +0100
@@ -1,5 +1,5 @@
#!/bin/bash
-# Copyright 2013-2017 ARM Limited
+# Copyright 2013-2024 ARM Limited
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff -up -r uiauto/gradle/wrapper/gradle-wrapper.properties uiauto-androidx/gradle/wrapper/gradle-wrapper.properties
--- uiauto/gradle/wrapper/gradle-wrapper.properties 2024-07-26 15:40:54.952614242 +0100
+++ uiauto-androidx/gradle/wrapper/gradle-wrapper.properties 2024-08-29 11:09:59.188231543 +0100
@@ -1,6 +1,6 @@
-#Wed May 03 15:42:44 BST 2017
+#Tue 27 Aug 2024 12:28:27 PM BST
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-7.3.3-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.8-all.zip
Only in uiauto-androidx: gradle.properties```
For the androidx version there were version bumps for the libraries, removal of deprecated properties (buildToolsVersion IIRC), updates to the JUNIT runner (to use the androidx one), the gradle version, the repositories to fetch libraries from and changes to the uiauto app itself to use androidx libraries.
The way I see it, we want to update all the other apps to use the updated uiauto framework using androidx, because the older android support library is now deprecated. Unfortunately I was never able to chase the right versions of the apk's for some of the workloads. So it is a bit tricky to test and make sure they work. In the end, I considered keeping the original uiauto framework as-is for now, in the hopes it would still work for whoever still has those valid apk's.
I'm not sure I have the cycles to look into this at the moment, but we could probably coordinate internally to get this updated so we don't have to use deprecated libraries.Drop the warning as we already have cases where it was disabled in geekbench and gfxbench.
Done now. |
|
I think I managed to get all the checks happy now. :-) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.