Maestro CLI unable to connect to XCUI test runner started from Xcode

[bartekpacia]

Context

I was developing #1809, I needed to run my local build of maestro-ios-xctest-runner. I wanted to run the server directly from Xcode (reasons: debugging, easy to see logs) by:

1. export USE_XCODE_TEST_RUNNER=1
2. clicking the "Play" button in Xcode, next to the long-running test:

screenshot 1

However this does not work. It tries to connect to port 7001, which is incorrect on iOS. The correct port on iOS to connect to is 22087.

I added some debug logs (see PR #1815):

TestCommand.handleSession(): driver host port: 7001
MaestroSessionManager.newSession(): host=null, port=null, driverHostPort=7001
MaestroSessionManager.createMaestro(): driverHostPort=7001
MaestroSessionManager.createIOS(): driverHostPort=7001

and discovered this wrong 7001 port comes from here:

maestro/maestro-cli/src/main/java/maestro/cli/command/TestCommand.kt

Lines 215 to 218 in 83f7c11

	val driverHostPort = if (!sharded) parent?.port ?: 7001 else
	(7001..7128).shuffled().find { port ->
	usedPorts.putIfAbsent(port, true) == null
	} ?: error("No available ports found")

screenshot 2

Details

• macOS 14.5 Sonoma
• simulator runtimes: iOS 17.5 and iOS 16.4

Workaround 1

Run my own build of maestro-ios-xctest-runner and let Maestro CLI start it (i.e. do not start the server by clicking the "Play" button in Xcode):

1. Make sure to unset USE_XCODE_TEST_RUNNER
2. ./maestro-ios-xctest-runner/build-maestro-ios-runner.sh
3. ./maestro test <flow>

Workaround 2

Setting the port explicitly to the correct value (22087 - port of the server running in XCTest) fixes the problem. To do it, replace this code:

maestro/maestro-cli/src/main/java/maestro/cli/command/TestCommand.kt

Lines 215 to 218 in 83f7c11

	val driverHostPort = if (!sharded) parent?.port ?: 7001 else
	(7001..7128).shuffled().find { port ->
	usedPorts.putIfAbsent(port, true) == null
	} ?: error("No available ports found")

with:

val driverHostPort = 22087

That code in TestCommand that determines the port was added in #1732.

[joepatosh]

I can confirm the behavior of maestro running on port 7001 instead of 22087. I found this while investigating jpudysz/react-native-is-maestro#16 and also tracked it down to the change in #1732 which adds sharding.

Here is my evidence to confirm this:

First let's confirm that 1.38.1 uses port 7001:

git checkout cli-1.38.1

Add this logging to startXCTestRunner in maestro-ios-driver/src/main/kotlin/xcuitest/installer/LocalXCTestInstaller.kt:

logger.info("[Start] Running XcUITest with `xcodebuild test-without-building` | port = $port")
xcTestProcess = XCRunnerCLIUtils.runXcTestWithoutBuild(
    deviceId,
    xctestRunFile.absolutePath,
    port,
    enableXCTestOutputFileLogging,
)

Run a flow (this is the path to my example flow, but any flow should work for this):

./maestro test ../react-native-is-maestro/example/example.yaml

In the logs, we see:

[INFO ] x.installer.LocalXCTestInstaller - [Start] Running XcUITest with `xcodebuild test-without-building` | port = 7001

We can see that now, maestro is defaulting to port 7001.

Now, check the behavior on: cli-1.36.0

git checkout cli-1.36.0

Change startXCTestRunner in maestro-ios-driver/src/main/kotlin/xcuitest/installer/LocalXCTestInstaller.kt

logger.info("[Start] Running XcUITest with `xcodebuild test-without-building` | port = $port")
xcTestProcess = XCRunnerCLIUtils.runXcTestWithoutBuild(
    deviceId,
    xctestRunFile.absolutePath,
    port
)

Run a flow:

./maestro test ../react-native-is-maestro/example/example.yaml

In the logs, we see:

[INFO ] x.installer.LocalXCTestInstaller - [Start] Running XcUITest with `xcodebuild test-without-building` | port = 22087

This confirms 1.36.0 was still defaulting to running on port 22087.

What about main?

git checkout main
git pull

Add this logging to startXCTestRunner in maestro-ios-driver/src/main/kotlin/xcuitest/installer/LocalXCTestInstaller.kt:

logger.info("[Start] Running XcUITest with `xcodebuild test-without-building` | defaultPort = $defaultPort")
xcTestProcess = XCRunnerCLIUtils.runXcTestWithoutBuild(
    deviceId = deviceId,
    xcTestRunFilePath = xctestRunFile.absolutePath,
    port = defaultPort,
    enableXCTestOutputFileLogging = enableXCTestOutputFileLogging,
)

Note there is a small change here: it is now defaultPort instead of port. But defaultPort is what we want, since that is what is being passed into XCRunnerCLIUtils.runXcTestWithoutBuild.

Run a flow:

./maestro test ../react-native-is-maestro/example/example.yaml

In the logs, we see:

10:23:55.374 [ INFO] xcuitest.installer.LocalXCTestInstaller.startXCTestRunner: [Start] Running XcUITest with `xcodebuild test-without-building` | defaultPort = 7001

Confirming that the latest main still defaults to port 7001 on iOS.

This breaks the logic for "Detecting Maestro in app"

Unfortunately, this behavior directly contradicts the Maestro docs on how to Detect Maestro in your app. Could the maintainers please change the docs to reflect this change in behavior? It would have saved me a significant amount of time. I'm sure that applies to others as well.

As far as I can tell, this change from #1732 makes the logic to detect Maestro in your app much less simple. It seems we need to check if any port between 7001 and 7128 is open on device. Is that the intended change? Some updated guidance here would be greatly appreciated!

Potential workaround

@bartekpacia I'm not sure if this will help solve your immediate problem. It worked for me as a temporary workaround so I figured I'd share in case it could help you or anyone else.

The top level --port argument does override this behavior. You can force maestro on port 22087 by running it like this:

./maestro --port=22087 test ../react-native-is-maestro/example

This only works when using a single shard. If using more than one shard, the --port arugment has no effect and ports in the range (7001..7128) will be used instead.

This is becasue selectPort references parent?.port:

private fun selectPort(effectiveShards: Int): Int =
        if (effectiveShards == 1) parent?.port ?: 7001
        else (7001..7128).shuffled().find { port ->
            usedPorts.putIfAbsent(port, true) == null
        } ?: error("No available ports found")

parent?.port in this case is App.port in maestro-cli/src/main/java/maestro/cli/App.kt - which can be controlled by passing arugments directly to maestro. Note that this --port arugment needs to come before the subcommand.

This works:

./maestro --port=22087 test ../react-native-is-maestro/example

This does not work:

./maestro test --port=22087 ../react-native-is-maestro/example