ControlKit runners

ControlKit is the native XCTest runner used by the CLI’s Apple-platform MCP tools. The public runner project is xcrs-controlkit. It uses one Xcode project generated by XcodeGen, with platform-specific targets and schemes:

The scheme and target names keep their platform suffixes so Xcode can select the correct SDK, while each built app is named ControlKit.

Build a runner

Install XcodeGen once, clone the runner project, and generate the Xcode project:

brew install xcodegen
cd <path-to-xcrs-controlkit>/Runners
xcodegen generate

Build the simulator runner for the platform you need:

xcodebuild -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'generic/platform=<platform> Simulator' \
  build-for-testing

Replace <platform> with iOS, tvOS, watchOS, or visionOS. macOS is built with -destination 'platform=macOS' and runs on the local Mac rather than a simulator.

Start the runner

Boot a compatible simulator and run the UI-test host:

xcodebuild -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'id=<simulator-udid>' \
  test-without-building

The shared XCTest runner starts a local ControlKit JSON-RPC server on 127.0.0.1:12004. Keep that test process running while calling the MCP tools. The macOS UI-test target includes the network-server entitlement required to bind the local port.

Run on a physical device

The same runner also works installed on a real iOS, tvOS, watchOS, or visionOS device, not just a simulator. Build and start it against the device’s UDID (find it with xcrun devicectl list devices) instead of a simulator destination:

xcodebuild -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'id=<device-udid>' \
  build-for-testing
 
TEST_RUNNER_CONTROLKIT_LISTEN_HOST=0.0.0.0 xcodebuild \
  -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'id=<device-udid>' \
  test-without-building

The CONTROLKIT_LISTEN_HOST environment variable controls what the in-process RPC server binds to (Shared/ControlKitRPCServer.swift) — it defaults to 127.0.0.1, which is loopback-only on the device itself and unreachable from your Mac. xcodebuild forwards any TEST_RUNNER_-prefixed environment variable into the test process with the prefix stripped, so setting TEST_RUNNER_CONTROLKIT_LISTEN_HOST=0.0.0.0 makes the server bind on every interface, including the one reachable over your LAN. Find the device’s address with its Bonjour hostname (<device-name>.local, discoverable via dns-sd -B _airplay._tcp local for an Apple TV) or its CoreDevice tunnel address (xcrun devicectl device info details --device <udid>, under Tunnel IP Address).

Connect through the automation MCP profile

Start either automation server:

xcrs --mcp
# or
smb --mcp --scope automation

The runner tools accept either simulator_name or simulator_udid for a simulator, or host (plus controlkit_port, defaulting to 12004) for a physical device or any other remote runner. Call device_select once to remember the target so later tools need no target arguments. Both commands expose the same unprefixed tool names.

The runner implements these JSON-RPC methods: device.capabilities, device.info, device.apps.launch, device.apps.terminate, device.io.tap, device.io.click, device.io.spatial.tap, device.io.swipe (x1, y1, x2, y2, optional durationMs; iOS), device.io.orientation.get/set (PORTRAIT or LANDSCAPE; iOS), device.io.text, device.io.button, device.dump.ui, device.ui.tap and device.hinge.get (iOS 27.1 and later; the runner lists hinge in its capabilities only on a device that reports one). Calls that name an app that is not running fail with an error instead of stopping the runner.

UI introspection requires a recent ControlKit runner that implements device.dump.ui. The ui_tap action requires a runner that implements device.ui.tap; it resolves and activates the requested element inside the target app, so an MCP client does not need to turn accessibility coordinates into a second action. The label, identifier, or value must match exactly one element. On tvOS, that element must already have focus; use input_button to navigate focus before calling ui_tap. All three UI tools require the target app’s bundle_id; this lets XCTest attach to an already-running app instead of accidentally reading the ControlKit host app.

iPhone Duo (foldable simulators)

The iPhone Duo simulator (Xcode 27.1, iOS 27.1) has two integrated displays, read from the device type’s capabilities.plist:

Only the display in use shows content. The other one is solid black, so a plain capture of display 1 on an unfolded device returns a black frame.

• device_capabilities adds displays and foldable (true when there is more than one integrated display) for simulators.
• screen_capture without display captures every display of a foldable device: a text block (screen 1 outer 1398x2034) then an image for each. Single-display devices return one image as before. display accepts outer, inner, all or a screen ID.
• posture_get calls the runner’s device.hinge.get and reports closed, partiallyOpen or fullyOpen with the angle in radians (0 closed, pi fully open). live_display is the outer display when closed and the inner display otherwise. A device without a hinge reports unsupported.
• posture_set has no simctl or CoreSimulator equivalent. It presses the Closed or Open button in the simulator’s Device Hub window through macOS accessibility (xcrs-host-inspect devicehub-posture), then polls the hinge for up to 5 seconds. It does nothing when the hinge is already in the requested state.

Limits of posture_set:

• Device Hub must be running with the simulator’s window open. The window is matched by the simulator’s name (its title is <name> – <runtime>), so two windows with the same name are an error.
• The app that runs xcrs needs the Accessibility permission (System Settings, Privacy & Security, Accessibility). Without it the call fails with a message saying so.
• The window does not need to be in front.
• Device Hub has a Partially Open button too, but its angle is not adjustable, so posture_set offers only folded and unfolded.
• The helper binary is found as for the other host-inspect commands (XCRS_HOST_INSPECT, next to xcrs, or .build/ in xcrs-controlkit).

Rotating the inner display and inspecting its UI use the same runner methods as any other device. Call posture_get first to know which display is live.

For a local macOS runner, omit simulator fields and pass host and controlkit_port when needed:

{
  "host": "127.0.0.1",
  "controlkit_port": 12004,
  "x": 400,
  "y": 300
}

For a physical device, pass its Bonjour hostname or tunnel address as host instead:

{
  "host": "<device-name>.local",
  "controlkit_port": 12004,
  "button": "select"
}

Platform-specific operations intentionally reject unsupported input. For example, visionOS rejects raw touch, watchOS rejects pointer and spatial input, and macOS rejects touch and Home-button requests.

Inspect the macOS host

The host subcommands describe what is actually on the local Mac’s screen. They are for the cases where a UI test’s own output is not enough: which windows are up, what text a screenshot contains, or which colors run along one line of pixels. Unlike the runner tools above they talk to no device and start no XCTest — they run a small Swift helper, xcrs-host-inspect, in process on the host. They need macOS 13 or later for the AppKit window server and Vision’s text recognizer, are CLI-only, and are not part of the --mcp tool set.

Build the helper

The npm, NuGet, PyPI and cargo releases of xcrs do not include the helper, so build it separately, once, with SwiftPM from the xcrs-controlkit checkout:

cd <path-to-xcrs-controlkit>
swift build -c release

The CLI uses the first xcrs-host-inspect it finds, in this order:

1. the path in the XCRS_HOST_INSPECT environment variable,
2. next to the xcrs binary itself, if you copied the helper there by hand,
3. .build/release, then .build/debug, relative to the current directory,
4. ~/bin, /usr/local/bin, and /opt/homebrew/bin.

Because steps 3 are relative, running xcrs from inside the checkout finds a swift build without any further setup. Set XCRS_HOST_INSPECT when you have several builds and want to choose between them.

An MCP client usually starts xcrs --mcp from a directory that is not the checkout, so the relative .build paths in step 3 will not match. Tools that call the helper, such as posture_set, then need XCRS_HOST_INSPECT in the server’s environment or the helper installed on one of the paths in step 4. When the helper is missing, the error names the variable and every path that was searched.

Read the screen

List the host’s on-screen windows, ordered by layer and then window id:

xcrs host windows
xcrs host windows --owner Mail --json

Each window reports its id, the owning application name (owner), its title, its layer, and its bounds. --owner keeps only windows owned by one application name.

Read the text in an image with Vision and print each string with its confidence:

xcrs host ocr --image shot.png
xcrs host ocr --image shot.png --language en-US --language sv-SE --json

--language is repeatable; omit it to use the platform’s own default.

Crop a rect out of an image and write it to a new PNG:

xcrs host crop --image shot.png --rect 20,40,320,180 --out header.png

--rect takes x,y,width,height. The result reports the written path and its pixel dimensions, and the input image is never modified.

Probe one row or column and describe it as a run of colors, so a solid surface collapses to a single line instead of thousands of pixels:

xcrs host probe --image shot.png --row --position 300
xcrs host probe --image shot.png --column --position 120 --from 40 --to 260

--position is the row or column to read, and exactly one of --row or --column is required. --from and --to clip the line to part of its length. Runs report start, end, width, and a quantized #rrggbb color, so low-order noise does not split one surface into several runs.

Coordinates and scale

Coordinates you pass are in caller units, and --scale says how many pixels one unit covers: the default 1 reads them as pixels, and 2 reads them as Retina points on a 2x display. crop and probe honor it for their rect, position, and bounds. Output geometry follows the same convention — an image reports width/height in caller units alongside pixel_width/ pixel_height — while window bounds are screen points regardless of --scale.

Exit codes and errors

The helper separates a malformed command from failed work: it exits with status 2 when it rejects the arguments, and 1 when the operation itself fails, writing the reason to stderr. The CLI validates rects, the scale, and probe bounds before it spawns the helper, so bad input usually never reaches it. When the helper does fail, the CLI reports it with its status — for example xcrs-host-inspect crop exited with status 1: … — and exits non-zero. A helper that cannot be started at all, such as a missing binary, is reported with a hint to run swift build in xcrs-controlkit.

See also

For the product-level overview of end-to-end testing with xcrs and ControlKit, see smbcloud.xyz/xcrs-end-to-end-testing .