How Shopify pushed mobile E2E test stability to 98%
Shopify's largest mobile app had to pull E2E tests out of blocking CI because flakiness was rejecting more good PRs than bad. The root cause, they found, was the framework, not the tests: Appium's low-level flexibility let tests tap elements before the screen rendered, and shortcuts like pause(1000) accumulated. Instead of patching Appium, they built an opinionated wrapper. The API is builder-style: every step (tap, wait, type) must declare the expected resulting state, escape hatches are prefixed UNSAFE_ to discourage misuse, and the small grammar is predictable enough for AI agents to write correct tests without codebase knowledge. Element lookup is done via computer vision—PaddleOCR for text and OpenCV for matching Polaris SVG icons in grayscale across sizes; Test IDs become an opt-in fallback. Each run produces an annotated video, so most failures self-diagnose in seconds. Test stability went from 50% to 98% a few weeks after promoting the new framework to blocking CI. The post closes with transferable principles: limit the API to core commands, require assertions with validated before/after states, and enforce a multi-run stability gate before admitting tests to the suite. Useful for mobile engineers and test-infra teams fighting flaky UIs.