Locator-based vs vision-based mobile app testing

Short summary

Mobile app testing traditionally relied on locator-based approaches (Selenium, Appium, Espresso) that query an app's element tree, but this breaks when the tree is unstable or absent—games, canvas UIs, OS dialogs, and cross-platform flows. LLMs now enable vision-based testing where a model reads screenshots and decides actions, removing the need for element IDs and enabling plain-language test cases writable by non-engineers. The two automated approaches are complementary: use locators where the tree is stable and vision where it isn't, though reproducibility remains an open challenge for LLM-driven tests.

• •Locator-based testing queries element trees but fails on games, canvas UIs, OS dialogs, and requires per-platform locators
• •Vision-based testing uses LLMs to read screenshots, enabling no-code tests that work cross-platform and handle popups automatically
• •Both approaches are complementary—locators for stable trees, vision for everything else—though LLM reproducibility is still unresolved

Generated with AI, which can make mistakes.