Shift Left Automated Testing in native Mobile apps
Why to Shift Left in Testing?
Primary focus of testing is always to find critical bugs at the earliest possible stage in development cycle. While it is ideally expected, the same is difficult to achieve in practical world in big mobile projects.
Shift Left approach helps in finding maximum bugs in the earliest possible stage (i.e., as soon as the developer commits), thus saving tonnes of time & money to business, while increasing confidence of the quality of the app.
While still most part of the world today uses Appium, SeeTest etc, like automation tools where the mindset of “One code-base for all platforms” over rule.
We still need to understand and accept that we automate not to show the world that we are automating, but to find value in the process by quicker releases to market due to reduction of manual testing time in each frequent release.
Today the automated tests run — not just to ensure that the existing tests are still running fine, but also to find bugs at the earliest possible stage within development cycle itself. This is popularly termed as Shift Left, where tests find bugs in each developer commit, saving tonnes of money, time and energy, while ensuring stable releases.
In my experience, an aggressive shift left automation is very effective in Mobile apps, as it saves more time and efforts for the overall team, as otherwise they would have to test it manually across platforms and device models consuming time & energy, leading to inefficiency in case of repetitions. It should also be noted that it is almost close to impossible to test all the scenarios manually, if the release cycles are biweekly or so, in case of mobile apps.
Automation using Shift left testing helps business save testing time as well as bug fixing time, while reaching the market faster every time with the latest feature, beating the competitors.
Third Party Tools
While Shift Left approach is very effective and efficient, choice of right tools are very significant in the project. If the mobile app is a webview contained app where reusability of web code is prominent on mobile app too, in such cases, Appium can be effective.
Limitations of Appium
- Slow test execution
- Flaky tests
- Complex maintenance
- Later bug detection raising cost to business
- Lesser tests are automated due to unwanted complexities
- Each test is to be handled separately for each platform, inspite of having one test-code.
While, Appium is sometimes effective, the same is not the best tool for native apps automation. The reasons are listed below:
Native Automation tools
Native Automation tools are developed by Apple & Google mostly and are most reliable and tests more efficiently in minimum time. These automation serves maximum benefit in projects, when test scripts are integrated within the development project itself, so that the tests can be ran and modified without much hassle.
If the mobile app is a native built app on each platform, probably its best to use native automation tools such as XCTest, XCUITest, Espresso, UIAutomator, or just JUnit in the respective development project itself. The advantage of these tools provide the testing team more freedom and transparency to automate almost everything that is developed more reliably, so that 90–95% of the tests are automated, and running on every developer commit across required devices in parallel.
Thus, more than 90% of manual testing time is saved and testing team can afford to invest their time and energy in conducting exploratory testing which adds more value to business and product during releases, with the identification of valuable bugs in the right stage, helping businesses achieve more confidence and proven stability in the product post release.
It was also observed that these native tools perform best, not only on native apps, but also with hybrid apps containing web-views and react native, when compared to third party tools such as Appium etc.
How integration tests are performed in mobile apps?
Mobile apps may sometimes be integrated to APIs or SDKs to achieve few business scenarios, where it is highly possible that the APIs or the integrated SDK are third parties.
3rd party tools such as Appium, SeeTest, etc face full limitations in SDK automation, as interacting with SDK and writing customised tests to test anything is not possible in black box automation testing. This is where XCTest & Espresso rises high and beats all the other market tools by automating almost 90+% of the scenarios in no time, in most efficient as well as reliable ways. Also, maintaining Shift Left approach with having the tests run in same development project in another test target.
While APIs are always preferred while automating using mobile apps, it is still found that most companies use Web apps automation approach while interacting with Mobile apps on the UI level, instead of using APIs, which is usually time-consuming and more flaky.
Why XCUITest & Espresso for native apps?
XCUITest and Espresso do not have much limitation as they are direct Google and Apple products.
Appium is a layer on top of both XCUITest and Espresso where, tests are unstable & flaky.
Whats special about XCUITest and Espresso?
- Everyone can automate tests — including development team.
- As all tests are part of the Dev codebase, app and test sit together.
- If Unique id are missing or not properly set, we can immediately add it effortlessly saving time & dependency.
- We can reuse unit test shortcuts in UI testing to save time.
- Earliest bug detection due to shift left approach.
- In-built detailed reports, logs, code coverage & screenshots
- Minimal scripting time required — almost no maintenance — zero flaky tests, hence no retries in CI.
What is achieved with Shift Left Native Automation tools?
- More tests are automated at lesser time
- More tests run in parallel devices in lesser execution time
- No more flaky tests or retry mechanisms for randomly failing flaky tests
- More varieties of testing is achieved
