Introducing the Ionic End-to-End Testing Reference Example

Many teams today are struggling to build robust end-to-end tests for their apps, often giving up on testing their native app at all or using a web-only solution that fails to provide an adequate test of the final production native app that will run in users’ hands.

At Ionic, we’re making a big push in 2022 to help teams build and scale robust end-to-end (E2E) tests for their Ionic and Capacitor/Cordova apps, and today we are taking our first step on that journey by releasing a reference Ionic app with end-to-end testing built in using what we consider to be the best stack for building Web Native end-to-end tests.

Let’s dig into the reference example to see Ionic’s opinion on how developers should build Web Native end-to-end tests:

New to end-to-end testing? Think of it as testing the actual app that users will use by writing tests that simulate user interaction. It’s often the top of the Testing Pyramid and is the best way to verify the app that will ship to end users actually works.

Creating a Testing Culture

As we’ve talked to many teams building mission-critical apps on the Ionic stack, we’ve seen a diversity of approaches to testing and a wide range of test cultures. Many teams today have dedicated QA teams writing and performing end-to-end tests. Surprisingly, QA teams are often using languages like Java or Python to write tests against apps that are written in JavaScript or native languages, and don’t have any Ionic-specific utilities to make writing these tests easier.

We’re also starting to see the growth of app dev teams writing their own end-to-end tests, and almost entirely in JavaScript/TypeScript. However, many of these teams are using web-only testing tools such as Cypress or Playwright. While these are great web app testing tools, they don’t provide any way to test the native app that will ship to users.

Finally, many teams aren’t writing end-to-end tests at all. We surveyed a number of app dev teams and 40% of respondents weren’t doing end-to-end testing at all.

Our goal this year is to create a stronger testing culture in the Ionic community by making it much easier to write and manage end-to-end tests on the Ionic/Capacitor stack. We think better, more reliable apps will benefit both Ionic teams and their users, and we think there’s so much more work to be done to help teams make end-to-end testing easier and more accessible.

To that end, we’ve assembled a reference app using an opinionated stack of testing tools to help teams get started writing end-to-end tests.

Ionic E2E Stack

The Ionic E2E stack is built on the shoulders of some serious testing giants. Namely, WebdriverIO, and Appium. These tools are both incredibly powerful and enable developers to build end-to-end tests that run on the web, iOS, Android, and many other platforms. WebdriverIO provides a JavaScript test authoring library and orchestration layer for different test services, and Appium provides native test running for iOS and Android.

Some of you may have experience with one or both of these tools. While powerful, they can be tricky to configure and write tests with, especially for the kinds of Web Native apps that developers using the Ionic stack are building. To help make that easier, this project comes with some pre-baked configuration and some useful built-in helpers and utilities that should make it much easier to get started writing E2E tests for your apps.

Ionic-focused Test Helpers

Our long-term goal is to make native and Web Native end-to-end testing easier by taking these powerful tools and providing a number of convenient helpers focused on Ionic and Capacitor (or Cordova) apps.

To start, we’ve built a few opinions and helpers into the reference app:

Page Object Pattern

All tests in this example follow the Page Object Pattern.

Put simply, this pattern uses two files when testing a page: a Page object that contains properties and methods exposing any functionality a test will need to perform against a Page, and a Spec file that interacts with the Page object to write test assertions against it.

Develop Mode

One of the challenges with building E2E tests is the slow develop/test cycle. One of the goals of Ionic in general is that you should be able to build the majority of your app directly in the browser for the fastest development cycle in the mobile market today, and we think E2E tests should be no different.
Thus, we’ve baked in a convenient development mode that runs tests in Chrome (using chromedriver, so only Chrome supported at the moment) for rapid test development.

To use this feature, first start a development server of your app (such as using ionic serve), and then run:

SERVE_PORT=8100 npm run ionic-e2e:develop

Passing in the port number for the local development server (8100 is the default Ionic Angular port)

Native Builds

Another challenge is building and supplying native binaries to the local WebdriverIO and Appium services. To make this easier, we’ve provided two commands which will build iOS and Android binaries for local testing, and make them available to be detected by the test runner:

npm run ionic-e2e:build:ios
npm run ionic-e2e:build:android

Running tests

To run tests, use the following commands:

npm run ionic-e2e:run:ios
npm run ionic-e2e:run:android
npm run ionic-e2e:run:web

This will use the binaries built above to run on iOS and Android.

Ionic Component Helpers

Finally, one of the things that teams often struggle with when building E2E tests around Ionic apps is that Ionic has a number of performance and user-experience techniques, such as offscreen pages, that can confuse traditional test libraries. To get around this, QA and engineering teams often write brittle, version-specific selectors to find and interact with elements, and that creates flaky tests or a dependency on specific versions of the Ionic UI library which is not ideal.

To make this easier, we’ve provided helpers for a number of core Ionic UI components to make them easier to interact with.

For example, the IonicSelect component makes it easier to interact with ion-select components in tests:

Import { IonicSelect } from ‘../helpers’;

class MyPage {
  get locationSelect() { return new IonicSelect('ion-select') }
}

And then in the test spec we interact with this select component, first opening it, then selecting the second item, and then activating the OK button.

import About from '../pageobjects/about.page';

describe('about', () => {
  it('Should switch location', async () => {
    const location = await About.locationSelect;

    await location.open();
    await location.select(1);
    await location.ok();

To see the full list of helpers, explore the repo.

Library, CLI, and Cloud Testing Coming Soon

The Ionic E2E example app is just the beginning. Our long-term plan is to create an end-to-end testing experience that makes testing your apps across your whole team and organization much easier.

One piece of this is a library and CLI tool that we’re working on which provides a much more streamlined experience using the opinions and concepts in the E2E example. This will also help automatically configure your environment for testing against devices and simulators/emulators, as well as generate new tests based on the structure of your app. More on that later this year.

While we’re working on improving the frontend test experience, we’re working on a fully integrated cloud testing platform powered by Ionic’s Appflow cloud product. As we talked with Ionic developers, we quickly realized it’s too difficult to integrate automated end-to-end tests into a Web Native development process, and existing cloud testing services aren’t able to focus exclusively on the specific technologies that Ionic developers are using.

We’re confident this cloud experience will be a significant evolution in cloud end-to-end testing for Ionic developers. We aren’t quite ready to open up the cloud experience but join the waiting list to be the first to know when it’s live.

How Does This Compare to X?

There are a lot of testing tools in the market, so how is this stack different and why did Ionic choose it?

The biggest difference that necessitated us creating this is that we needed a solution that tested your actual native app and the web app inside of it. This rules out a whole set of great projects like Cypress, Puppeteer, and Playwright because they only test your web app. This might be sufficient for some projects but for proper QA of Web Native apps it is not.

Then, on the native side, you have tools like XCUITest for iOS and UIAutomator2 for Android, but these tools are platform-specific and don’t test your web app, so we can’t simply use those.

This leaves tools like Appium and those built around the WebDriver protocol. These tools have the benefit of testing multiple platforms and having a robust set of plugins and features useful for Web Native apps. There are many projects in the WebDriver world, but we wanted something built fully around JavaScript/TypeScript as that is the primary language our developers use. This left WebdriverIO as the best JS-focused project in the WebDriver space, and Appium as the best native cross-platform testing tool in the WebDriver space. Finally, we wanted to make the experience awesome for Ionic developers because these tools can be challenging to use on their own, so we identified a number of key areas we could put WebdriverIO and Appium together and provide a great experience on top.

Getting Started

The ionic-e2e-example is now live as an alpha reference app that demonstrates as a basic end-to-end testing setup using the stack we are building the rest of our end-to-end testing vision around. It’s very early, and we want to get feedback from the community about how well it’s working for your use cases.

To get started, check out the ionic-e2e-example repo and follow the getting started instructions. Also make sure to join the cloud testing waiting list. We look forward to your feedback!