AugmentedDriver, an Automated Framework for Web, Android and IOS E2E tests
At SalesforceIQ, we deploy to production every day. A year and a half ago, we jumped onto the Automated End To End (E2E) Testing wagon, using WebDriver and Appium. The idea behind this was to have a last line of defense in our process that would give us more confidence in our daily release. It had to be fast and reliable, two features not typically associated with E2E automated tests.
A year and a half later, we have 250+ Automated E2E tests (WebDriver for our Web App and Chrome Extension, Appium for our IOS and Android client), that are running continuously. The whole cycle takes less than 15 minutes!. During October, we ran more than 70000 tests, more than 180000 minutes of E2E testing. Our tests are catching at a rate of 6+ issues per week - that’s more than one per release!.
How we did it?
Following some simple golden rules.
- Test Data Independence. If you follow that principle, each test can run independently from each other, meaning that we can run all 250 in parallel. To achieve data independency, each test creates its own unique user and data (using the same REST API that is accesible to our Android/IOS/Chrome Extension/Web) before running.
- Follow The Page Object Model. The biggest cost of E2E tests is not associated with the development of the tests, but with the maintenance. Once you get to the order of hundreds of tests (and if you don’t structure your tests properly), you will not be able to keep up with the fast pace development velocity of your team. Your tests will become the bottleneck of the production release process.
- Make your tests reliable. Slow but steady wins the race. Once there is one test that is not reliable in the suite, the whole suite becomes unreliable. Write your tests to avoid the common pitfalls, such as trying to find an element that is not present yet, clicking on an element that is not yet clickable, clicking on elements that are still moving due to an animation, etc.
- With the help of our friends in SauceLabs. We heavily rely on them for providing all the infrastructure needed in order to run our tests in parallel.
- Focus on the main flows of the application. Don’t automate all the things, be smart about the tests that you add to raise the coverage bar. Each test added increases the maintenance cost, and you want every test to count.
We just open sourced the framework that is supporting all our tests for Web, Android and IOS. It’s called AugmentedDriver. Basically, it focuses on addressing what was previously described: providing some easy configuration for running tests in parallel, against any Selenium Grid, and in particular against SauceLabs. Also, the framework encourages you to follow the Page Object Model Pattern and provides a thin layer on top of WebDriver/Appium that aims to avoid the common pitfalls that I have encountered during my more than 8 years of experience on automation.
And remember, we’re hiring!