Get familiar with test automation for Flutter apps

Flutter is our key technology used in RebelApps Studio; our flagship Flutter team that is part of Codemate. We are actively involved in core flutter expansion and development with Google.

Here we look at some of the tools available, the different layers and goals of each and share some of our thinking. Hopefully these are valuable to others. Especially for the developers and testers who are embarking on their first steps with the Flutter framework. It also a bit of a kudos shout out to the Flutter community in general, a key element for the continued ongoing success of Flutter as an often preferred framework.

Community Factor

One of the reasons the Flutter framework is thriving is the community of awesome people utilising it and contributing to improve it.

Community provides fantastic learning opportunities often from teams or individuals who are facing similar situations and challenges. So when we wanted to extend our use of test automation tools it was initially the community we looked to.

At some recent community meetups as Rebel App Studio we encountered LeanCode, they have been working on Patrol which extends the abilities of the inbuilt automation tools. LeanCode open sourced Patrol to community. We have it in active use so we will also touch on this in the article. 

Our test guild members have been contacted from the broader testing community looking to get started on Flutter automation. We’ve included a section “Tester Insight” that the community may also find useful. 

Kudos to the community.

Unit testing – Inbuilt Flutter tools

At unit level the goals are fairly common across most projects: develop quality software at optimal pace, rapid almost immediate feedback loop, verification that single functions or classes work in isolation.

Whilst not limited to unit tests there is also value in mocking any dependencies. This keeps the focus on the specific class you are testing and keeps the run speed optimal.

The Flutter documentation on both unit and widget test level is fairly good.

Widget testing

At widget level this shares the same optimal pace and rapid feedback loop goals but also adds in some basic verification that each widget looks and behaves as expected alongside some regression risk coverage goals.

The widgets can be run in isolation so it is not a requirement to build the entire app and you can run these tests without emulators or devices. You can gain insight at UI level whilst maintaining very rapid test runs.

Golden tests are another option here comparing the UI visually to catch regressions. Note that there is an indication that these can take longer to run so consider a balance between speed and value.

One value of adding Patrol finders is its optimised finder feature. It helps to create less and cleaner coder which in turn leads to lower development and maintenance costs. A significant value in our view. 

We were very early adopters of these tools at the above levels and we created these tests as we built the product. It will vary from project to project but generally we tend to have more coverage at the unit and widget levels than the end to end user flow layer.

Continuous Integration

In order to gain the full value of these tests, particularly the rapid feedback aspect, it makes sense for these to be run on every change as part of your CI implementation.

User end to end layer or Integration layer in Flutter terms

In the paired article we raised a couple of challenges at this layer and these factored into our general choice of value goals and the model that impacts the choice tools. We do evaluate and choose models and tools on a case by case basis.

The choice of in-built integration test tools is aligned with some of our values of close team collaboration. Our professional testers and developers often pair together and it allows the test automation code to remain fairly close to the product code base. Therefore we gain higher levels of efficiency.

The following is often a reasonable starting goal for this layer: provide a consistent health test level coverage of user core end to end flows. This offers value in terms of basic feature verification, some regression risk coverage, timely feedback loops, alongside a waste reduction readiness check for deeper investigation into those unknowns our testers love to explore.

Flutter inbuilt integration test layer

The Integration testing layer did meet our very basic needs. It was suitable for the creation of the straight forward health tests of key user flows within the apps. Like in many other frameworks we needed to add a lot of code to identify the widgets and elements we wanted to interact with. We do however regard it currently as still limited from a few aspects such as interaction with webviews outside the app or native device settings that could restrict any expansion beyond these basics.

Patrol – Open source UI testing framework

Patrol has hot restarts which provides significant time saving when developing your tests.

The next valuable area to us was the Native Automation features that Patrol introduced.  It’s a key feature that allows your tests to interact with the native side of things including notifications, webviews, permissions and device settings. By doing so you can broaden the coverage in your tests. Generally we found that it extends the Integration test tool and was fairly straightforward to start using it. It is ongoing development and we did run into a few challenges but overall the value seems good so far. At the time of writing a new version is recently out and there are some key improvements that we are following with interest and look forward to experimenting with. 

CI: End to end tests are slower to run than the other layers. You need to choose when to run these as part of your CI process. This is one of the reasons we often opt for light coverage at this layer. Patrol has sharding features that can help with speeding up your test runs but you may decide to schedule some of the tests to avoid slowing down the development.

Testers insight

This model encourages close collaboration and is a close to product code approach, it’s also a mobile application solution so it’s not as straightforward as pointing to web url and be scripting in an hour or so.

You will be using developer tools, building locally, reading product code, be comfortable doing low risk changes to product code like adding keys to widgets and also doing some actual code within your automation scripts. The level of your coding skills needed will vary but we recommend at least basic skills. Pair with a mobile developer at least initially if possible. In addition to the team collaboration and efficiency levels your understanding and communication levels with developers will also improve.

Automation is great, but when it comes to finding deeper, often more impactful issues and edge cases, hands-on risk based testing is very much needed. The aim should be to maintain a bigger picture holistic view of quality and testing. Testability of Flutter apps can get a bit complex. For example it’s not easy to see user traffic when your app gets delivered via Testflight or Appcenter. Utilising those developer focused tools and doing local builds will also be a valuable addition to your toolbox as you look to discover and investigate risks during your testing sessions. This provides a strong feedback loop into improving your automated coverage.

Wrap up

We hope you have enjoyed reading some of our thoughts on Flutter test automation and perhaps will take a few ideas that can promote active discussion before you jump in. Stay tuned for our next article that is even a deeper dive into Flutter test automation. And follow our Rebel App Studio X account if you haven’t already. We share timely topics and provide updates on Flutter related events we are arranging or participating in.