Author: Andrew Kelly

This is the second article of our three-article series on test automation for everyone interested in the quality of digital projects. The first article helps you identify how to achieve your goals through test automation. This second article provides an introduction to Flutter test automation, catering Flutter testers and developers. The third and final article in the series delves deeper into Flutter test automation, specifically tailored for Flutter developers.

Flutter is our key technology used in RebelApps Studio; our flagship Flutter team that is part of Codemate. We are actively involved in core flutter expansion and development with Google.

Here we look at some of the tools available, the different layers and goals of each and share some of our thinking. Hopefully these are valuable to others. Especially for the developers and testers who are embarking on their first steps with the Flutter framework. It also a bit of a kudos shout out to the Flutter community in general, a key element for the continued ongoing success of Flutter as an often preferred framework.

Community Factor.

One of the reasons the Flutter framework is thriving is the community of awesome people utilising it and contributing to improve it.

Community provides fantastic learning opportunities often from teams or individuals who are facing similar situations and challenges. So when we wanted to extend our use of test automation tools it was initially the community we looked to.

At some recent community meetups as Rebel App Studio we encountered LeanCode, they have been working on Patrol which extends the abilities of the inbuilt automation tools. LeanCode open sourced Patrol to community. We have it in active use so we will also touch on this in the article. 

Our test guild members have been contacted from the broader testing community looking to get started on Flutter automation. We’ve included a section “Tester Insight” that the community may also find useful. 

Kudos to the community.

Unit testing – Inbuilt Flutter tools

At unit level the goals are fairly common across most projects: develop quality software at optimal pace, rapid almost immediate feedback loop, verification that single functions or classes work in isolation.

Whilst not limited to unit tests there is also value in mocking any dependencies. This keeps the focus on the specific class you are testing and keeps the run speed optimal.

The Flutter documentation on both unit and widget test level is fairly good.

Widget testing.

At widget level this shares the same optimal pace and rapid feedback loop goals but also adds in some basic verification that each widget looks and behaves as expected alongside some regression risk coverage goals.

The widgets can be run in isolation so it is not a requirement to build the entire app and you can run these tests without emulators or devices. You can gain insight at UI level whilst maintaining very rapid test runs.

Golden tests are another option here comparing the UI visually to catch regressions. Note that there is an indication that these can take longer to run so consider a balance between speed and value.

One value of adding Patrol finders is its optimised finder feature. It helps to create less and cleaner coder which in turn leads to lower development and maintenance costs. A significant value in our view. 

We were very early adopters of these tools at the above levels and we created these tests as we built the product. It will vary from project to project but generally we tend to have more coverage at the unit and widget levels than the end to end user flow layer.

Continuous Integration

In order to gain the full value of these tests, particularly the rapid feedback aspect, it makes sense for these to be run on every change as part of your CI implementation.

User end to end layer or Integration layer in Flutter terms.

In the paired article we raised a couple of challenges at this layer and these factored into our general choice of value goals and the model that impacts the choice tools. We do evaluate and choose models and tools on a case by case basis.

The choice of in-built integration test tools is aligned with some of our values of close team collaboration. Our professional testers and developers often pair together and it allows the test automation code to remain fairly close to the product code base. Therefore we gain higher levels of efficiency.

The following is often a reasonable starting goal for this layer: provide a consistent health test level coverage of user core end to end flows. This offers value in terms of basic feature verification, some regression risk coverage, timely feedback loops, alongside a waste reduction readiness check for deeper investigation into those unknowns our testers love to explore.

Flutter inbuilt integration test layer

The Integration testing layer did meet our very basic needs. It was suitable for the creation of the straight forward health tests of key user flows within the apps. Like in many other frameworks we needed to add a lot of code to identify the widgets and elements we wanted to interact with. We do however regard it currently as still limited from a few aspects such as interaction with webviews outside the app or native device settings that could restrict any expansion beyond these basics.

Patrol – Open source UI testing framework

Patrol has hot restarts which provides significant time saving when developing your tests.

The next valuable area to us was the Native Automation features that Patrol introduced.  It’s a key feature that allows your tests to interact with the native side of things including notifications, webviews, permissions and device settings. By doing so you can broaden the coverage in your tests. Generally we found that it extends the Integration test tool and was fairly straightforward to start using it. It is ongoing development and we did run into a few challenges but overall the value seems good so far. At the time of writing a new version is recently out and there are some key improvements that we are following with interest and look forward to experimenting with. 

CI: End to end tests are slower to run than the other layers. You need to choose when to run these as part of your CI process. This is one of the reasons we often opt for light coverage at this layer. Patrol has sharding features that can help with speeding up your test runs but you may decide to schedule some of the tests to avoid slowing down the development.

Testers insight

This model encourages close collaboration and is a close to product code approach, it’s also a mobile application solution so it’s not as straightforward as pointing to web url and be scripting in an hour or so.

You will be using developer tools, building locally, reading product code, be comfortable doing low risk changes to product code like adding keys to widgets and also doing some actual code within your automation scripts. The level of your coding skills needed will vary but we recommend at least basic skills. Pair with a mobile developer at least initially if possible. In addition to the team collaboration and efficiency levels your understanding and communication levels with developers will also improve.

Automation is great, but when it comes to finding deeper, often more impactful issues and edge cases, hands-on risk based testing is very much needed. The aim should be to maintain a bigger picture holistic view of quality and testing. Testability of Flutter apps can get a bit complex. For example it’s not easy to see user traffic when your app gets delivered via Testflight or Appcenter. Utilising those developer focused tools and doing local builds will also be a valuable addition to your toolbox as you look to discover and investigate risks during your testing sessions. This provides a strong feedback loop into improving your automated coverage.

Wrap up

We hope you have enjoyed reading some of our thoughts on Flutter test automation and perhaps will take a few ideas that can promote active discussion before you jump in. Stay tuned for our next article that is even a deeper dive into Flutter test automation. And follow our Rebel App Studio X account if you haven’t already. We share timely topics and provide updates on Flutter related events we are arranging or participating in.

We have prepared a three-article series on test automation for everyone interested in the quality of digital projects. The first article helps you identify how to achieve your goals through test automation. The second article will provide an introduction to Flutter automation testing, catering to Flutter testers and developers. The third and final article in the series delves deeper into Flutter test automation, specifically tailored for Flutter developers.

Here we will look at the early consideration of automation tools, and how these tools can be leveraged to enhance the achievement of your goals, advance your missions, and effectively tackle the challenges you seek to resolve.

It’s high level content aimed at the whole team, whilst it won’t provide you with specific solutions it will hopefully give your team some thoughts and ideas to discuss to empower good decisions with regards to automation.

Don’t jump in, just yet.

Automation can at times get labeled with almost magical expectations that it is “the” go to solution for quality problems.

The reality is generally very different. Up front quality evaluations can often highlight both automation opportunities and flag other opportunities to improve things that may be better suited to the problem you are trying to solve. When it comes to product automation we suggest a few fundamental principles to consider before getting started.

Some of these may seem initially counterintuitive but they are based on a lot of real experience and projects in this field so please bear with us. We believe the concepts could be invaluable to our customers particularly when it comes to getting everyone’s expectations aligned.

Basic principles before we start. 

Your decision to leverage automation follows the same consideration as any other tool; what problem are you trying to solve, what are your goals and the expected value you want it to provide. 

With specific regards to test automation, if your primary goal is cost or effort savings it is an amber flag that you have other issues in your development or testing process. We can help you evaluate that risk.

Test automation scripts are really good for verifying known risks; test cases either pass or fail.

New products have far more unknown risks, even when automation is fully in place. It makes sense to maintain a holistic testing view that also includes other needed testing areas and development practices. Automation will not replace this good practice but is a necessary extension.

Potential goals of automation

Consider your automation goals in advance of any other decisions. Automation can be used in many areas of software development. The following list includes a few of those ideas even though we won’t dive into all the areas within this article.

These goals not only impact your decision to leverage automation but will also help with other factors such as who will automate, the level you automate at and what tools you will select to assist you.

Layers of automation

The consideration of layers is very important because it asks the question, “At what level is it most efficient to catch a specific type of issue should an issue of that kind exist?” The unit and api layers tend to have narrower goals;  to get the rapid feedback loop from test results to the team.  It has great value in daily development work as developers quickly learn how well the code works. Ideally these tests are quick to create and run but also easy to maintain. 

Some of the ideas behind the Automation pyramid align with these key attributes. The pyramid also indicates the track to find a root cause for a single feature error. One should start tracking root causes for errors from the lowest level of a pyramid and only then move upwards. We don’t recommend blindly following this model but it is potentially useful for considering if it’s a good match for your current project.

The pyramid is very specific to test automation and does not apply to your overall testing model.  For a much broader viewpoint in risk management the holistic approach  is recommended.

User facing end to end flows

The layer is the most complex layer to automate at, it has more dependencies, higher maintenance costs and a slower feedback loop than the lower levels. At this layer good development architecture practices are needed with automation, as are basic coding skills.

Mobile App automation can further complicate things compared to web apps, as there are less tools available and it’s often less testable. For example on a web application often it’s a straightforward case of directing a tool to a url, whereas on mobile apps you may need to consider things like emulators, debuggable builds, WebView’s outside of the application and native device configurations. It remains a very important opportunity for automation though particularly when it comes to the significant device variation risk. When it comes to the end to end layer and finding issues, hands-on risk based testing will often offer much more value.

Common automation models

Here are two example models we have leveraged from both where our own team is building the product and where our automation engineers step in to help out our customers teams.

Efficiency focused team model

This model really suits new products where close team collaboration is at the center of development. It’s designed to keep the automation code even at the end to end layer as close to the whole team and product code as possible.

The combination of high levels of collaboration, whole team contribution and building in parallel to product code make this a highly efficient approach to leveraging automation. 

It can be challenging initially for testers as some coding is required and we’ll often pair developers and testers together on automation to get the most out of this model. Our next article about Flutter automation will go into more detail on this approach.

Regression or complexity risk model. 

There are also many products/services out there which have run into regression issues and over time it has become detrimental to the team.  Automation can be used to primarily reveal the issues and to cover this risk.

It’s often worth optimising the efficiency of the Agile team model by utilising test automation so that it frees the developers from fighting regression risk fires on a daily basis.

In the hands of full time automation engineers the end to end coverage will often go broader and deeper. As it is distanced slightly from the developers and the code, there is the potential to utilise more tester friendly tools.

Once broad coverage of regression risk is in place it can be easier for the team to start addressing the root causes of the risk and consider a more optimal model going forward. 

Wrap up

We hope you have enjoyed reading some of our thoughts on automation and perhaps will take a few ideas that can promote active discussion before you jump in. It’s a great tool to add to your software development toolbox and help teams from many angles. We have a lot of experience in this area so get in touch for a chat and we can explore some options together. Or stay tuned for our next article that focuses on Flutter test automation.