Before being published to app stores, each mobile app is tested. However, the mobile app testing process is not so simple and clear-cut. We have outlined the key steps of the mobile application testing process that will help you obtain high-quality results for the entire project and establish a solid testing strategy.
Preparation for the testing process
When conducting a technical audit, we at Orangesoft often face the fact that QA specialists join the project later than they should. If you start the software testing process only after you get the first build of the application, there is a risk that mistakes were made at the stage of determining technical requirements and specifications. Reworking the final product is always more challenging. Below we describe the important first steps of your testing strategy that will save you time and money.
1. Device models and OS versions
Once you have your project idea, your business logic, and your target users, you should determine the mobile devices and the OS versions your mobile app will be compatible with. These will be the devices you will run your tests on as well.
To figure out the best mobile devices for your app, you need to answer the following questions:
- What devices are the most popular in the application launch region?
- If the project already has a website, which mobile devices are used more often to access it?
- What devices are the most popular with your target audience?
- What OS versions (Android and iOS) are required for the business logic of the project?
- What OS versions (Android and iOS)are the most widespread in the world at the moment?
As a rule, we use the following resources: Apple and Google distribution dashboard, Device Atlas, and Statcounter (a very convenient resource with information about platforms, devices, and data organized by country).
Your choice of the devices and the operating system will affect the functional requirements, application design, and the devices and OS used for software testing.
Usually, we recommend choosing at least 3-5 reference devices for each platform with a different screen aspect ratio, different DPI, and different versions of OS.
2. Specification and design prototype testing
Once the technical specification is completed, a QA specialist should be involved. If the design is ready as well, having a QA specialist at this stage is even more important. The specification and design prototype testing allow you to identify defects before they become an integral part of the mobile app.
A QA specialist will identify the incompleteness of descriptions, ambiguous wording, contradictions, and imprecisions. Design prototype testing eliminates the discrepancy in the technical specification and can help to identify problems with end user experience.
This approach will save you from challenges later down the line, and it will save time for the entire team and keep you on budget.
Test planning
At this point, the project has the final approved specification and design. The project management methodology, which usually determines the appropriate software testing method, has also been chosen. Everything is ready for the next stages of development. However, before you start testing, you need to take a few more key steps.
3. Test plan
Well planned is half done.
The fundamental document for the QA process is the test plan. This is a regulation document that describes everything related to testing a project. It includes:
Project overview
This includes the introductory information that will allow any specialist to easily understand the idea and purpose of the project, its target audience, description of all parts of the project, and the parts that are subject to software testing. Most of the time, the project has a back-end, an administration system, as well as at least one mobile app. Sometimes the project may contain several administration systems or a website in addition to mobile applications.
Links to the main project documents
This includes everything that may be useful for QA specialists, including the links to project space, technical specifications, project stack information, design source files, and a list of test logins and passwords.
Links to test checklists, test scenarios and test cases should be added to this block as soon as they are prepared. At the start of the description in this section, links to document templates can be placed.
Necessary equipment
In the first step, we identified the device models and OS versions that the application should support. Now, we need to determine which device models and OS versions will be tested. It’s extremely rare to test 100% of OS versions and device combinations.
Using emulators is another issue to consider. When you perform mobile app testing using an emulator, some difficulties may arise. For example, issues may come up with Face / Touch ID, fingerprint sensor, camera, voice control, device’s memory testing, voicing interface elements, shaking the device, multi-tap, and other control gestures. Plus, there is a significant difference between the application's behavior in the emulator and on a real device. Thus, some bugs are reproduced only on real devices and vice versa.
Interaction with other applications or devices
If the mobile app involves integrations with other apps or different devices, the QA should know the details. It is important to determine how integration testing will be performed and how to access the devices. For example, this is relevant for testing IoT mobile applications, where you need to have specific knowledge about the device function or the object which the app is integrated with.
Testing team
The test plan should specify a team of testers and the role of each member. However, it’s important to not only fix the team members on paper but also to hold an intro meeting with the team to discuss the roles, plans, tasks, and expectations of all project participants. All QA specialists must keep all testing documentation up-to-date throughout the entire development process.
Project space and bug tracking tools
The whole team should know the exact list of systems and tools used on the project. For example, here are a few of our favorites: ADB, Crashlytics, Postman, Instabug.
We recommend devoting time to learning new testing tools regularly.
Testing start and finish
Formulate the criteria and the deadlines for testing if the release date is fixed. If you plan to arrange private demonstrations during the development process, these dates are also important to fix in the QA team work plan.
4. Test design
At this stage, it is necessary to analyze the project’s requirements, assess the risks, and consider options for reducing them with minimal costs. Usually, during this analysis, we try to answer the following questions:
- What types of mobile app testing should be used based on the project’s technical requirements?
- What problems may arise during the testing process? How to avoid them or minimize their impact on the project?
- What product risks can be identified? How can we reduce them, and what are the possible consequences?
- What factors outside of the team’s control could influence the project?
- Are there any known release dates for new OS versions? What actions could be taken for them not to affect the planned release date of the app?
- Are there any risks that may arise during the process of publishing apps on the App Store and Google Play?
- How will a possible user load increase on the app be handled? What can be done to test the load?
- How fast should the application load, and how fast can users surf the app menu and its functions?
- What test documentation is needed for the project?
5. Types and forms of testing
Let's take a closer look at all the possible forms and types of mobile app testing. Usually, you would decide on the type of testing based on the project requirements, the budget for testing, the planned release date, and the team’s expertise.
Testing forms are the following:
- manual,
- manual + farm,
- manual + automatic,
- manual + farm + automatic.
As a rule, the most winning scenario is a combination of testing forms and types. For example, automated tests give the most accurate results and are available for reuse, but writing them can take an unreasonably long time. You can also perform manual testing of any app, sometimes even without any preparation (so-called “hallway testing”). The farm allows you to run testing simultaneously and even remotely on multiple devices.
The correct combination of manual and automated testing forms and types allows you to optimize the testing budget and get a high-quality mobile app.
Let's move on to the testing types.
Functional Testing
This is the type of testing that ensures that the application performs its functions correctly and meets the expectations described in the technical specification of the project. Depending on the features of mobile devices, this type of testing can include:
- Installation and version update testing, which checks that the app can be installed, launched, and updated to the new version without any problems.
- Interruption testing that is aimed at preventing users from losing data in forms they fill out or contents they create in case they get interrupted and leave the application to perform another task on the device.
- Testing the app’s performance with different network conditions and during the network switch, which makes sure that when the wi-fi signal disappears, the app will connect to an available mobile network. If there is no access to any network, the application should display a correspondent error. The app should resume where the user left off without losing user input once the network is available again.
- Testing the interaction with other devices and mobile apps, which runs such tasks as taking a photo using the device's camera app or sharing content in social network apps.
- Testing the effect of the app on battery drainage and the device’s memory.
- In-app payments testing, including Apple Pay, Google Pay, and purchase restore.
- Testing the behavior of the app with different screen orientations.
- Testing the tap on push notifications.
- Testing the error handling not only on the client’s side but also on the server.
Automation testing
Automation testing does not replace the manual one since the creation of autotests and their maintenance are time-consuming, especially when the requirements change quickly enough. However, autotests are a good option for performance testing and regression testing of important application features. Regression tests must be carried out every time after a bug has been fixed or new functionality has been added to make sure that those actions haven’t affected the rest of the functions. Due to the frequent repetition of the same actions, the human factor comes into play, and QA engineers could miss critical errors. Autotests avoid the human factor and free up time for more creative manual checks and new features testing.
User interface testing
This concerns the assessment of the interface elements' impact on performance. The app's user interface is checked to be compliant with the approved design prototype and the current guidelines from Apple and Google. Also, this type of testing includes checking the syntax (texts and formulations) for compliance with the style of the application, its requirements, and logic.
A/B and usability testing
This is an evaluation of the application’s clarity and ease of use. It is difficult for team members to see the app with the eyes of the unprepared and unfamiliar with the specification and design users. For usability testing, analytics is the first thing to set up. Analytics allows testers to see how much time users stay on the screen, how much time they take to fill in the forms, how often they go back to the app, and how many times they activate a certain button. Once this has been accomplished, the users are divided into two or three groups that are given access to the application builds that contain small differences in one element of the interface or any particular part of the functions. Based on the collected statistics, we can decide which changes are perceived by users better. This type of testing should be taken into account whenever you estimate the time and budget for testing.
Localization testing
If the mobile app is designed for users from different regions, text translation and its display should be accurate. Cultural specificity and differences in the format of data output, as well as different units of measure should be equally taken into account. For example, the decimal separator can be a period or a comma, and the date display can differ depending on the region. Our team loves to use the Localise tool to organize localization in mobile apps.
Accessibility testing
This is the assessment of how convenient the work with the app is for people with various health conditions. For example, it tests whether the interface elements are correctly voiced. We recommend reading about the basic rules and tips for checking the inclusiveness of a mobile application on the Global Infrastructure Hub website or in a series of articles from Usable Net on Medium.
Security and compliance testing
Security testing is targeted at finding vulnerabilities in the security mechanisms of handling end user data. For example, you cannot proceed with the registration in the application with a password that is too simple, like “qwerty” or “1234567.” The app should not request unnecessary permissions and should work properly, even if the user hasn’t given permission to use the requested data; sensitive data like logins and passwords should not be transmitted in an unencrypted form in requests to the server.
Performance testing
With the help of special tools, this type of testing checks the back-end performance with normal predicted load (the number of simultaneously active end users and the amount of transmitted data), with higher than planned load, and during prolonged work with a normal load.
Read also: Different types of mobile application testing for boosting app quality
6. Testing documentation
The test plan is the first document on the list. Based on the test plan, forms, and types of testing, the following documents are drawn up:
Checklists
It is important to describe the acceptance criteria, as these criteria will help evaluate whether the application performance is acceptable.
Test cases
The test case includes a sufficient number of tests that cover all the functionality. As we previously mentioned, it is unfeasible to test all the possible combinations of input parameters. Choosing the right data is key to identifying potential bugs.
Once test cases are written, they get divided into sets that will run in different situations, for example:
- Tests for separate application modules.
- Smoke test, which is a set that covers only the main functionality and is performed on each new build to decide whether the build is ready for more precise testing.
- Acceptance test, which is a set of tests covering all the functionality. This set runs until the product’s quality level required for publication is reached.
As a result of this stage, you will get a list of test cases containing a description of the idea validation, the necessary steps, the expected result, and test-suites. Test suites are sets of sequentially executed test cases, for example, for end-to-end testing.
7. Bug report
When working with checklists and test cases, QA specialists look for possible inconsistencies between the expected and the actual results in the app. However, simply finding them is not enough. It is important to describe all the bugs' details in a way that will allow the development team to easily reproduce and fix them.
For this reason, we use our bug report template for each project. No doubt, bug reports may look different for different types of testing and at different project stages. The type of project space and the project management methodology can influence the reports’ contents as well. However, there is a standard set of data/fields in every bug report:
- Bug # in case the system does not assign numbers automatically.
- Title. A short description of the defect, which gives an understanding of what goes wrong, where, and under what conditions.
- Severity.
- Priority from low to block level depending on how the business logic of the mobile application is affected.
- Steps to reproduce the bug.
- The environment. This always includes the mobile device information, such as version and build number, as well as the description of specific conditions under which the defect is reproduced, for example, screen resolution for GUI bugs.
- Expected and actual results. This describes the contradiction between the actual app behavior and the expected behavior described in the requirements. This discrepancy proves that the issue is not trivial, but is a defect that needs to be fixed.
- Attachments (screenshots, videos, sketches, logs).
Read also: How to write a bug report that will make your engineers love you [+free bug report template]
8. Testing
The testing process lasts until all the planned functionality is implemented and the required level of quality is achieved. For this reason, keep in mind the following:
- Each new part of the functionality obtained at the new iteration is covered by the tests developed at the previous stage;
- The interaction of new features with other implemented modules has to be tested;
- Ensuring that bugs that were previously discovered and fixed by the developers are no longer reproduced is achieved through confirmation testing;
- Regression testing is performed to make sure that the new functionality and bug fixes have not affected previously stable functionality;
- Written test cases have to be updated and supplemented. Missing this step will render future tests ineffective over time (the so-called “pesticide effect”).
Based on the results of a significant intermediate or final testing, we draw up an additional report - a test report. Essentially, this is an extended version of the bug report, which gives the Project Manager and the Product Owner an overview of the product’s quality. This kind of test report may contain a link to the executed test suites indicating the devices and the OS covered, functional quality assessment, information about the severity status of the remaining bugs, and compliance with the acceptance criteria, as well as recommendations for improvement.
9. Pre-release testing
Acceptance test includes precise and complete testing, which may still be required to stabilize the app performance and identify minor bugs. At this stage of testing, it's significant to compare the planned functionality and the one accepted to be sent to the stores for publication. You may need to eliminate some redundant functions or add new ones.
Final report on pre-release testing
The whole process of mobile application testing should be well-documented and be as up-to-date as possible. Double-check that the necessary entries have been made, then make your test summary report.
Test summary report should describe:
- Important information identified as a result of tests;
- Information about the quality of testing;
- Summary information about the quality of the tested mobile application;
- Statistics obtained from reports of various incidents;
- Information about the types of testing and the time spent on each of them;
- Indication that this mobile application is suitable for use in the current state of quality and meets all the criteria of accepted functionality and quality of work.
The decision to publish the app in mobile app stores should be made after the test summary report.
10. QA responsibilities after release
The work of the QA team does not end after the app release. If the team continues to work on the second version of the project, the whole process may start from the very beginning. However, even when further work is not required and the project is functioning successfully, it is best to set time aside for testing. In this case, the testing process will be a periodic check, which may include one or all of the following events:
- Testing after the new OS versions release. This is necessary because changes not compatible with a previously developed app may occur.
- Testing after SDKs updates. If the project exploits third-party SDKs or libraries, their updates could also lead to errors.
- Testing following user comments. End users can report errors by email or in comments in the AppStore and Google Play. It is important to try to reproduce the reported bugs and deliver them to the development team or the designers for fixing.
- Testing according to the minimum checklist. For post-release projects, it may be sufficient to take the previously used checklist and execute a sample check for the most significant functions.
Mobile application testing is a complex process, and setting it up requires effort from the entire team. You can adjust the steps of the testing process described above for each project. However, do not skip the above 10 steps during the testing process. This way, you are guaranteed to get a reliable result in the form of a high-quality mobile application.
Ready to test your app? Our dedicated testing professionals are here to help. Book a consultation to know more about our quality assurance services.
