No matter how great, innovative, and shiny your mobile app is, if it fails to deliver a seamless user experience to demanding consumers, it’s doomed. Mobile testing ensures you’re launching functional, efficient, and performant native and hybrid apps that can set you apart from competitors and build a loyal customer base.
However, it’s not enough to carry out mobile app tests ad-hoc. You need a calibrated, iterative mobile app testing process along with a solid testing strategy to nip bugs in their buds. In this blog post, our mobile testing engineers have curated best testing practices for each stage of the testing process to help you ensure a high-quality and bug-free mobile app.
Step 1: Preparing for the testing process
When conducting a technical audit, Orangesoft’s QA team often sees companies making the same mistake — putting off the testing process till the first app build. However, putting mobile app testing on the back burner almost always leads to fundamental issues that could’ve been avoided had you kicked off the QA process early in the development. To avoid costly reworks and ensure comprehensive coverage, your testing process should draw upon a well-defined testing strategy.
Understanding the scope of testing
Once you have solidified your project idea, determined the business logic, and identified your target users, you should outline the testing scope, i.e., device models and OS versions your mobile app will run on.
Testing on a representative set of devices is essential to ensure compatibility and a consistent user experience. To figure out the best mobile devices for your app, you need to answer the following questions:
- What devices are the most popular in the target market(s)?
- If the project already has a website, which mobile devices are used more often to access it?
- What devices does your target audience use?
- What OS versions (Android and iOS) are required for the project's business logic?
- What OS versions (Android and iOS) are the most widespread in the world at the moment?
Pro tip: To gather device and OS data, our team usually turns to the Apple and Google distribution dashboard. We also use Device Atlas and Statcounter to gather information on platforms and devices by country.
The selection of mobile devices and operating systems will impact the functional requirements, application design, and the devices and OS used for software testing.
We recommend choosing at least 3-5 reference devices for each platform with different screen aspect ratios, DPIs, and OS versions.
Specification and design prototype testing
Your overall mobile app testing strategy should also build on technical specifications and design prototype testing. These two types of tests are conducted early in the development process to identify defects before they become deeply ingrained in the app's code.
As for specification validation, a QA specialist usually reviews it for clarity, consistency, and completeness. Design prototype testing eliminates discrepancies in technical specifications and can help identify problems with the end-user experience.
Step 2: Test planning
With the finalized specification, design, and project management methodology in place, your QA team can proceed with test planning. A mobile test plan is a regulation document that describes everything related to testing mobile apps.
This artifact includes the following key elements:
Project overview
This section describes basic information related to the idea and purpose of the project and its target audience. A project overview should also clearly define the scope of the project, including the specific components and features in line for testing. Usually, a project includes a back-end, an administration system, and at least one mobile app. Sometimes, the project also comprises several administration systems or a website in addition to mobile applications.
Links to the main project documents
In this section, your development team compiles everything that may be useful for QA specialists, including links to project space, technical specifications, project stack information, design source files, and a list of test logins and passwords.
As test checklists, test scenarios, and test cases are developed, they are also added to the document. The development team can also insert links to document templates at the beginning of this section.
Now that your testing team knows what devices and OS versions your app will support, they should prioritize the specific device models and OS versions for testing, as it’s not feasible to test all devices and OS versions possible.
Often, testers use emulators to simulate the behavior of a real device, such as a smartphone or tablet. However, emulators have limitations, meaning that some features, like facial recognition or device-specific gestures, may not function as intended in a virtual environment. Plus, there can be a significant difference between the application's behavior in the emulator and on a real device. Therefore, we recommend using real devices to validate the application’s quality.
In this section, your team also outlines the hardware, software, and network configurations that make up the test environment, as well as the tools or resources necessary for test data generation and management.
Interaction with other applications or devices
The QA team should also know about planned integrations to determine the right testing approach and the ways to access the devices. This is especially relevant for IoT mobile applications, where testing engineers require knowledge of the specific devices and functions that the app will interact with, as this will impact the QA process.
Read also: Taking the edge off edge cases in mobile and IoT product testing
Testing team
The test plan should define the roles and responsibilities of each team member involved in the testing process. To ensure each team member is fully aware of their responsibilities, the development team should hold a kick-off meeting to discuss the roles, plans, tasks, and expectations of all project participants.
Project space and bug tracking tools
A test plan also includes comprehensive information about testing tools, automation frameworks, and project management systems and tools used on the project. Typical testing tools include:
- Bug tracking tools, such as ADB, Crashlytics, Postman, and Instabug.
- Automation frameworks, such as Selenium and Appium.
- Mobile app testing frameworks, such as Espresso (for Android apps), XCTest (for iOS apps), Detox (for React Native apps), and others.
Entry and exit criteria
A test plan should clearly define the duration of the testing process and lay out the criteria that must be met before kicking off and wrapping up the testing. Exit criteria can be based on the number of test cases passed, defect density, or other predefined metric. If your team plans to conduct private demos during development, these dates should also be included in the QA plan.
Risk assessment
When preparing a test plan, a QA team also takes heed of the potential risks associated with the project and draws up a risk mitigation strategy that allows the team to reduce identified risks with minimal costs. During risk analysis, your QA specialists usually answer the following questions:
- What types of mobile app testing should be used based on the project's technical requirements?
- What problems may arise during the testing process? How to avoid them or minimize their impact on the project?
- What are the potential risks associated with the product, and how can we minimize their impact?
- What external factors could influence the project?
- Are there any upcoming releases of new OS versions that could impact our project timeline?
- Are there any risks associated with publishing the apps on the App Store and Google Play?
- How can we ensure the app can handle a surge in user traffic without experiencing performance issues? How can we test the load?
- How fast should the application load, and how fast can users navigate the app?
- What other test documentation is necessary for the project?
Step 3: Test case design and development
In parallel with test planning, the QA team prepares test scenarios that will lay the ground for test cases. At this stage, they decide on the testing approaches (automated, farm, or manual) based on the project requirements, testing budget, timeline, and the team's expertise. The rule of thumb is to use a combination of manual, farm, and automated testing to optimize the testing budget and maximize the coverage.
As for test scenarios, they provide high-level descriptions of the tasks users want to achieve with the solution. They also provide a framework for creating more specific test cases that can be used to verify the functionality and performance of the application. To execute test cases, the team also needs to prepare a set of the right testing data, specify data management standards, and prioritize test cases based on their importance and risk level.
Once test cases are written, they are grouped into sets based on the testing target, for example:
- Tests for separate application modules.
- Smoke tests that cover the core functionality and estimate the readiness of a new build for deeper testing.
- Acceptance tests that go through the entire functionality until the predefined product's quality level is achieved.
Regardless of the test case development approach, a QA team should ensure that it covers 100% of software requirements. A requirement traceability matrix helps the team track the relationship between requirements and test cases, ensuring that nothing is left behind.
Types of test cases
There are various types of test cases, each serving a different purpose in testing. A quality assurance team selects the set of test cases based on the following factors:
- Software requirements (e.g., web apps require compatibility testing, UI testing, and data validation);
- Risks — e.g., security testing may come to the fore in projects with higher security risks;
- Project constraints — if the project is on deadline, the QA team might prioritize certain types of testing over others;
- Testing standards and guidelines, such as the International Organization for Standardization (ISO) standards for software testing.
Functional testing
This type of testing ensures that the application functions as intended and meets the expectations described in the technical specification of the project. Depending on the features of mobile devices, this type of testing can include:
- Installation and version update testing, which validates that the app can be installed, launched, and updated to the new version without any problems.
- Interruption testing that is aimed at preventing users from losing data in forms they fill out or contents they create in case they get interrupted and leave the application to do another task on the device.
- Testing the app's performance under different network conditions and during the network switch to make sure that when the Wi-Fi signal disappears, the app will connect to an available mobile network. If there is no Internet access, the application should display a correspondent error. The app should resume where the user left off without losing user input once the network is available again.
- Testing the interaction with other devices and mobile apps, such as taking photos using the device's camera app or sharing content in social network apps.
- Measure the app's resource footprint, including the device battery and memory.
- In-app payments testing, including Apple Pay, Google Pay, and purchase restore.
- Testing the app's behavior on different screen orientations.
- Testing the tap on push notifications.
- Testing the error handling both on the client's side and the server.
User interface testing
UI testing focuses on ensuring that the application's UI is user-friendly, meets the needs of the target audience, and doesn’t eat into performance. A QA team also checks the UI against the approved design prototype and validates its compliance with the guidelines of the target platform (e.g., Apple and Google). Also, this type of testing validates that the UI text and formulations are consistent with the application's style guide and requirements.
Usability testing
No matter how feature-rich and high-performing your mobile app is, if it fails to successfully guide the user from point A to point B, it’ll be deleted immediately. That’s what usability testing is for: it is designed to check whether the target audience can easily navigate the system.
This type of testing requires a QA team to field-test the application with the real end users, observe how they interact with it, and gather feedback on their experiences. A testing team can also test different app versions to evaluate the effect of UI or functionality changes on user behavior.
Localization testing
If the mobile app is launched in multiple regions, your testing team should ensure that it’s suitable for each locale and free from visual or functional issues. QA engineers also verify that the application uses the correct language, date and time formatting, units of measurement, and currency for each target market. Our team’s go-to tool for localization testing is Localise, which automates the localization of design and UI copy for mobile apps.
Accessibility testing
Accessibility testing helps the QA team analyze whether your application has any digital barriers that prevent users with visual, hearing, cognitive, or other disabilities from effectively navigating it. Using WCAG guidelines as a benchmark, testers ensure your mobile app is perceivable, operable, understandable, and robust. We also recommend conducting accessibility user testing to make sure your app is truly inclusive.
Security testing
Security testing aims to find vulnerabilities in the security mechanisms handling end user data. Based on the system’s characteristics and the identified risks, QA engineers select an optimal combination of security testing methods, which may include penetration testing, vulnerability scanning, fuzz testing, and code reviews.
Compliance testing
Compliance testing is designed to validate the adherence of your mobile application to a defined set of internal standards and external regulations, such as HIPAA, GDPR, or PCI DSS. Also, this type of testing is necessary to provide evidence of compliance to auditors and regulators, which is especially relevant for healthcare and fintech applications. Performing compliance testing ad-hoc may lead to negative consequences for the company and hefty penalties.
Performance testing
Performance testing evaluates how well the system’s back-end performs in terms of speed, stability, and responsiveness under various workloads. This type of testing comprises a variety of checks, including load testing, stress testing, endurance testing, and others. It is usually carried out in a controlled environment isolated from the user acceptance testing (UAT) environment deployed for usability tests, functional testing, and regression testing.
Step 4: Test environment setup
Before getting down to testing, a QA team must also create and configure the necessary infrastructure, representing real-world conditions.
A typical test environment setup consists of the following elements:
- Hardware, including the devices, servers, and other hardware components required for testing.
- Software, including operating systems, databases, and other software applications.
- Testing data, such as sample data and test scripts.
- Network configuration tools that simulate different network conditions.
- Security measures to safeguard sensitive data and prevent unauthorized access to the testing environment.
A consistent, controlled, and reproducible test environment allows QA engineers to isolate the code, check its functionality, and catch bugs in their tracks.
Step 5: Test execution
The testing process continues until all the planned functionality is implemented and the required level of quality is achieved.
Here are a few best testing practices to keep in mind during this stage:
- A QA team needs to make sure that new functionality added during recent interactions is covered by the tests developed at the previous stage;
- QA engineers should validate the compatibility of new features with existing modules;
- Confirmation testing is necessary as a final touch to ensure that bugs have been successfully fixed and no longer plague the application;
- Along with other checks, a QA team should also run regression testing to make sure that the new functionality and bug fixes have not introduced any unintended side effects;
- A QA team should continuously update the written test cases to maintain the effectiveness of test cases and avoid falling into the trap of the pesticide paradox.
Step 6: Bug reporting
With every new bug discovered, testers provide detailed information about the bug, documenting core information about it — such as the severity, priority, and affected components — in a bug report. The more comprehensive the description is, the easier it will be for the development team to reproduce and fix the reported bug.
While bug report templates differ based on the type of testing and project stage, there is a standard set of data/fields present in every bug report:
- Bug # — in case the system does not assign numbers automatically.
- Title — a short description of the defect that indicates the issue and specifies where and when the bug cropped up.
- Severity — the degree of impact a bug has on the app’s functionality and user experience.
- Priority — assigned priority to the bug (from low to block level) depending on how it affects the app’s business logic.
- Steps to reproduce the bug.
- The environment — includes information about the mobile device, such as the version and build number, as well as a description of the specific conditions under which the defect is reproduced, such as screen resolution for GUI bugs.
- Expected and actual results — describe the difference between the actual app behavior and the expected behavior described in the requirements.
- Attachments — these can be screenshots, videos, sketches, or logs.
Based on the results of a significant intermediate or final testing, your QA team also draws up a test report. Essentially, this is an extended version of the bug report, which gives the project manager and the product owner an overview of the product's quality. A test report may contain a link to the executed test suites (indicating the devices and the OS covered), functional quality assessment, information about the severity status of the remaining bugs, compliance with the acceptance criteria, and recommendations for improvement.
Read also: How to write a bug report that will make your engineers love you [+free bug report template]
Step 7: Pre-release testing
Pre-release testing, also known as acceptance testing, is the final leg of the testing process that makes sure a new product version is bug-free and meets all the predefined quality standards. Acceptance tests should include complete testing and precision testing to ensure that the product is up to the mark and 100% defect-free.
The results of testing are documented in the final pre-release testing report, which should describe:
- Significant insights obtained from testing;
- Information about the quality of testing;
- Information about the quality of the tested mobile application;
- Statistics derived from incident reports;
- Information about the testing types and the time spent on each;
- Conclusion on whether the app is ready for release based on the current state of quality and compliance with the pre-established criteria.
The final decision to publish the app in mobile app stores should be made after carefully reviewing the test summary report.
Step 8: Post-release monitoring
The testing process isn’t a set-and-forget exercise. After your app is published on the app store, the QA team continues to keep an eye on it, constantly polishing and perfecting the functionality.
Here are the post-release testing activities your QA team should take on:
- Testing after the new OS versions are released — to make sure updates are compatible with the application.
- Testing after SDK updates — as updates in third-party SDKs or libraries can trigger errors.
- Testing based on the user feedback — the QA team reproduces the reported bugs and delivers them to the development team or the designers for fixing.
- Testing according to the minimum checklist — for post-release projects, your QA team can reuse the testing checklists from previous interactions and execute a sample check to validate the core functions.
It’s all about quality
As mobile app users demand high-quality applications with excellent user experiences, companies should rethink how they build and test them, moving away from ad-hoc testing to comprehensive quality engineering performed early in development. Skimping on mobile app testing will leave companies with a sub-par product that fails to gain traction in the market and secure early adopters.
Ready to test your application? Our dedicated testing professionals are here to help. Book a consultation to learn more about our quality assurance services.
