How Google Tests Software
Google was founded in September 1998 by Larry Page and Sergey Brin and back then same as initial stages of the other Software Companies, Google also used traditional processes of software testing.
But, with the rapidly growth of the Company, they learnt some lessons from traditional processes and innovated new processes of Software Testing.
As Google is one of the most successful software companies in the world, let’s see the lessons they learnt and Innovative techniques they use for software testing and how could we apply those processes and techniques to improve ourselves in software testing.
We will go through the following;
- Some lessons from initial Google’s QA
- Testing at Google
- Engineering Roles
- Crawl, Walk, Run Approach
- Testing on the Toilet (TotT) Concept
- Fixing Test Hour Glass
- Flaky Tests at Google
- Few Catchy Statements found in the Book ‘How Google Tests Software’
- Conclusion
- Recommendations
Some lessons from initial Google’s QA
- In early stages, team relied heavily on manual operations and only a small handful of software engineers focused on building, testing and releasing software.
- As Google grew, lengthy manual test cycles delayed feature launches.
- Identifying bugs later in the development cycle took longer and longer time to fix them.
- Therefore, they determined that pushing testing upstream via automation would help address these issues and quicken velocity.
Testing at Google
Google considers 3 main important areas during Testing and I like to call them as “3 Gears”.
Think Small and Fast
- Favors “small, frequent releases” and small, concise tests.
- Instead of releasing new updates / features in large releases, tries to release as soon as they provide some benefit to the user, get feedback and reiterate as fast as possible.
- Invest heavily in Unit and Integration Testing and use end-to-end testing more sparingly.
- Googlers are encouraged to look for ways to speed up testing cycles and minimize test failures.
Software Testing is a Focus, Not a Team
- Initially Software Engineers used to test their own code, but with the growth of the team, they identified the obstacles in it.
- Next, instead of rolling out a traditional testing team, they started creating specialized Engineering roles with different Testing Focuses.
(This will be further discussed in the section “Engineering Roles”)
Communicate Effectively
- Engineers who focus on testing aren’t isolated in their own areas of Google’s offices, but are integrated with the rest of the engineering team.
- Google also prominence on communicating product quality internally to increase ownership and outgrow the team to focus on quality every day.
→ They internally publish data about code coverage which help increase adoption of best practices.
→Practice “Testing on the Toilet’’ Method to foster a Culture of Quality Ownership.
Engineering Roles
In most of the Software Companies, Test Engineers usually focus on the law hanging fruit, where they concern on Manual / Automated UI Testing Only.
Google blends development and testing together in order to prevent bugs in quality instead of trying to catch them later when it is more expensive and harder to fix.
Google has a saying “You build it, You break it”.
They have essentially 3 Engineering roles / Focus Groups;
1. Software Engineers (SWEs)who focus on write code, design architecture and documentation and do unit testing in their code.
2. Software Engineers in Test (SETs) who focus on testability, improve the code in Performance and security wise since they are responsible for quality of the feature, refactor code and write Automated unit testing frameworks.
3. Test Engineers (TEs) who focus on User Testing by thinking in End User Perspective, Test with their knowledge of Domain / Product expertise and Automate API & UI Test Features.
Crawl, Walk, Run Approach
One of the key ways that Google achieves good results with fewer testers than many companies is that they rarely attempt to release a large set of features at once.
Google builds the core of a product and release it in the moment as it is useful to a large crowd as feasible, then get their feedback and iterate. This gives the chance to run tests and experiment on their applications early and obtain feedback from real human beings.
Ex. from a Google blog : Gmail → A product that kept its beta tag for four years. That tag was our warning to users that it was still being perfected. We removed the beta tag only when we reached our goal of 99.99% uptime for a real user’s email data. Obviously, quality is a work in progress!
In order to get a feature out to production Google has 5 Channels to get through & All the Automation Suits run in each of these channels every day.
If a bug is found in the field, a tester can create a test that reproduces it and run it against builds in each channel to determine if a fix has already been implemented.
Dogfooding → Internal adoption of software that is not yet released. The phrase “eating your own dogfood” is meant to convey the idea that if you make a product to sell to someone else, you should be willing to use it yourself to find out if it is any good.
Google breaks down their types of testing into 3 broad categories that include both manual and automated testing.
1. Small Tests : Written & Used by SWEs and SETs and Focus on Single Module.
2. Medium Tests : Written & Used by SETs and Focus on Integration of two or more features .
3. Large Tests : Written & Used by TEs and Focus on Real User Scenarios.
Testing on the Toilet (TotT) Concept
- Sometimes it can be little bit hard for us to remind what we learnt or find some better solutions while writing a test code / fixing a code failure.
If we could happen to see the similar types of posts , tips where you can’t ignore , it goes in to our mind easily and inspire us rather than try to by heart or google the solutions. - That is where Google came across this secret concept in the year 2006 with a small band of volunteers who are passionate about software testing.
- They started to write flyers about everything from dependency injection to code coverage and then regularly plaster the bathrooms in all over Google with each episode, almost 500 stalls worldwide.
- This is one of the most effective ways to spread ideas, share important tricks and techniques, generate discussion and drive new internal tool adoption to educate yourself and the rest of your company.
- This is basically share the best practices in all levels of testing thorough out the Company to inspire the Software Engineers & Test Engineers .
Google wanted to spread this secret weapon ‘Testing on the Toilet’ Concept with the rest of the world and they are putting episodes on the below blog on a regular basis and providing PDFs. ‘https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html’
Ex :http://gowest.nilin.se/wpcontent/uploads/2010/12/WhoTestsTheTests.pdf
https://www.flickr.com/photos/stevec77/2509234715/
Fixing Test Hour Glass
As Google grew, often the shape of the Google test distribution became undesirable, either top heavy or like an hourglass.
- With the Ice-cream Cone Approach, it took more time for End-to-end Testing and then they converted to Hour Glass Approach.
- Later, Release cycles were becoming even shorter, CI/CD and micro services were becoming a trend and then, Hour Glass Approach of testing also didn’t support this at all.
- Due to less time for run thousands of Unit and Integrations Tests compared to manage huge UI Tests and could find the bugs in earlier stages rather waiting till UI stage, Google Tests converted to Pyramid Approach.
At a glance, following chart gives you a good understanding of Benefits of Test Pyramid Approach ahead Traditional (Ice-cream Cone) Approach.
Flaky Tests at Google
What is a Flaky Test ?
Flaky test is a test that both passes and fails periodically without any
code changes. Initially Google found out that almost 16% of their tests have some level of flakiness associated with them.
This is one of the main challenges in Automated Testing and The goal of a good test suite is to make this signal as clear and directed as possible.
What are the Reasons of Test Flakiness?
It was a main challenge for them to determine whether a new failure is a
flaky result or a legitimate failure.
What are the Test flakiness Mitigation Plans at Google?
- An option to re-run tests automatically when they fail & Report a failure only if it fails 3 times in a row.
- A tool that monitors the flakiness of tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the critical path and files a bug for developers to reduce the flakiness.
- Another tool detects changes in the flakiness level of tests and works to identify the change that caused the test to change the level of flakiness.
Google has a new team dedicated to provide accurate & timely information about test flakiness and analyze the data from flaky test executions.
Few Catchy Statements found in the Book ‘How Google Tests Software’
Conclusion
- Google has crafted their own process of moving fast. It’s a very Agile process that doesn’t get bogged down with someone else’s idea of what it means to be Agile.
- Both Developers & Testers think about the Quality of Product according to their Engineering Role (SWE, SWET & TE) & Focus.
- They focus on break the features in to small chunks first, Code little & test.
- They do more Test on lower stages than higher stages by adhering to Test Pyramid Approach.
- They don’t have specific manual testers & Test Automation is a must in each level of Functional & None Functional Testing.
- A Good Test Engineer is who is Technical and cares about the Product in UI Perspective as well as understand the product at a system and end-to-end perspective.
Recommendations
- I suggest that we can Implement Google ‘Testing on the Toilet’ Approach at other software companies too.
Method 1: We can grab the PDF Versions of Google’s TotT episodes or create our own posters that are more relevant to the company and put them in places where both developers and testers can’t be ignored.
Method 2 : We can initiate something called ‘Tip of the day’ Mailing System from Quality Engineering Department. - I Suggest that, we can start testing according to the new Test Pyramid Method & Automate Tests in all levels of Testing.
(We can make an awareness of developers about this Pyramid as well) - Implement a Code Coverage Tool after the Unit Test with Quality Gates in possible Projects.
- Apply Auto Retry for E2E UI Test Automation Suits to reduce the Test Flakiness.
Ex : npm — g protractor-retry
As Test Engineers we all should try to be a Full stack Quality Engineers who will think about the many different aspects of product quality across all the application’s layers using different testing methods such as functionality, none-functionality (usability, performance, security etc. ) and will also be familiar with test automation strategies and technologies.
References
Google Testing Blogs:
https://testing.googleblog.com/
Articles on How Google Tests Software :
