How Google Tests Software

Reference : https://www.mammoth-ai.com/mammoth-ai/software-testing-trends-top-7-to-look-for-in-2020/
  1. Some lessons from initial Google’s QA
  2. Testing at Google
  3. Engineering Roles
  4. Crawl, Walk, Run Approach
  5. Testing on the Toilet (TotT) Concept
  6. Fixing Test Hour Glass
  7. Flaky Tests at Google
  8. Few Catchy Statements found in the Book ‘How Google Tests Software’
  9. Conclusion
  10. Recommendations

Some lessons from initial Google’s QA

  • In early stages, team relied heavily on manual operations and only a small handful of software engineers focused on building, testing and releasing software.
  • As Google grew, lengthy manual test cycles delayed feature launches.
  • Identifying bugs later in the development cycle took longer and longer time to fix them.
  • Therefore, they determined that pushing testing upstream via automation would help address these issues and quicken velocity.

Testing at Google

Google considers 3 main important areas during Testing and I like to call them as “3 Gears”.

3 Gears of Success
  • Favors “small, frequent releases” and small, concise tests.
  • Instead of releasing new updates / features in large releases, tries to release as soon as they provide some benefit to the user, get feedback and reiterate as fast as possible.
  • Invest heavily in Unit and Integration Testing and use end-to-end testing more sparingly.
  • Googlers are encouraged to look for ways to speed up testing cycles and minimize test failures.
  • Initially Software Engineers used to test their own code, but with the growth of the team, they identified the obstacles in it.
  • Next, instead of rolling out a traditional testing team, they started creating specialized Engineering roles with different Testing Focuses.
    (This will be further discussed in the section “Engineering Roles”)
  • Engineers who focus on testing aren’t isolated in their own areas of Google’s offices, but are integrated with the rest of the engineering team.
  • Google also prominence on communicating product quality internally to increase ownership and outgrow the team to focus on quality every day.
    → They internally publish data about code coverage which help increase adoption of best practices.
    →Practice “Testing on the Toilet’’ Method to foster a Culture of Quality Ownership.

Engineering Roles

In most of the Software Companies, Test Engineers usually focus on the law hanging fruit, where they concern on Manual / Automated UI Testing Only.

Engineering Roles

Crawl, Walk, Run Approach

One of the key ways that Google achieves good results with fewer testers than many companies is that they rarely attempt to release a large set of features at once.

5 Channels

Testing on the Toilet (TotT) Concept

Testing on the Toilet
  • Sometimes it can be little bit hard for us to remind what we learnt or find some better solutions while writing a test code / fixing a code failure.
    If we could happen to see the similar types of posts , tips where you can’t ignore , it goes in to our mind easily and inspire us rather than try to by heart or google the solutions.
  • That is where Google came across this secret concept in the year 2006 with a small band of volunteers who are passionate about software testing.
  • They started to write flyers about everything from dependency injection to code coverage and then regularly plaster the bathrooms in all over Google with each episode, almost 500 stalls worldwide.
  • This is one of the most effective ways to spread ideas, share important tricks and techniques, generate discussion and drive new internal tool adoption to educate yourself and the rest of your company.
  • This is basically share the best practices in all levels of testing thorough out the Company to inspire the Software Engineers & Test Engineers .

Fixing Test Hour Glass

As Google grew, often the shape of the Google test distribution became undesirable, either top heavy or like an hourglass.

  • With the Ice-cream Cone Approach, it took more time for End-to-end Testing and then they converted to Hour Glass Approach.
  • Later, Release cycles were becoming even shorter, CI/CD and micro services were becoming a trend and then, Hour Glass Approach of testing also didn’t support this at all.
  • Due to less time for run thousands of Unit and Integrations Tests compared to manage huge UI Tests and could find the bugs in earlier stages rather waiting till UI stage, Google Tests converted to Pyramid Approach.
Evolution of Test Pyramid

Flaky Tests at Google

What is a Flaky Test ?

  • An option to re-run tests automatically when they fail & Report a failure only if it fails 3 times in a row.
  • A tool that monitors the flakiness of tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the critical path and files a bug for developers to reduce the flakiness.
  • Another tool detects changes in the flakiness level of tests and works to identify the change that caused the test to change the level of flakiness.

Few Catchy Statements found in the Book ‘How Google Tests Software’

Conclusion

  • Google has crafted their own process of moving fast. It’s a very Agile process that doesn’t get bogged down with someone else’s idea of what it means to be Agile.
  • Both Developers & Testers think about the Quality of Product according to their Engineering Role (SWE, SWET & TE) & Focus.
  • They focus on break the features in to small chunks first, Code little & test.
  • They do more Test on lower stages than higher stages by adhering to Test Pyramid Approach.
  • They don’t have specific manual testers & Test Automation is a must in each level of Functional & None Functional Testing.
  • A Good Test Engineer is who is Technical and cares about the Product in UI Perspective as well as understand the product at a system and end-to-end perspective.

Recommendations

  • I suggest that we can Implement Google ‘Testing on the Toilet’ Approach at other software companies too.

    Method 1
    : We can grab the PDF Versions of Google’s TotT episodes or create our own posters that are more relevant to the company and put them in places where both developers and testers can’t be ignored.
    Method 2 : We can initiate something called ‘Tip of the day’ Mailing System from Quality Engineering Department.
  • I Suggest that, we can start testing according to the new Test Pyramid Method & Automate Tests in all levels of Testing.
    (
    We can make an awareness of developers about this Pyramid as well)
  • Implement a Code Coverage Tool after the Unit Test with Quality Gates in possible Projects.
  • Apply Auto Retry for E2E UI Test Automation Suits to reduce the Test Flakiness.
    Ex :
    npm — g protractor-retry

References

Google Testing Blogs:
https://testing.googleblog.com/

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dilusha Rasangi Kumarage

Dilusha Rasangi Kumarage

Senior Lead — Software Quality Engineering