Much more than a framework: Streamlined Automation Engineering within MassMutual’s FIT Environment

SDET, SEIT, AE, QAATE – lots of acronyms for the same job. So what differentiates QA Automation Test Engineers at MassMutual from those elsewhere? 

It is what we call our FIT Environment. Many teams will debate the merits of one test framework over another, and yes, the “F” in FIT stands for “framework.” But even though frameworks are the heart of automated testing, they’re just one part of what makes the process productive. 

With this in mind, at MassMutual we speak in terms of the Framework, Infrastructure and Tools (FIT) Environment in which we work. This environment is made up of several distinct but tightly integrated parts. These pieces all work together to create an environment that supports the rapid development, testing, and deployment of quality code. This environment is common across all development teams in our division and used beyond, as well. We treat the elements of this environment like any other production-ready system. We follow development best practices such as branching and merging, Clean and DRY code, and even unit and automated tests on some components.

Framework and Process

At its core, the framework is a Ruby-based PageObject model with Cucumber and Gherkin libraries. The framework leverages Watir, a Selenium derivative for web testing, and Appium for native mobile testing. It also leverages advanced data modeling to build tests. It has encryption, Visual QA, Accessibility testing, API testing and more baked in. It has capabilities for both Web and Native Mobile testing. With this framework, there is no need to recode complex but core functionality. Each project team maintains their own repository of tests and code but following DRY and Clean Code principles, all common methods migrate up to the framework and are available for all teams to leverage.

Everything is built around our core concepts of Readability and Maintainability. (Check out my 2011 blog post on this topic.) Keeping maintenance costs low requires discipline and consistency. To ensure our code quality stays high, and to share solutions, all automation code pull requests are reviewed by the Community of Practice engineers and approved by senior AEs. These reviews maintain our high standards for DRY and Clean Code. This process also helps share new solutions across the community. 

Infrastructure and Tools

Building tests is just the start. We have customized job handling in Jenkins that allows anyone to kick off a highly configurable test run. A run can be a single specific test, a smoke set, up to a full regression run across all projects.  

This tool can control several things, including:

• Test environments
• Where tests are executed. We have both in-house and service-driven execution platforms and can run over 100 tests concurrently.  
• Projects. We currently have almost 20 different supported projects that can be run in any combination.
• Test code branches. We can specify the branches of the framework and the project test repo that will be used. This is critical when new features are being tested in lower environments. 
• Major suite designations: Cross browser (including browser choices), responsive view (including screen size choices), Visual QA and Web Accessibility test suites.
• Tag-based test selection, per project

Test execution is then controlled and managed by a queue server that sends tests to execution nodes one at a time, so no node is idle when tests are waiting. This server can handle tests from multiple runs and manage concepts like team priority and license ownership. True story: our cloud-based execution platform provider thought something was wrong with their tracking because they had never seen a client consistently run at their license level without dropping tests for over-queuing.

Once the tests have run, we have a machine learning-based reporting engine that will pre-bucket test results, reducing analysis time and highlighting common failure points, as well as providing a highly customizable dashboard for all kinds of test results.

All of these components are fully containerized and tie directly into our automated CI/CD process. 

Roles and the Team

Automation is built by people in many roles here at MassMutual. They fall into several general categories. We have a senior team of engineers called the FIT team that is focused on building tools, training, and supporting the other engineers. The majority of the work is done by our Automation Engineers. They belong to a community of practice called the AutoBOTs. They are assigned to the squads they support, with time set aside to contribute to the FIT team work of improving the FIT environment. We also have offshore testers who contribute to test development, and teams from other divisions leveraging the FIT tools. Finally, our manual testers contribute as well, mostly by writing Gherkin, but also some code, too. All these people together form the Automation Community of Practice, a collaborative and self-supporting group working to drive quality forward.

The QA Automation discipline at MassMutual is unlike those you will find at most companies, regardless of industry. We place a high value on community and supporting our peers – our low turnover is a testament to the culture and practice we have developed. Our culture fosters innovation, collaboration, and inclusion, and our entire team is continually honing and improving the tools we use to deliver the best product possible. If this sounds intriguing, we are always looking for talented SDETs.