Fannie Mae Turns to DevOps for Speed and Efficiency

The Federal National Mortgage Association, commonly known as Fannie Mae, is a US government-sponsored enterprise that is a leading source of financing for mortgage lenders.

To meet the challenge of working leaner and faster, Fannie Mae began an enterprise-wide program with the goal of becoming a more competitive, customer-centric organization that would be quicker to market while safely managing risk.

According to Gregg Davis, Fannie Mae's quality management manager, the program announcement inspired thousands of improvement suggestions. In 2015, three ideas were approved: adopting an Agile software development approach, simplifying governance, and implementation of DevOps. The project's primary objectives were to partner with customers, reduce complexity, empower developers, reduce costs and accelerate development.

Image: Shutterstock

Getting started

Project work began with transitioning existing development tools into a DevOps pipeline, Davis says. The technologies used to implement the pipeline included Jenkins, iCART, CAST AIP, HP ALM\QC, Rally\Jira and Nexus). "As developer needs evolved, additional tools have been added to the pipeline or swapped as needed during our journey," Davis notes. The additions included the self-serve deployment of virtualized servers, self-serve test data generation and test data management, integration of code quality and security scanning into the automated build process and the use of automated function points to measure productivity gains throughout the transition and adoption process.

"We had one particularly mature Agile project team that served as our lighthouse project that paved the road for DevOps in 2015," Davis says. "They not only helped to pilot individual technologies, but also their automation and integration."

Davis notes that Fannie Mae provided a robust ecosystem of technologies to its developers. "Selecting a specific set of technologies needed to support the complete end-to-end development lifecycle can be overwhelming for development teams," Davis observes. To simplify the selection process, Fannie Mae integrated a set of technologies known as "The Paved Road,” a concept formalizing a set of expectations and commitments between centralized and local teams, into its DevOps platform.

Having been a success at Netflix, The Paved Road has become a core concept at Fannie Mae. "It encompasses a steel thread of all tools used to accelerate and simplify our software development lifecycle without it being a mandate," Davis says.

Empowering the developer was a key driver in adopting The Paved Road, Davis says. "The Paved Road makes the technical infrastructure transparent to the effort of creating a software solution for a business need," he explains. "This is achieved through streamlining the developer experience via automation without implementing it as a least common denominator." Fannie Mae's development teams have the option of using The Paved Road or paving their own path. "The choice to pave their own path allows innovation, but the team will lose the streamlining benefits and will have full responsibility for the tools they are adopting," Davis explains. Paved Road owners may engage "off-roading" teams to mature their current road, or merge the two roads into a single one. "This is where the best of breed solutions take root," Davis says.

Overcoming challenges

Many of the tools used within The Paved Road require a significant number of manual setup steps to be completed before the developers are able to begin working. New developers needed time to become acquainted with these steps, as well as the individual tools. "The time and cost associated with onboarding was definitely an impediment to the adoption of the tools," Davis notes. "To overcome this challenge we automated the onboarding process, thus enabling development teams to onboard themselves with minimal effort."

Streamlining and automating enterprise risk control processes in support of The Paved Road was another challenge, one that required extensive cross-organizational collaboration, Davis notes. Another headache was a legacy technology stack that wasn't DevOps compatible. "Some of our starting point technologies were over-customized and became an impediment that needed to be transitioned to a new solution," Davis says.

As the rollout proceeded, project leaders discovered employee pockets that were resistant to the new working methods and unsure of the impact of automation would have on their current jobs. These staff members needed to be supported and brought into the program.

In the project's later stages, it became apparent that an important element was missing. "In the beginning, we had engineering teams driving the DevOps solutions," Davis says. "We later realized that we were missing input from our customers -- the development teams -- and then shifted our approach in 2016 to include the Developer’s Experience." The new effort helped make the development teams key players in DevOps solutions development.

Positive results

As the project neared completion, positive results began pouring in. "Provisioning development and test servers, which used to take two to four months, now can be done in minutes via virtualization and automation," Davis says. The implementation of automated testing, along with test data generation and management, realized an 80 percent savings in test time while increasing the quality of the testing being performed.

Integrating code quality and security scanning into the automated build process has greatly increased their adoption and consistent use. "Build and deployments continue to rise each year," Davis says. Overall productivity has increased by an average of 30 percent to 40 percent while costs have dropped by 30 percent. "Overall quality has grown by 32 percent, with some projects realizing a quality [score] as high as 70 percent as the speed of development allows them to rapidly address a backlog of technical debt," Davis says.

"Employees have reacted very positively and play an active role in developing our DevOps solution," Davis says, noting that the approach has given Fannie Mae's development teams the tools and culture necessary to learn new skills, break down barriers and get faster feedback. "By reducing or eliminating the overhead processes, development teams are able to focus more on their development activities rather than administrative paperwork," he notes.

The overall results were impressive enough to make the project a 2017 InformationWeek IT Excellence Awards winner in the DevOps category at Interop ITX in May.

Looking ahead

With DevOps now firmly embedded into daily operations, Davis says he is anticipating a continued refinement of The Paved Road through higher levels of integration and further promotion of its adoption by development teams. "In the long-term we are working to leverage the data generated by our DevOps pipeline to support DevOps Intelligence (BI for DevOps) and the evolution of our Paved Road into a 'Smart Road,' which incorporates TensorFlow Machine Intelligence to predict the outcomes of changes before the changes are made," he says.

Davis advises other organizations embarking on DevOps initiatives to ensure that they have a solid commitment from their executive team before starting. "Because, as you begin your DevOps journey, you will want to tackle it in a holistic manner -- people, processes and technology," he says.

Davis suggests looking beyond DevOps tools and working toward creating a change in mindset and culture. "Remember to take everyone along with you on the journey," Davis says. "Don’t be tempted to run out ahead of everyone else with technical solutions and forget to educate both your management and development teams on the benefits of making the change and securing their support."