Why Credit Karma Crafted a Tool to Automate Its DevOps Cycle

With about half of its team made up of engineers, personal finance company Credit Karma wanted to streamline its development cycle and created a deployment tool to automate and accelerate delivery.

Jeremy Unruh, head of developer efficiency and platform engineering for Credit Karma, says one of his team’s goals is to speed up the company’s engineering. Given the complexities of many platforms and the technologies they use such as Kubernetes, he says the typical product engineer must focus on shipping features. That can include experimenting and shifting traffic before customers see those features, Unruh says. “Part of my team’s motive is to abstract that away and provide a seamless experience where they really don’t have to think about anything but what they’re developing.”

Unruh says part of his challenge when he joined Credit Karma about three years ago was to increase efficiency of releasing code across the company. The engineers there had been using an older Jenkins-style system, he says, which served as a generic job runner. Developing products on that system meant clearing a few hurdles along the way, Unruh says, including jumping through a remote desktop running on a Windows computer. On top of that, teams building new microservices were required to write custom deployment code to move production forward, he says. That would be the basis for the job for the system to execute the service, Unruh says.

That meant everything was different because every team took their own approach, he says, which slowed them down. “It linearly required 15 steps just to deploy your service into production,” Unruh says. “It was really cumbersome and there was no way for us to standardize.”

Looking for ways to improve efficiency, he wanted to eliminate the need to jump to another host just to access the system. Unruh says he also sought to end the need for custom code for deploying a service. “I just build a service and I can deploy it,” he says.

That is where Falcon came in, Unruh says, which is used by every engineer in the company, who number upward of 700. He says many early users indicated that the tool cut their time in half. Now Falcon manages the release lifecycle through all of Credit Karma’s environments, Unruh says.

Falcon has also streamlined the manager approval process for features, he says, and increased visibility. It allows the teams to weigh traffic to services, which helps detect anomalies and errors with the new code, Unruh says, while only affecting a minimal amount of Credit Karma’s customers.

The years since the introduction of Falcon have prepared the company for new phases to its DevOps strategy, he says, which include multiple instances of services running across multiple datacenters as the company expands. Credit Karma has also invested in another product called Flare built by Unruh’s team. “Flare became the big warehouse,” he says. “Every time something’s deployed, every time an action happens, any time we do a poll request and we’re waiting for reviews -- all those metrics on how long things take, how many vulnerabilities happen in a security scan, all that data goes into Flare’s warehouse.”

That gives Credit Karma more visualization into the health of services from a code lifecycle perspective, he says. The plan for 2021 is to integrate Falcon and Flare together, Unruh says. “That will give us true CI/CD [continuous integration/continuous delivery] where the machine is making the decision.”

Such automation translates into production cycles progressing to a point where human engineers can be hands-off and let the system take over automatically while the engineer turns their attention to other tasks.

Change naturally invites some questions, and Unruh says engineers at Credit Karma early on wanted to understand how Falcon would fulfill all its promises while also being one-size-fits all. He met with leaders within the organization to show early mockups that demonstrated what the experience would be to sell them on it. Unruh also took in feedback during the development of Falcon, which led to snowballing adoption of it by the engineers.

When evaluating how to approach the creation of Falcon, he says they weighed building everything themselves, turning to open source, or other resources to find the right approach to get deliverables out faster. Credit Karma includes a feedback tool embedded in all the products built internally for engineering, Unruh says, to capture details quickly to help teams be more responsive. “That’s been invaluable,” he says, “especially for my teams. It gives us continuous feedback, which is really important if you’re supporting engineering from a DevOps perspective.”

For more content on DevOps, follow up with these stories:

What's Holding DevOps Back?

Ways to Fight the Skills Gap in DevOps and the Cloud

AIOps, DevSecOps, and Beyond: Exploring New Facets of DevOps

The Growing Security Priority for DevOps and Cloud Migration