Case study: Scaling DevSecOps at Comcast

Piloting a DevSecOps transformation

Starting with a staff of just 16, Maccherone launched a small DevSecOps pilot program at Comcast. Out of the telecom conglomerate’s 600 application development teams, he identified around 10 already practicing what he considered true DevOps, making them ideal candidates for a DevSecOps transformation.

DevSecOps is just DevOps done right.

“By my definition, DevSecOps essentially means that empowered engineering teams take ownership of their products all the way to production,” Maccherone said. “It’s really the same definition I have for DevOps. To me, DevSecOps is just DevOps done right.”

Before Maccherone’s arrival, however, even Comcast’s most mature DevOps practitioners were supposed to hand off their software to the company’s siloed application security team, which would then “bolt on” security.

“Security would send it back to the developers weeks or, sometimes, even months later,” Maccherone said. The process interrupted programmers’ flow and undermined DevOps’ effectiveness, he added.

For the DevSecOps pilot, Maccherone’s team procured and customized a suite of automated security testing tools that Comcast developers could easily integrate with their own continuous integration/continuous delivery (CI/CD) pipelines, eliminating the need to surrender their software to the AppSec team for weeks at a time.

“We basically said, ‘Here’s tooling that works the way developers work and thinks the way developers think, that plugs into your pipeline and provides the feedback directly to you,'” Maccherone said.

Bringing in DevSecOps coaches

Maccherone rolled out the new tools to each participating development team in a 90-minute introductory workshop. He then assigned each group a DevSecOps coach, who would help the developers choose a few core practices to adopt over the next three months — for example, installing a software composition analysis (SCA) tool in the CI/CD pipeline and scanning for critical vulnerabilities.

The tooling would typically flag a handful of high-priority findings that the team could resolve over the course of a development sprint or two, with extra help available if necessary. “As your coach, I can always bring in an expert in slam-dunking or free throws to get you over a training hump,” Maccherone said.

After working its way through the critical alerts, the development team could then adjust the policy dial to not just scan code, but also block it from merging unless clean. “Critical SCA findings accounted for roughly 35% of all security incidents at Comcast,” Maccherone said. “Turn the dial to ‘blocking,’ and you’ll never have another one of those get into production ever again.”

Over time, a DevSecOps coach would encourage developers to slowly turn up the heat, perhaps also scanning for high- and medium-severity vulnerabilities, for example, or adding interactive application security testing or static application security testing findings to the mix. “The coach’s job was not to call your baby ugly, and it was never to tell you that you were doing it wrong,” Maccherone said. “It was to say, ‘What’s the next opportunity to improve?'”

With regular coaching and hourlong workshops every 90 days, a typical DevOps team at Comcast could typically reach DevSecOps maturity in about a year and a half, he added.

Practicing DevSecOps at scale

Soon, about 100 of Comcast’s software development teams were practicing DevSecOps. The results were compelling, with those groups seeing 85% fewer security incidents in production than their legacy counterparts.

Importantly, Maccherone designed Comcast’s DevSecOps program to scale, with his dedicated coaches able to work with up to 100 development teams per quarter. To further increase scalability, Comcast also created a federated coaching program, in which someone from outside the DevSecOps pilot team — say, a security specialist from a standalone business unit — could train to be a DevSecOps coach.

“They had to use our framework and our tooling,” Maccherone said. “And they had to shadow us three times, and then we reverse-shadowed them two times.” If they passed, the federated coaches could then lead DevSecOps workshops in their own business units.

Within five years, about half of Comcast’s 600-odd development teams had joined the DevSecOps transformation program. At that point, the company decided to transition the remaining teams and shut down its traditional AppSec program. Instead of a siloed team of 400 AppSec specialists, the company would have 100 DevSecOps pros.

“That essentially solved Comcast’s cybersecurity hiring problem,” Maccherone said. “We were able to do 85% better risk reduction with a quarter of the staff.”