More Articles
Information Technology

How Security Champions Drive Change and Decrease SDLC Vulnerabilities

Rob Hornbuckle

Rob Hornbuckle

CISO at Allegiant Air

Rob Hornbuckle, Allegiant Air

Introduce a fun and engaging way to increase the secure coding your organization produces by appointing a Security Champion to your program.

Quartz Network Executive Correspondent Britt Erler sat down with Rob Hornbuckle, CISO at Allegiant Air to discuss ways to effectively secure the software development lifecycle (SDLC) through process.

Rob shares strategic ways to:

  • Simplify scaling your SDLC program by appointing Security Champions
  • Get buy-in for the program from executives and the development teams
  • Utilize a structure where results create opportunities to educate the organization on security

Quartz Network: Can you share some background about yourself and your current role?

Rob Hornbuckle: I am the CISO for Allegiant Air. Pertaining to the topic, the major thing here is, we’re a very large internal development shop. We do a lot of our own internal programming, specifically customer-facing. We view it as a differentiator for us in the market. And because of that, we maintain a good number of persistent teams at any given time. Securing the development process is part of my responsibilities. Prior to this, I was an interim CISO for United Technologies Aerospace Systems. And prior to that, I was the CISO for the Arby’s Restaurant Group.

Quartz Network: How is a SDLC program implemented into the organization and how does it scale?

Rob Hornbuckle: The idea of the program is the way that the program comes in place. Scaling becomes a very easy to accomplish thing. So, we call it the Security Champion Program. I have several people on my team that specialize in development security, but I have significantly more persistent teams than I have people. And in this instance, I have two people. And the idea is how do I get their expertise to pertain across all of those teams? And how can I be involved in that process from the entire beginning to end when I don’t have my developer or my teams as part of those persistent teams in the developer’s groups?

We started what we call a Security Champion Program. We have one developer on every persistent team that is designated as that security champion. The only rules that we have associated with who that is, is they can’t be the scrum master. They have to have at least some kind of an interest in the subject.

After they’re designated, we bring them in under my team. We do about two weeks of fairly intense training on exactly what it is we’re trying to do, what we’re trying to accomplish, why it’s there. We run them through a couple of breach simulations on different things of software, OWASP Top 10, showing you how different things become problems.

We’re not necessarily teaching them security but teaching them why security run applications are important. Putting them on the other side to see what happens when applications aren’t secure or when they’re not developed or coded securely. Then, that person is put back onto their persistent team and they’re responsible for maintaining the security practices and the secure coding practices on that team, with my members being available 24/7 for them to call and ask questions if anything comes up. If they have any questions, or if they get a weird error, or if they have some remediation they don’t understand, or something’s in the news that freaks them out, then they can call call my people and talk to them.

Beyond that, we then meet on a monthly basis and we bring all the Security Champions together. That’s the general basics of the model. The reason that it’s effective is because that person not being the Scrum Master, since they’re in charge of security and making sure, at the basics, things like code scanning, and stuff like that is accomplished and done, and meets all the standards for deployment.

They have to meet that in order to be part of the actual development lifecycle, while they’re also developing. At the same time, the Scrum Master is also responsible for the timelines, but they’re also responsible for that person, by extension, having those responsibilities. Giving us two people on every persistent team who have security as one of their responsibilities for making sure release happens in the proper time scale.

This is with your classic designs, or with your new CI/CD designs. It fits in with pretty much any of those programming models unless you don’t have persistent teams. If you don’t run a persistent team model, which I don’t know who necessarily does that anymore, but if you’d happen to not do that, that is not really going to fit in very well with you.

At that point, we bring them in for those monthly meetings. We have a lot of fun in them. We have a lot of games in them. We do a lot of competition type things. We track how each persistent team is doing, not just with what they find, how well they do, or if they release things. But how often they’re finding things, and we can show a progression of how that Security Champions working with their team and pushing those security knowledges and that security practice and that whole security culture down onto that particular persistent team. Therefore, increasing their secure coding from beginning to end. As that happens, they have less things come up, they have less things they have to fix. We can track them all on a big board, if we have people who are on lead and have what you need to do to catch up, and just make kind of a whole game out of the thing.

We do awards, including most improved. If anybody here has ever done any work on a swim team, when they were a kid, or if they have children that were on swim teams, you know, there’s like 12 places for every race. There’s so many different awards and recognition. We do the same general kind of thing. So you have most improved, you have best in categories. All these different things where people are improving, and you’re recognizing that improvement. Then they get shared further on, as well as up into the different people within management.

From a scaling standpoint, once you have all that in place, the scaling is just bringing in an additional person that is a Security Champion on every persistent team, and they bring that into the meeting. Our monthly meetings get a little bigger, depending on how many you add, but it scales wonderfully. And I have the two experts on my team that can manage probably as many as 30 or so persistent teams before I actually have to look at trying to increase my staffing size to support. And we don’t hit those particular numbers of persistent teams right now, of course, we’re not an Amazon or an Apple, where these people that have like hundreds of persistent teams, they’d have to have a bigger staff on my end to support it, but the model would still extend across them as well fairly well.

Quartz Network: How does the structure of this program benefit the organization as a whole?

Rob Hornbuckle: Part of the benefit is we can also do education across the entire organization through this. That’s what those monthly meetings, and that tracking, and that awarding can come from. And we can use it as touch points for FYI style-based emails across the entire organization. Because programming and these applications – especially in our business model – is becoming more and more of what the business is, especially since we’re public facing.

If you’re primarily a business entity, it might not work out that way for you. Being that we’re public facing, and that’s where those particular pieces come in, everyone in the company is paying attention to it. Everyone knows how integral it is to our business model, so it gives us the ability to both push the security knowledge out to the entire team and recognize the developers that are pushing that business practice forward. Everyone’s quite happy about those kinds of activities when you’re able to pull them off.

Quartz Network: How do you get buy-in for the program from the development teams?

Rob Hornbuckle: That’s where the whole interest in security happens. Once you have the whole thing running, you don’t have that much of an issue with it, unless you don’t do it right and you make it somewhat of a toxic environment. As long as you’re doing those recognitions properly, it’s an inclusive environment, you’re not slapping hands. It’s more of a, “Ooh, that was hard, let’s help so that doesn’t happen again.” You maintain that, and once the systems in place, then it’s easy to continue to get buy-in as more teams come in and you don’t have that difficult of a time. Prior to that, when you’re very first starting it up, that’s going to be your biggest difficulty for buy-in. Just like about anything else involving security, you need to get buy-in from the top down before you start. Because if you don’t have that executive buy-in, it’s not really going to go from there.

You just really have to dig in, start to get to know the programmers, send your people out. They shouldn’t be working with them anyway, if you’re running a classic model where your guys are running scans, and you’re hitting them with things they need to fix. It makes you seem a little villainous, but you at least know them, so you’d have your people go in there and start to try to get to know them better. Work on developing those business relationships. It’s skills that your people really should be developing in security anyway. Because as they start going up in the ranks, those interpersonal skills, business relationship type skills, they’re going to really become necessary. It’s a good exercise for them on top of it anyway.

You have them get in there and get to know the people in there, develop those business relationships, get to understand who likes the topic of security, or at least has an interest in it. Enough that they would take a week or two to actually learn what’s around it.

From their standpoint, the main way you get buy-in is either that interest or the fact that as an Application Programmer, you also have at have a touch of expertise and security. It’s nothing to always be able to maintain. If anything were to happen, or they wanted to move on somewhere else, it thoroughly stacks their resume for being able to find another application job. We’re not trying to say, “Hey, let’s teach you this so you can move on somewhere else,” but it’s always helpful to have a personal motivation on their part for doing these kinds of things.

Quartz Network: How you handle responsibility for security within the development teams?

Rob Hornbuckle: That responsibility has to be a bit of a delicate balance between the Scrum Master and the Security Champion. It really has to play out where the Security Champion is the one that has the responsibility for it. They’re the one that we’ve developed the connection with that we’re available to—that if they run into any problems we can work with. And we have them trained up on knowing what’s there, but you have to have it part with that release cycle. It has to be in the pipeline with that Scrum Master as something that has to be done before they can release. That’s when you work with the development teams and when you work with development management in order to help make sure that it’s actually included there.

There is some resistance when it comes to that, because they’re going to take the standpoint of, “It’s going to slow down my releases, it’s this big unknown.” You just have to push through it and say, “If anything else, it’s much better than taking it down after the fact. If there’s some kind of major security thing, let’s just give it a try.” And then make sure that it succeeds.

Doing what you need necessary to make sure everyone’s actually following the model and is on the right path. The most delicate part of the entire system is when you’re setting it up in the very beginning. That’s when there’s the biggest chance for error, that’s where there’s the biggest chance for it blowing off and where you really have to have that buy-in in that state, in order to get things started.

Quartz Network: How do you keep that engagement and that productivity up when you’re investing in this program?

Rob Hornbuckle: The beauty of it is the people in the application programming side that have no interest in it, don’t have to have an interest in it because we only really have that one Security Champion that’s on that team. There’s a lot of models out there that talk about going deep on a single team until they’re trained and then moving on to the next one. Those models have that big problem if there’s a couple of programmers on there that have no interest in security aspects of it. It can start to derail what you’re trying to do, and it takes a significant amount of time longer.

In many ways, we’re doing the same thing in that we’re going deep with a whole team, but instead of a single team, we’re taking a Security Champion from every team to make a single team, and then going deep with them to teach them everything that’s associated. Once you’re there, it’s developing that friendly environment, that no fault environment, that no blame environment, and making it fun and enjoyable and a knowledgeable experience. It’s almost akin to being party director to a certain degree, because these people don’t really work for you. They work for the company, but they work for your peers. And you need to make sure that it’s something that is engaging, that is interesting, that they have personal benefit from, and that has a bit of fun to it.

If you can pull all those off, you won’t have an engagement problem. That brings us to the other part of it though, where if you don’t, if you have that resistance at a senior level, that’s where you can have more of a major blowback and you don’t have as many tools in your belt to try to deal with it.

You really have to fall back on the business relationships that you happen to have with the rest of those executives. If there is too much friction there, then it might not be something that you can pull off now. At which point, you have to, from a leadership standpoint, be able to recognize that, be able to put it on hold, double down and reinvest in those business relationships with that senior staff until you can build enough trust and build enough belief that you are going to do what’s in the business’s best interest, so that you can get their buy-in and move on from there. As long as you have the buy in at those two levels in that nature, then the program won’t have any issues. You’ll start seeing a good increase over time at the amount of secure coding that your organization is producing.

Quartz Network: Any final pieces of advice for executives and organizations as they move forward with this?

Rob Hornbuckle: When you’re going through all of this and you’re deep into the security principles and practices, from time to time it gets easy to forget that people are still people. When you work to their interests, when you work on more of an emotional intelligence scaling and build those relationships, it really drives much more change and much more extension of security down into these teams than most any technological control and anytime that you just start barking statistics and facts.

For more industry best practices and insights from leading IT executives like Rob, join Quartz Network.