Effectively Securing the SDLC Through Process

Rob Hornbuckle

CISO & VP at Allegiant Air

Learning Objectives

Please Join the CISO of Allegiant Air, Rob Hornbuckle in this Executive Interview where he will discuss how the SDLC program scales and how to handle responsibility for security within the Dev teams.


"Work on developing those business relationships."

Rob Hornbuckle

CISO & VP at Allegiant Air

Transcript

Britt Erler

Hello, and welcome to the CISO VISIONS Cybersecurity Virtual Summit hosted on Quartz Network. My name is Britt Erler, QN Executive Correspondent. Thank you so much for joining us. I am pleased to welcome our executive speaker Rob Hornbuckle, CISO at Allegiant Air, as he discusses how to effectively secure the software development lifecycle through process. Welcome, Rob.


Rob Hornbuckle

Thank you for having me.


Britt Erler

It’s a pleasure to have you here and thrilled to dive into this topic today. But before we kick off, if you wouldn’t mind providing the audience with a quick background about yourself and your current role.


Rob Hornbuckle

Hi, my name is Rob Hornbuckle. I am the CISO for Legionary Lines. Pertaining to the topic, the major thing here is, we’re a very large internal development shop. We do a lot of our own internal programming, specifically customer facing stuff. We view it as a differentiator for us in the market. And because of that, we maintain a good number of persistent teams at any given time, and securing the development process is part of my responsibilities. Prior to this, I was an interim CISO for United Technologies Aerospace Systems. And prior to that, I was the CISO for the Arby’s restaurant group.


Britt Erler

Fantastic. You clearly have seen how the IT space has changed, what works, and what doesn’t. And let’s go ahead and dive right into that SDLC program, and really how it’s implemented into the organization. How does it scale?


Rob Hornbuckle

So, the idea of the program is the way that the program comes in place. Scaling becomes very easy and easy to accomplish thing. So, we call it the Security Champion Program. I have several people on my team that specialize in development security, but I have significantly more persistent teams and I have people. And in this instance, I have two people. And the idea is how do I get their expertise to pertain across all of those teams? And how can I be involved in that process from the entire beginning to end when I don’t have my developer or my teams as part of those persistent teams in the developers groups? We started what we call a Security Champion Program. We have one developer on every persistent team that is designated as that security champion. The only rules that we have associated with who that is, is they can’t be the scrum master. They have to have at least some kind of an interest in the subject. After they’re designated, we bring them in under my team. We do about two weeks of fairly intense training on exactly what it is we’re trying to do, what we’re trying to accomplish, why it’s there, run them through a couple of breach simulations on different things a software, Owasp Top 10, showed you how different things become problems, not necessarily teaching them security, but teaching them why security run applications are important. Putting them on the other side to see what happens when applications aren’t secure or when they’re not developed or coded securely. Then, that person is put back onto their persistent team and they’re responsible for maintaining the security practices and the secure coding practices on that team, with my members being available 24/7 for them to call and ask questions if anything comes up, or if they have any questions, or if they get a weird error, or if they have some remediation they don’t understand, or something’s in the news that freaks them out and then call my people and they can talk to them. Beyond that, we then meet on a monthly basis and we bring all the security champions together. That’s the general basics of the model. The reason that it’s effective is because that person not being the Scrum Master, since they’re in charge of security and making sure, at the basics, things like code scanning, and stuff like that’s accomplished and done, and meets all the standards for deployment. They have to meet that in order to be part of the actual development lifecycle, while they’re also developing. At the same time, the Scrum Master is also responsible for the timelines, but they’re also responsible for that person, by extension, having those responsibilities, giving us two people on every persistent team who have security as one of their responsibilities for making sure release happens in the proper time scale. This is with your classic designs, or with your new CICD designs. It fits in with pretty much any of those programming models, unless you don’t have persistent teams. If you don’t run a persistent team model, which I don’t know who necessarily does that anymore, but if you’d happen to not do that, that is not really gonna fit in very well with you. At that point, we bring them in for those monthly meetings. We have a lot of fun in them. We have a lot of games in them. We do a lot of competition type things. We track how each persistent team is doing, not just with what they find, how well they do, or if they release things, but how often they’re finding things, and we can show a progression of how that security champions working with their team and pushing those security knowledges and that security practice and that whole security culture down onto that particular persistent team. Therefore, increasing their secure coding from beginning to end. As that happens, they have less things come up, they have less things they have to fix. We can track them all on a big board, if we have people who are on lead and have what you need to do to catch up, and just make kind of a whole game out of the thing. We do awards, most improved. If anybody here has ever done any work on a swim team, when they were a kid, or if they have children that were on swim teams, you know, there’s like 12 places for every race. There’s so many different awards and recognition. We do the same general kind of thing. so you have most improved, you have best in categories. All these different things where people are improving, and you’re recognizing that improvement.Then they get shared further on the persistent team that their own, as well as up into the different people within management. From scaling standpoint, once you have all that in place, the scaling is just bringing in an additional person that is a security champion on every persistent team, so that they bring into that meeting. Our monthly meetings get a little bigger, depending on how many you add, but it scales wonderfully. And I have the two experts on my team that can manage probably as many as 30 or so persistent teams before I actually have to look at trying to increase my staffing size to support. And we don’t hit those particular numbers of persistent teams right now, of course, we’re not an Amazon or an Apple, where these people that have like hundreds of persistent teams, they’d have to have a bigger staff on my end to support it, but the model would still extend across them as well fairly well.


Britt Erler

Fantastic, and how does this program—the structure—benefit the organization as a whole?


Rob Hornbuckle

Part of the benefit is we also can do education across the entire organization through this. That’s what those monthly meetings, and that tracking, and that awarding can come from. And we can use it as touch points for FYI style, they’re based emails across the entire organization. Because programming and these applications, especially in our business model, is becoming more and more of what the business is, especially since we’re public facing, where if you’re a primarily a business business entity, it might not work out that way for you. Being that we’re public facing, and that’s where those particular pieces come in, every one of the companies paying attention to it. Everyone knows how integral it is to our business model, so it gives us the ability to both push the security knowledge out to the entire team, and recognize the developers that are pushing that business practice forward. Everyone’s quite happy about those kinds of activities when you’re able to pull them off.


Britt Erler

I think my next question, a lot of executives wonder how do you get buy in for the program from the development teams?


Rob Hornbuckle

That’s where the whole interest in security happens. Once you have the whole thing running, you don’t have that much of an issue with it, unless you don’t do it right and you make it somewhat of a toxic environment. As long as you’re doing those recognitions properly, it’s an inclusive environment, you’re not slapping hands, it’s more of a “Ooh, that was hard, let’s help get so that doesn’t happen again.” You maintain that, and once the systems in place, then it’s easy to continue to get buy in as more teams come in and you don’t have that difficult of a time. Prior to that, when you’re very first starting it up, that’s going to be your biggest difficulty for buy-in. Just like about anything else involving security, you need to get buy-in from the top down before you start. Because if you don’t have that executive buy in, it’s not really going to go from there. You just really have to dig in, start to get to know the programmers, send your people out. They shouldn’t be working with them anyway, if you’re running a classic model where your guys are running scans, and you’re hitting them with things they need to fix. It makes you seem a little villainous, but you at least know them, so you’d have your people go in there and start to try to get to know them better. Work on developing those business relationships. It’s skills that your people really should be developing in security anyway. Because as they start going up in the ranks, those interpersonal skills, business relationship type skills, they’re going to really become necessary. It’s a good exercise for them on top of it anyway. You have them get in there and get to know the people in there, develop those business relationships, get to understand who likes the topic of security, or at least has an interest in it. Enough that they would take a week or two to actually learn what’s around it. We’ll even make a little crazy certificates that don’t mean anything, like we’re going through an official certificate body or anything, but we’ll plant out a little thing for them that they should put on their own thing and their security champion and they went through the training. From their standpoint, the main way you get buy-in is either that interest or the fact that as an Application Programmer, if you also have at least to me, touch of expertise and security. It’s nothing to always be able to maintain. If anything were to happen, or they wanted to move on somewhere else, it thoroughly stacks their resume for being able to find another application job. As an Application Programmer, if you can claim any level of expertise within security or within secure coding, it makes it so much easier to find that next position, whenever that comes up. Not really—we’re not trying to drag them out of the company. We’re not trying to say, “Hey, let’s teach you this we can have we move on somewhere else,” but it’s always helpful to have what a personal motivation on their part for doing these kinds of things.


Britt Erler

Of course, and I think with all executives upskilling your teams is beneficial not only while they’re with your company, but for the person as a whole. They feel invested, you know. They feel like they’re actually getting something out of this change that you’re putting in place. And I know at the beginning, you touched on, you know, this checks and balance system, so to speak, that you have in place for your teams. But talk to me a little bit more about how you handle responsibility for security within the development teams.


Rob Hornbuckle

That responsibility has to be a bit of a delicate balance between the Scrum Master and the Security Champion. It really has to play out where the Security Champion is the one that has the responsibility for it. They’re the one that we’ve developed the connection with that we’re available to—that if they run into any problems we can work with. And we have them trained up on knowing what’s there, but you have to have it part with that release cycle. It has to be in the pipeline with that Scrum Master as something that has to be done before they can release. That’s when you work with the development teams, when you work with development management, in order to help make sure that it’s actually included there. There is some resistance when it comes to that, because they’re going to take the standpoint of it’s going to slow down my releases, it’s this big unknown. You just have to push through it and say, “If anything else to say, look, it’s much better than taking it down after the fact, if there’s some kind of major security thing, let’s just give it a try,” and then make sure that it succeeds. Doing what you need necessary to make sure everyone’s actually following the model and is on the right path. The most delicate part of the entire system is when you’re setting it up in the very beginning. That’s when there’s the biggest chance for error, that’s where there’s the biggest chance for it blowing off and where you really have to have that buy-in in that state quite a bit in order to get things started.


Britt Erler

And I think that leads perfectly into my next question in terms of engagement. You know, when there’s any sort of new program, any sort of change, there are always going to be those people that are reluctant, a little hesitant maybe just because their job function may change, their day to day tasks may change. So, how do you keep that engagement, that productivity up when you’re investing in this program?


Rob Hornbuckle

The beauty of it is the people in the application programming side that have no interest in it don’t have to have an interest in it because we only really have that one Security Champion that’s on that team. There’s a lot of models out there that talk about going deep on a single team until they’re trained and then moving on to the next one. Those models have that big problem if there’s a couple of programmers on there that have no interest in security aspects of it. It can start to derail what you’re trying to do, and it takes a significant amount of time longer. In many ways, we’re doing the same thing in that we’re going deep with a whole team, but instead of a single team, we’re taking a Security Champion from every team to make a single team, and then going deep with them to teach them everything that’s associated. Once you’re there, it’s developing that friendly environment, that no fault environment, that no blame environment, and making it fun and enjoyable and a knowledgeable experience. It’s almost akin to being party director to a certain degree, because these people don’t really work for you. They work for the company, but they work for your peers. And you need to make sure that it’s something that is engaging, that is interesting, that they have personal benefit from, and that has a bit of fun to it. If you can pull all those off, you won’t have an engagement problem. That brings us to the other part of it though, where if you don’t, if you have that resistance at a senior level, that’s where you can have more of a major blowback and you don’t have as many tools in your belt to try to deal with it. You really have to fall back on the business relationships that you happen to have with the rest of those executives. If there is too much friction there, then it might not be something that you can pull off now. At which point, you have to, from a leadership standpoint, be able to recognize that, be able to put it on hold, double down and reinvest in those business relationships with that senior staff until you can build enough trust and build enough belief that you are going to do what’s in the business’s best interest, so that you can get their buy in and move on from there. As long as you have the buy in at those two levels in that nature, then the program won’t have any issues. You’ll start seeing a good increase over time at the amount of secure coding that your organization is producing.


Britt Erler

Now, I would assume that you had this program the structure in place before COVID hit last year, were there any adjustments that you had to make to the overall framework?


Rob Hornbuckle

We had to learn how to make virtual meetings. I was proud that the system actually works quite well with the remote potential associated. If it keeps going on, if we end up with this model, going further from as far as the work from home model, there are a little bit of changes that I’m looking at doing. Primarily, if we all have to be remote for so long and I’m losing that touch point of these engaging activities. Once a year, I probably try to do more of a bringing everyone together type of event, sort of activity, almost like a team building exercise, viewing these security champions, then as an extension of my own team. That would allow us to have that additional touch point, at least once a year where we can make a real good strong connection across the teams. Then from there, pick it up on the normal day to day zoom type activities for the rest of the year, but making sure we’re at least holding that monthly cadence on those zoom meetings. It also makes it so that we have to actually track attendance a lot more than we used to have to consider doing it. It’s just something that you have to do with this new COVID work from home environment, even outside of this program, it’s just something that has to be paid attention to now.


Britt Erler

I think every organization is on the same page in that respect. I think that goes perfectly into my next question, you know. This program clearly has so many benefits for not just your employees, your teams, the organization, but as with any new structure that goes in place, there are always some obstacles, some challenges that you realize you wish you would have been better prepared for, or known about ahead of time. What are some of those? How do you recommend that executives make sure that they’re prepared?


Rob Hornbuckle

The biggest obstacle ends up being the curriculum during that one, two week boot camp type session that you do for the programmers. The first time that we or the first time that I’ve tried to do that at an organization, it didn’t go so well. You end up doing it just like you would consider doing security training for either your own staff or for any type staff. It ends up feeling kind of like mandatory PCI training or some other form of mandatory training because you sit them down in front of a computer, you have your people there to answer questions, you start making them go through modules, and it’s almost like you put them back in the middle school or something. Switching from that to more of an interactive and demonstrated type plan, is the biggest hurdle that we ended up having to get over. The biggest lesson learned that we had to get from it, those application programmers are going to be significantly less interested in the theories and in reading the OWASP Top 10 and reading what they all happen to do. They’re going to have a lot of interest in watching your people actively in front of them break applications that they themselves could have written. Showing exactly how these attacks happen, showing exactly what an injection is and how it works and what it functions with, showing the code, going out, and then showing what can return out of these test databases. It is a new environment, we basically had to build our own little—it’s not really a QA environment, it was kind of equivalent of a QA environment, at least from a technology standpoint. It gives us this this demo type capability that we can even then sit down and let them try to do similar queries, similar attacks, give them the ability to sit in the driver’s seat of that other side and drum interest into it that way. If I could have known that going ahead, I would have not crashed and burned so hard my first time I tried to do some form of boot camps for programmers to teach them security.


Britt Erler

That’s why this interview is so helpful because I think into the end of this year, and also top a next, a lot of organizations, a lot of executives are going to look to implement this sort of program into the company as a whole. As we wrap up this conversation today, obviously you see the benefits that this has and this is put into your organization as a long term plan. Any final pieces of advice for executives and organizations as they move forward with this?


Rob Hornbuckle

It gets easy to forget from time to time when you’re going through all of this when you’re deep into the security principles and practices that people are still people. When you treat them, especially when you need buy-in, and they’re not working with you as people. When you work to their interests, when you work on more of an emotional intelligence scaling, build those relationships, it really drives much more change and much more extension of security down into these teams than most any technological control and anytime that you just start barking statistics and facts.


Britt Erler

I completely agree. I think those are fantastic insights and just advice for companies, executives who really want to make sure that they’re moving their business forward and this is a great way for them to do so. Rob, thank you so much for being here and speaking to our audience today. Thank you to everyone who has joined us as well. I’m sure you will have further questions for Rob and we will have a discussion forum underneath his presentation where you can do so. Thank you again for joining us and enjoy the rest of the CISO Vision Cybersecurity Virtual Summit. Thank you.


Get full Q/N Access

Sign up to Q/N with a few details to watch this presentation.

  • Hidden
  • Hidden