DevSecOps 101

Everyone at some time or another has received an email from a service saying their password has been reset because data was compromised. Security has always been a big topic in IT, and it’s sad that some companies don’t take security as seriously as others. New vulnerabilities are found every day, and most of them are discovered long after they’re introduced. Equifax is currently feeling the pain of a breach, and it’s a lesson companies should learn from. Luckily, many IT professionals already are thinking ahead about security. But we still have things to overcome.

Part of the problem is that we’re still using traditional methods. We’re thinking about security only at the final stages of the software development lifecycle (SDLC). But security isn’t something you should consider only at the end. Imagine if car manufacturers started considering security after the car had been designed and assembled. It’s too late by then. Now let’s use the same analogy for testing. In order for a change in code to get to the end user, it has to pass through a set of stages. It’s good practice to start testing at the beginning of the lifecycle of the project—not only because you’re addressing security before it’s too late but also because it’s cheaper.

Just think of it. An application is ready to go live, and we just need to make sure it’s secure. Boom! The potential for a breach was found during testing, and it needs to get fixed in the code. This forces you to start all over again. That change in code needs to be validated to make sure it doesn’t break any existing functionality, meaning the end user will have to wait longer to get the fix or new feature in his/her hand.

There are other problems, too. What about the frustration of the application needing to be scanned just before the deployment? The security team can become a bottleneck. As the Agile Manifesto’s first principle says, “Individuals and interactions over processes and tools.” It’s more about a cultural change than just acquiring new tools to make apps more secure. It’s not enough to have leverage only on firewalls. We’ll get better results when, instead of being a bottleneck in the process, we work with the team to make security everyone’s responsibility. And that’s where development, security, and operations (DevSecOps) combine.

DevSecOps. It’s about feedback and a blameless culture. How will security teams know that someone found a hole in the app’s security? Well, the Ops team is good at this. They can provide the needed feedback by collecting metrics and reviewing logs. This same information can be shared with the development team so they can apply patches in the code.

Some say that DevSecOps is just a new buzzword. Others say that DevOps and DevSecOps are the same thing. In the end, the name doesn’t matter much. What matters is the intention to shift security to the left of the cycle.

To help move that shift along, let’s explore several topics worth seeing in more detail.

Security As Code

In order to innovate by pushing new changes in a timely manner and at the same time minimize risk, we need to pay attention to security right from the beginning. This becomes a no-brainer when you start treating your infrastructure as code (IaC). Think of security as code, like the test cases you’ll be running against the IaC. But don’t stop there. There are some tools out there that perform static application security testing (SAST) and dynamic application security testing (DAST) on the code, and they can help you go the extra mile.

SAST analyzes the source code by looking for common vulnerabilities—and it does this without needing the application to run. It won’t catch all problems, but it certainly reduces the attack surface. It’s a white-box testing methodology, testing from the inside out. In this regard, it’s similar to unit testing.

DAST covers scenarios by acting as if it was an external user. It runs against the real application, similar to integration tests. It’s a black-box testing methodology and tests from the outside. When you implement this type of testing in the SDLC, you’re making sure that, for example, SQL injections are not possible or that sensitive information is not being exposed to the end user. You’re also controlling what the user is able to see when there’s a managed and/or unmanaged exception in the code.

How much test coverage will you be able to get for on-premises? That will depend on if there’s an API you can use to interact with your infrastructure’s security. In the cloud, this can be achieved easily, as every interaction is done through APIs. You can run common checks, like seeing if the SSH port is open to the world, if some servers are in a private network, and if secrets are encrypted, among other things. Basically, you’ll need to make sure you’re giving least privilege and granular access to resources.

Injecting Failure Continuously

The only way you become better at something is by practicing. No matter how hard or painful it is, you have to do it. It’s just like when you were learning to ride a bike. We need to inject failure into our systems. The best example of this is the well-known Simian Army by Netflix, with its security monkey that’s constantly watching the state of the app’s security.

Don’t chase a final state for security. You can’t make security a project that has an end date. It never ends. New security holes are found every day, so it doesn’t make sense to stop thinking about how our apps can be protected if hackers are constantly trying to find new breaches. Vulnerability scans are not enough in today’s world; consider doing penetration tests as well, which can be performed as soon as you start development. Having exploit testing can bring many benefits. Instead of just relying on security scans, you’re actively monitoring.

Make use of the information you already have at hand and take actions based on the valuable information you can get from logs. All types of logs. Use log management tools like Splunk or the famous ELK (Elasticsearch + Logstash + Kibana). Gone are the days when you download logs from the servers and analyze them. Thanks to the cloud, it’s common to constantly generate new servers when something happens or when implementing blue/green deployments. You can’t afford to lose the last known state of a server. It can be your chance to find new ways to protect your application or see what the attack did.

Automate to Improve Agility

The time you take to react when a new vulnerability is found is crucial—especially if you have hundreds of servers (or containers). You need to act rapidly or be ready to face the consequences. You can take the approach of patching each and every server, as normal operations would. But if you’re using immutable infrastructure, you just need to create a new version with the patch applied and replace all servers. This is accomplished when you treat your infrastructure as code. There’s no better way to react in a consistent and predictable manner than with automation.

You can even protect yourself automatically when an attack is taking place. There’s an excellent example of this that I always like to present when talking about the value of automated protection. A site was constantly having DDoS attacks, and in order to remediate this, the administrator temporarily blocked the attacker’s IP addresses with Azure Functions. After some time, those IPs were again whitelisted, but the administrator was constantly monitoring the process. In other words, he was applying automation that he was actively supervising.

You can also use AWS Lambda functions to act based on API calls. For example, you can set up an alarm that will alert your team when a security group (like a firewall) changes. This is what the security monkey mentioned above does. You can even get to the point that if something changes in a way that’s outside of your defined parameters, you can revert it. That’s taking security to the next level.

Self-Service Reviews

Anyone should be able to see the state of security—especially auditors. It’s pretty common for them to request a static snapshot of an app’s security status, but that isn’t a very useful way to demonstrate that you’re applying security practices. This is like a picture. You always see the same thing when you look at it. Plus, it’s taking up someone’s valuable time, as they construct these boring reports. Even recording a video of how security checks are being performed limits you. You’re just watching a recorded period of time, with no idea what’s happening outside that window. And it’s still wasting someone’s time. I was pretty disappointed when an auditor requested I simply send a set of screenshots of password policies, active users, groups, and roles for a certain tool. And easily-manipulated image files shouldn’t be the only source of information, should they? In this circumstance, the phrase “trust but verify” makes a lot of sense to me.

Why don’t you instead invest some time in creating the building blocks that make security visible to anyone? When auditors come around again and start asking you about security, you can just give them access to whatever tool you built and make them do the reviews themselves. This time, we’re not wasting people’s work hours, and we’re gaining so much. We’re demonstrating that security is important to us and that we’re practicing it constantly. And our report becomes more credible because it’s not subject to human error or fraud, intentional or unintentional.

Learn more about everything you need to know about Test Automation and DevOps.

Security Is Everyone’s Responsibility

This is not just about a person or a team. We’ve already learned that isolating development and operations brings conflict. Don’t make the same mistake with an aspect of your application as important as security. DevSecOps is not introducing a new concept. It’s just making the point that we need to think about security for more than just a moment during development. It’s something that we should think about every day, and it’s everyone’s responsibility. It’s about collaboration. And it means that you can’t afford to have a finger-pointing culture, where everyone’s trying save his/her own skin. We all make mistakes, but having a team that supports you makes all the difference. This mindset is an enabler of experimentation, and that opens the door to innovation.

In the end, if you’re going to take away anything from this post, let it be this. Leverage automation to ensure your applications are secure, and act rapidly when there are potential breaches that could affect you.

Learn More

Want to know more? We now offer a 3-day instructor-led course on the DevSecOps! You can learn more about the course here.

Or, you can check out the DevSecOps manifesto and this awesome content list in GitHub full of links to tutorials, videos, tools, and more.