Mix, but do not shake. How we added Sec between Dev and Ops

Three in (one) boat

I think there is no point in explaining on tekkix what DevOps is and how it works, so I will skip the theoretical part (if you want, you can read, say, this post). The important thing is that this methodology helps to develop more efficiently and is therefore used everywhere. However, in traditional DevOps, security issues are often in the background. Code vulnerability checks are usually carried out at the end of the development cycle, when fixing them turns out to be long and expensive.

This is how it happens. We have a product, its source code is stored in a version control system. This code is compiled, functional tests are run, which pass successfully. Pipelines are automated and do not require constant human involvement. So, we are ready to release a new version of our product!

Actually, not quite. We need to check the code for security because compromising the product (and the data it works with) will negatively affect the business. Therefore, before deploying a new version to production, the security team starts checking the code for vulnerabilities and other issues: for example, whether outdated or vulnerable libraries are used, whether licenses are violated, whether sensitive information can leak, and so on. That is, they look at things that no one paid much attention to during development.

All the time while the security team is checking the release, the team cannot deploy it to production and is engaged in something else — for example, developing the next release. After a couple of weeks, a report on the detected vulnerabilities comes, and the release goes for revision. The latter requires time that the development team planned to spend quite differently, and as a result, the next releases are delayed. Various strange thoughts arise — maybe not release this release at all, but fix the vulnerabilities immediately in the next one? Then why did we make the last release? Won't it happen that the security team will find some more problems in the next release and we will also want to skip it, moving straight to the third one?

As a result, we get three groups of dissatisfied people. The business is dissatisfied that releases are delayed, deadlines are not met, and the cost of the project increases. Developers are dissatisfied that they are hindered from working, plans are interfered with, and sprints are spoiled. And the security team is also dissatisfied — the better they work, the less they are loved, because their quality work forces others to change plans and shift deadlines. And if an incident occurs, all the responsibility will lie with them — this also does not add positivity.

To solve the problem of general dissatisfaction, we use the Shift Left approach — this is when security is thought about from the earliest stages of development. It allows you to identify and eliminate vulnerabilities not when deadlines are already pressing and management is constantly asking for status, but when you are still calmly writing code for the product. Security becomes part of the development lifecycle, starting from design. In other words, DevOps turns into DevSecOps. At the same time, the security team itself is involved in the processes not just as a controlling and punishing body ("there will be no release until we allow it!"), but as active participants in the working group who contribute to the design of the application or the collection of functional requirements. This allows, although not fully, to implement the concept of Security by Design.

The use of the DevSecOps approach allowed us to identify and fix vulnerabilities at an early stage, as well as improve code quality through monitoring. The business sees: customers and users trust products more, whose developers systematically care about security. Moreover, the established interaction accelerates time to market — security research goes in parallel with development, saving the overall time of delivering the finished product.

All this, of course, does not guarantee the happiness of all participants in the development process, but still makes their lives easier. Again, management is never completely satisfied, but in our case (spoiler!) the result was accepted.

Seven questions to the manager

As it sometimes happens, what sounds good and coherent in theory does not always work out in practice. At least not the first time. Our implementation of DevSecOps was complicated by the fact that "Basis" was formed by merging several companies, each of which had its own processes and approaches to software development. We needed to study the products and create unified rules for everyone, including security rules.

The result of brainstorming and trial and error was a list of questions that we ask before starting work on a new project:

1. Who will we contact?

First of all, we find a person whom we call a security champion. This person should have a good understanding of the product code and be interested in improving its quality, so usually it is the team lead or project manager. The security champion is the link and our main ally in the development team. In our case, this person can assign tasks to developers and is assigned to all tasks in the tracker.

2. How is the project assembled?

We find out how the project is assembled: by which teams and on which hosts. This helps us understand the infrastructure and determine which tools and methods we will use to integrate security.

3. What languages are used?

Knowing which programming languages are used in the project, we can select appropriate tools for static and compositional code analysis.

4. What is the team's git-flow?

We study the team's git-flow to understand how the code work process is organized. Which branches the team commits to, which branches they create, whether there are release branches, and so on. This helps us implement automated security checks at different stages of development.

5. What is the release schedule?

We clarify whether there is a release schedule and what the exact release dates are. This is important for planning security checks and ensuring that vulnerabilities are fixed before a new release.

6. Who will be involved in fixing vulnerabilities?

We identify those who will have access to security scanners and vulnerability management tools.

7. What are the deadlines for fixing vulnerabilities?

We always discuss the deadlines for fixing vulnerabilities, as well as the periods when DevSecOps engineers will come to check the status of their closure.

Only after receiving comprehensive answers to these questions do we take on the project.

DevSecOps Inventory

Now about what we actually work with at "Basis" (besides our brains). We have a whole zoo of tools - some we use on a regular basis, some situationally, some are still in distant plans, and some we are already abandoning.

Static Code Analysis (SAST)

Static code analysis allows you to identify vulnerabilities at early stages of development. Previously, we used SonarQube for analysis - it checks a large number of programming languages, easily integrates with other tools. Now we have switched to AppScreener and Svace. Both tools are of Russian origin: the first analyzes source code and executable files, the second performs static analysis at the build stage.

Dependency Analysis (SCA) and Proxy Repository

Code analysis for the use and dependency on open source components includes checking their licenses and identifying known vulnerabilities. Previously, we used the Dependency Check tool, but over time we abandoned it due to limited functionality. Then we switched to Dependency Track — a platform with its own interface and a plugin for Jenkins (where we automate development processes). This tool supports more programming languages compared to its predecessor, performs license checks, and integrates into build pipelines.

However, DT did not support the C and C++ languages that we use. We also lacked automatic policies that would prohibit the use of a library if it had critical vulnerabilities. These and other issues led us to now use CodeScoring - a Russian solution for compositional software analysis. It ensures safe work with open source, checks license compatibility, and assesses code quality within the team. This product has an OSS Firewall that blocks third-party components in our Nexus proxy repository according to configured policies. A pleasant bonus was the ability to check Docker images here as well, but more on that later.

Dynamic Analysis (DAST) and Fuzzing

For dynamic analysis (i.e., on a running application) we use:

• OWASP ZAP — a tool for testing the security of web applications;

• Nessus — a vulnerability scanner that analyzes network applications;

• Burp Suite — another tool for testing the security of web applications;

• Crusher — a Russian tool for identifying security issues. There is a fairly detailed article about it from our colleagues on tekkix.

Recently, we have been actively developing fuzzing in our company. The main assistant in this is Sydr-fuzz — a Russian dynamic program analysis tool that implements hybrid fuzzing using Sydr and modern fuzzers (libFuzzer and AFL++), and also automates the coverage collection stages.

Vulnerability Management

Vulnerability management is a separate big topic. Therefore, I will not dwell on this in detail, but will briefly describe the algorithm and tools that we use:

1. Scanning and detection. We conduct regular code and application scans to identify vulnerabilities.

2. Analysis and reporting. Using the DefectDojo vulnerability management platform, we analyze scan results and generate reports on identified vulnerabilities. The platform integrates with various security scanners, so it is convenient to work with.

3. Prioritization. We assess the criticality of vulnerabilities and prioritize their remediation.

4. Recommendations and remediation: the ASVS standard, developed by OWASP for verifying the security of web applications, provides recommendations for eliminating identified vulnerabilities in order of priority.

5. Monitoring and control: continuous security monitoring helps maintain a high level of protection and respond promptly to issues.

Containers and Security

Separately, I will tell you how we ensure the security of containers. Project builds are done differently, but they have one common feature — they are somehow related to containers. Projects have base images, on the basis of which final ones are assembled, as well as those in which applications are compiled.

Vulnerabilities in Docker images can be conditionally divided into:

• Vulnerabilities of the base image and system packages included in this image;

• Vulnerabilities in third-party dependencies;

• Vulnerabilities directly in the application code that runs in containers;

• Unsafe instructions for building the image.

To identify the first two types of vulnerabilities, we used Trivy — a tool for scanning container images, but now we have switched to Codescoring.

In addition to the toolkit, we have developed the following recommendations:

• Minimizing privileges. We do not give containers more rights than necessary. For example, we restrict access to system resources;

• Secure base images. We use verified base images with a minimal set of necessary packages;

• Scanning. We use tools like Hadolint to check Docker files for unsafe instructions and potential vulnerabilities;

• Container isolation. We limit interaction between containers and use secure network settings.

I note that now we work with all problems - in the code, in the containers, etc. - according to one algorithm:

1. We regularly conduct automated scanning of code and containers using various tools (Codescoring, AppScreener, and others mentioned above).

2. We analyze the scan results and generate reports that are sent to responsible developers and security teams for prompt response.

3. We assess the criticality of vulnerabilities considering their potential impact on the system and data. Priority vulnerabilities are addressed first.

4. We develop recommendations for eliminating vulnerabilities and provide instructions to developers. This includes updating dependencies, fixing code, and configuring settings.

5. In cases where the vulnerability cannot be quickly fixed, we apply mitigation measures, such as restricting access to vulnerable components or using additional protective measures.

What we have achieved

We managed to build unified development processes for all teams and ensure security at all stages. At the same time, not a single developer or security specialist was harmed, although sometimes it was not easy for us. Support from business units also played a role in this: our customers are interested in products that have passed FSTEC certification, and it requires a serious approach to DevSecOps. But my colleagues will tell you about this separately. If you have faced similar tasks of implementing DevSecOps, tell us how you solved them, share your experience and tools.