Ensuring Continuous Security with DevSecOps in Deployment Pipelines: Q&A with Sathish Krishnan

May 22, 2023 No Comments

Written by Adam Darby

Today’s organizations are increasingly reliant on their technology and the foundational data that defines their enterprise and delivers business value. The need for proficient IT and software security continues to grow as advancements are made in the tech industry. DevSecOps currently serves as the central technology that integrates security into the IT lifecycle. According to Infosec Institute, market data shows that in 2023, DevSecOps is now used by 36 percent of respondents when developing software as opposed to 27 percent in 2020. Unfortunately, as the demand for this technology significantly increases, so does the amount of time and effort it takes to develop and test for security vulnerabilities within the code to ensure safety and privacy.

It is important to engage field experts for important insights on how to ensure more efficient and timely deployment from a strategic standpoint. Sathish Krishnan is a DevSecOps specialist with extensive experience in implementing DevSecOps pipelines for customers in healthcare, financial service tech, and payment processors. He offers insights and perspectives on how DevSecOps is used currently to ameliorate the development process and what the industry needs to do to move forward.

Q: How can the DevSecOps process drastically reduce deployment time, even after adding security validations?

Krishnan: To support DevSecOps, the security tools should be built into the native tools and delivery pipeline being used by DevOps teams. Automated security controls and activities do not replace an organization’s existing software development life cycle (SDLC) policies, standards, and procedures. Rather, automation augments existing capabilities to secure processes and assets. Organizations can start by automating controls and activities that are being performed manually. These stages include secure design and architecture, secure coding from integrated development environment (IDE) that detects static security issues, continuous build with static application software testing (SAST) and software composition analysis (SCA), integration and testing using dynamic application security testing (DAST), continuous fuzzing, deployment using artifacts repository and scanned container images, and finally, continuous monitoring and runtime defense using runtime application self-protection.

Q: How can the DevSecOps pipeline safely include multiple programming languages without jeopardizing operation and security?

Krishnan: There are a number of security best practices for GO and Python. One of these is the use of govulncheck, which is a reliable tool that helps Go users learn about known vulnerabilities that may affect their projects. Govulncheck analyzes the codebase and integrates it with continuous integration pipelines to identify vulnerabilities. Similarly, for Python code, requires.io scan can identify vulnerabilities and multiple runtime dependencies for container scanning using Aquasec and Trivy.

Q: What does this innovation mean for the future of the security teams such as “red” and “blue” teams?

Krishnan: Red and blue teams work on threat modeling and baseline/assessment for secure coding when a code is pulled, cloned, or committed. Both of these teams work with continuous integration and continuous deployment (CI/CD) tools to identify security issues and resolve them in the development phase. Certain crucial security activities, including threat modeling, penetration testing, and peer code review, can be integrated into the DevSecops pipelines. Continuous pretesting provides a powerful comprehensive scanner capable of detecting vulnerabilities based on a large vulnerability database collected from known common vulnerabilities and exposures (CVEs), intel, OWASP Top 10, and SANS 25. Tools such as Burp Suite are integrated with continuous integration tools like Jenkins that monitor for vulnerabilities daily. The vulnerabilities are remediated by different PowerShell or Python scripts. DevOps covers the areas of software development and information technology operations by combining the respective cultural philosophies, practices, and tools necessary for success. These combined practices enable companies to deliver new application features and improved services to customers at a higher velocity. DevSecOps takes this a step further by integrating and automating the enforcement of preventive, detective, and responsive security controls into the pipeline.

Q: How can this innovation be applied to other pipelines such as data engineering or machine learning?

Krishnan: DevSecOps can further be extended to machine learning (ML) and data engineering. It’s essential that running the training or inference along with the data pipelines adheres to established governance, regulatory, and compliance standards. This practice is also known as machine learning security operations (MLSecOps). The basic security for ML or data engineering starts with the build-out of the infrastructure as a code. The infrastructure as a code pipeline is validated with Sentinel, a tool to run security scans on the infrastructure. Any ML model must be deployed securely inside a private zone without being exposed to the public. In one project for a large healthcare customer, I implemented an ML model with Google Vertex AI. The implementation of the model was done in VPC Service Controls, where the models were not exposed to external clients. This implementation helped the customer to comply with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) standards. MLSecOps simplifies distribution, provides proper configuration of trusted graphical processor units (GPUs), enables post-delivery monitoring to ensure it performs as expected, and provides a reliable and performance-efficient infrastructure.

Q: How has DevSecOps improved the key performance indicators (KPIs) for the incident response teams?

Krishnan: DevSecOps has improved several key metrics for distinct security risks/incidents. They include:

– Vulnerability detection rate and the time necessary to resolve each security issue. The vulnerability detection rate tracks the new vulnerabilities that were identified during the initial production release and the vulnerabilities are tracked until it is fixed in a single release.

– Mean time to detect and respond to security incidents. Mean time to detect (MTTD) is the time it takes to detect a vulnerability and respond to the security incident after the new code containing a vulnerability has been checked in. This metric applies to vulnerabilities that were detected and is the duration in days from the time when the new code containing vulnerability was checked in until the time this vulnerability was detected.

– Secure code coverage percentage. This metric indicates the percentage of software assets that are covered by DevSecOps pipelines or the percentage of code that is integrated with Static Security tools such as SonarQube or Dynamic Security Testing tools such as OWASP Zed Attack Proxy.

– Compliance status. This metric indicates the results of the standard security checks for the resources. If the workload is running in an Amazon Web Services (AWS), the compliance status will be identified by AWS Config or AWS Security Hub.

– Security training and security risk assessment completion rates. These are completion percentages on compliance and security training. This will effectively train users and create awareness of the DevSecOps process. This metric can also be used to identify security gaps at a team level.

– Patch management success rate. This metric indicates the successful patches that were applied to mitigate the vulnerabilities.

– Incident response plan effectiveness. This metric tracks the number of incidents that were not discovered in the CI/CD pipelines. This is the percentage of incidents that are discovered after releasing the code in production.

– Security debt. Security debt is the change in several production vulnerabilities by severity (critical, high, medium, low) during the transition to the next release.

– False positive/negative rates for incorrect threat identifications. This metric is used to determine the accuracy of the threats determined by the Static Security testing tools or Dynamic Security testing tools. This is a good indicator to improve precision. Lesser false positives indicate that all the vulnerabilities detected indicate a security issue.

– Overall percentage of builds with security testing. This metric denotes the percentage of builds which have been integrated with DAST, interactive application security testing (IAST), container and image scans, or run time protection.

– Overall security test pass rates. This metric indicates the percentage of code builds that have successfully passed SAST, DAST, or IAST.

When these key metrics were implemented for a large payment processor, the security team reported an almost 75 percent reduction in security debts. Also, the incident response teams spend less time in detection and incident triaging to focus on preventive controls. All these metrics play a crucial role in identifying prime security issues and ensuring they adhere to protection and privacy standards while reducing the time necessary for completion.

Until recently, security testing and detection were implemented at the end of the IT development cycle. Depending on whether the testing permits or rejects each production application, it could be passed back to the developers for remediation. This often results in long delays in development and increased risks of releasing vulnerable software. Developers realized that it was time for a change in the process and began to “shift the security left” by implementing security measures throughout the lifecycle’s entirety.

By shifting left and procuring security measures earlier in the deployment pipeline, developers can seamlessly detect and fix potential security issues before it affects the final product. This allows DevSecOps to design software with pre-built security practices in place for easier, faster, and more affordable implementation.

Shift left for success

As organizations continuously become more tech driven, DevSecOps remains the strongest way to manage integrative IT systems, applications, and security. While promoting collaboration between the DevOps, engineering, security, and compliance teams, it serves as the easiest way to detect gaps throughout an enterprise and opens the door for automation opportunities.

Security is an essential priority for any business. If leadership places its faith in automated deployment and development, it is imperative to take the proper security precautions to achieve success.

About the Author:

Adam Darby is a freelance writer for various magazines and news outlets and a content writer for professional web pages, blogs, and social media. He is also the content marketing specialist and producer of the CDO Matters podcast for Profisee, a data software company headquartered in Alpharetta, GA. For additional information, contact adamdarby4@gmail.com.

Click here to view more IT Briefcase content!