Every product/development organisation will reach the breaking point where traditional security activities create friction to the delivery. Hence our journey started with having this pain.
We wanted to increase the security assurance across our product portfolio by helping the development teams to mature their security posture and simultaneously reduce friction introduced by those traditional security activities. Here, I’m discussing that having a yearly pentest is neither sufficient nor makes your people happy when there is a finding reported at the very last moment of deploying to production.

Shift Left

Our proposal to solving this problem was by promoting a shift to the DevSecOps manifest and vision culture (aka Shift Left). We decided to build a program to bring the level of confidence that our software is free of security vulnerabilities and that the organisation has minimised the risk posed by vulnerabilities through early detection and remediation, among other practices. The main concept behind security assurance was building security into the development lifecycle to be able to have a high level of confidence that by deployment time, your application is secure and ready for the world.

Program

The intention was to form a program using a bottom-up, collaborative approach enabled by management commitment. Everyone was welcome to contribute and influence the direction of the initiative. In addition, the security team would provide the other teams with the tools, training, mentorship, and guidance they can use to succeed in taking ownership of their security posture.

Additionally, a security champions role would be implemented so that team members with a stronger interest in security could develop their capabilities and promote security within their teams.
For scalability, we would promote security automation as part of the delivery pipeline and develop best practices in reusable frameworks and libraries.
Initially, we thought this should be how it looks, but we ended up with a better approach.

Framework

We found out there are two frameworks that could help us, BSIMM and OpenSAMM. The difference between them can be found here. We decided to go with a hybrid approach, using BSIMM to benchmark our maturity against specific industries (FinTech, Financial Services etc), using OpenSAMM to get the influence about the activities and processes for the maturing domains.

Phased Approach

We planned to roll out the program in two phases:

Phase1: Foundations

  • Visibility over application portfolio and risk profiles
  • Internal security standards and guidelines
  • Vendor/tool selection
  • Security infrastructure setup

Phase2: Implementation

  • Automation pilot programs
  • Process Refinement
  • Security Stakeholding
  • Team onboarding

Success Criteria

We published qualitative and quantitative criteria to measure the success of the program, such as:

  • Team maturity levels will be assessed and tracked: we are using Ninja’s seniority: Black-belt, yellow belt teams etc.
  • Product teams are comfortable working with security teams
  • Assets are classified based on criticality, more focus and efforts given to high critical apps,
  • Teams proactively and without a prompt reach out to the security team for advice or to ask questions about how they can improve the security of the product
  • Team members care enough about the security of their product that security champions start to form who acts as the security voice for their team
  • Defects from both manual and automated assessments are tracked, and metrics are collected and classified, the trend is analysed
  • Team and product-level timeline graphs for identified vulnerabilities are tracked to measure the performance

Security Champions Role

Security Champions are “active members of a team that may help decide when the Security Team needs to be involved. They act as a core element of the security assurance process within the product or service and hold the Single Point of Contact (SPOC) role within the team. They are trained, coached by the security team to be our ninjas.
We defined a role and made it official within the organization. We informed the Product Owners that Security Champions need to spend up to 20% of their capacity within security activities. Did we get full support? NO. Did we give up? NO. Throughout the year, we found our way to inject this role into the culture.
This role was the backbone of the program, and luckily, we got so many talents as volunteers. They helped us make the right decisions for the tools and vendors to automate the security.

How did it go?

We subscribed to a few commercial tools to perform automated assessments and integrated all the tools. We used an open-source project, “Defect-Dojo”, as defect management tool at the heart of the process. This tool acts as a repository for findings, bi-directionally integrated with all other tools and processes.
Trained people for using the tools, asking them to integrate their pipeline for early feedback.

The security team published an internal service catalogue for the dev teams if they need any support for design review, manual source code review, threat model etc.
Security posture meetings have been organised periodically to provide feedback to the teams based on their security trends.
Security champions recently started to race with each other to bring their components free from any vulnerabilities.
Audits and security assessments turned out with zero findings.

The final process looked like this:

Step Name Description

0

Security Design Review The Security Champions perform the security design review to detect any issue from a security perspective.
The Core Security team provides guidance, tools, knowledge, and assistance to Security Champions.

1

Threat modelling Sessions The Security Champions are trained in threat modelling activity based on application criticality (via appsec control matrix). Security Champions will gradually take full ownership of the architectural risk analysis process (with support from the core Security team), promoting it and sharing knowledge within the team.

2

Findings are pushed to Vulnerability Management Tool (VMT) Identified security requirements from the architectural risk analysis process are fed to a centralised vulnerability management tool (currently OWASP DefectDojo) and reflected in Jira. Security Champions receive notifications of the new findings and will promote the identified security requirements within the team, liaise with product owners for prioritisation, QA for security testing, and developers for mentoring during implementation.

3

Any input to the VMT are synchronised to JIRA Bi-directional sync: as JIRA tickets get updated, the corresponding items in the vulnerability management tool are updated accordingly. This allows Security Champions and the team to use the same tools to triage and manage security requirements. This bi-directional sync is performed for other security gates as well.

4

Security Libraries Security Champions identify additional security requirements that apply to the features under development and liaise with product owners to plan and prioritise them. The core security team provides guidance that Security Champions and the team can use for reference. The aim is to create a repository for future re-use cases.

5

IDE – Static Code Analysis Continuous feedback on potential security bugs is available within the IDE to devs. A lightweight scan is triggered on every local build, and feedback on possible security bugs or dependencies is provided in the IDE so they can be reviewed and remediated before committing any changes. For each identified vulnerability, extensive guidance is provided on how to verify and remediate it.

6

Sandbox – Static Code/Software Composition Analysis A sandbox SAST environment is available to test code changes against the defined security policy without publishing the results or generating any tickets. At any given time, an individual developer or team can trigger a full scan in an isolated environment without impacting the application’s policy compliance status.
This happens based on merge-request pipelines, so that the integration creates a discussion that needs to be resolved.
The scan and the MR message contain both SAST, SCA and open-source license (blacklist & whitelist based, refer to open-source license policy) information.

7

Container Scan

SAST & SCA

Docker images binaries are scanned for vulnerabilities, and any findings are fed back to the VMT for review.
Noise from the container scan to be lowered by utilizing light-weight images (i.e. Alpine)
Similarly, to step #6, the SAST and SCA findings will be performed and checked against the application’s policy compliance. Again, the results are reported to VMT, and bi-directionally synced with JIRA.
Additionally, secret leak detections are performed.

8

Integration Test An IAST will instrument integration to identify vulnerabilities at runtime. Depending on test coverage, this might be complemented with additional dynamic testing. Finally, identified vulnerabilities will be fed back to the VMT and handled as likely true positives during triaging.

9

Bug bounty and penetration testing The crowdsourcing process is utilised, several security researchers perform tests in the non-production environment.
All findings from the bug bounty and responsible disclosure programs, and findings from external and internal penetration testing engagements will be fed into the VMT and synchronised with JIRA.
Security champions are expected to prioritise the findings with the POs and deliver the mitigations.

10

Continuous vulnerability scanning and monitoring Findings from the tests of continuous docker scanning, new SCA findings for long-time not scanned repos, periodic AWS Inspector scan for OS and platform scans will be fed into the VMT and synchronised with JIRA for triaging by the Security Champions and the Security team.
The results are being sent to the VMT, so that Security Champions and the Security team can track them and make sure of follow-up (i.e. the decision for OS findings, either to patch or to wait for monthly rebuild/rotation of infra).

11

Continuous Observation SIEM (security monitoring tool), Continuous Security Monitoring Process.

Reference architectures from other companies / people

See more for devsecops reference architectures: https://waterplacid.files.wordpress.com/2018/04/devsecops-reference-architectures-2018.pdf

Special thanks to:

Luis Charruadas for planting the foundation of the program.

Reinier Vegter and Jubba Smail for implementing the program.

Mustafa Kucukaytekin
Mustafa is the Head of Security, passionate security leader at Payconiq, holding CISSP, CCSP credentials. Originally from Turkey, he likes skiing, scuba diving.