Automation for Automation – An Executable Framework for DevSecOps

In an age where automated Continuous Integration and Continuous Delivery (CI/CD) is becoming more and more critical to the success of any organization, why are we still building our DevSecOps environments by hand?  Instead why don’t we leverage automation for our automation?  DevSecOps support teams are frequently faced with multiple challenges and using an automated DevSecOps framework will directly aid in meeting the expectations of the organization and development teams.

Commonly heard DevSecOps team frustrations:

We need to work with several corporate groups (network, infrastructure, security, etc.) to modify our automation tool ecosystem
We don’t own our environments and need to get multiple approvals to make improvements/updates
Installation and integration of any new tooling takes a very long time, including updates to all impacted development pipelines
On-boarding new teams to pipelines is very time-consuming and requires significant manual pipeline configuration
Pipeline and tool integration governance is poor across the organization, making support by our tiny team all but impossible

Development teams too are held back by the very automation that is intended to make their lives easier.  The issues include pipeline failures, numerous approvals for deployments, and difficulties in getting feedback on security vulnerability information from CI/CD build results.

Commonly heard Development team frustrations:

We just want our automation to work – we don’t have time to figure out pipeline failures

How then can we best resolve these issues?  Here we introduce a new approach to DevSecOps automation – The CoStar Automation Framework

The CoStar Framework focuses on three key benefits: building the DevSecOps automation environment, delivering software into production with greater consistency, and maintaining the overall platform across the organization.

Build

The first benefit noted is the creation of system build automation (aka Continuous Integration).  The intent is to establish a fully integrated and functional DevSecOps environment in minutes rather than months through the use of now standard Infrastructure as Code (IaC) techniques.  This approach uses automated environment creation, configuration, and depends on cloud hosted resources.

DevSecOps as Code – Rapid automated environment resource creation and integration

As shown in the figure, the CoStar Framework architecture is implemented through the use of a cloud-resource manager (e.g. Terraform), and resource configurator (e.g. Ansible), and a workflow management tool (i.e. Azure DevOps).  Each of these tools is leveraged where it is best suited; the cloud-resource manager allows for cloud-agnostic deployment of the initial infrastructure, which in this example is the creation of a set of Virtual Machines (VMs) or Kubernetes clusters that host Docker-contained DevSecOps tools.  The resource configurator is then employed for configuration of each tool, for example to generate integration security tokens or register a client application.  Finally, a workflow manager ties together and orchestrates the overall execution of the framework instantiation.

Figure 1. CoStar DevSecOps Automation Framework

An example of one way to integrate these activities is shown in Figure 2.  The automation pipelines are separated to allow them to be run independently. After the initial build step is completed the token generator can be re-run to perform token rotation, or the configuration playbooks can be run separately to register new applications into the framework.

Figure 2. CoStar pipeline orchestration

Another benefit of this approach is the ability to treat the DevSecOps environment as a product rather than a platform.  This enables creation of a DEV/TEST area to explore new DevSecOps tools, improved pipeline orchestration, or test updates to supported tooling without impacting production pipelines.  By the very nature of the framework, both the lower and upper environments are built identically which simplifies testing and approval of changes prior to roll-out across the organization.

Generate Pipelines from Templates – Rapid pipeline establishment and execution

Pipeline configuration for on-boarding new teams to a DevSecOps platform can be complex and require a great deal of manual editing to pipeline definition files (i.e. yaml).  To simplify and improve pipeline consistency, the CoStar Framework uses a set of configurable template pipelines.  These pipelines are pre-configured with connectivity and in-line tasks to integrate with the supported DevSecOps tooling.  For example, with SonarQube connectivity the same access tokens generated during initial configuration are re-used by the pipeline connectivity tools, in-turn established as part of the CoStar system build-out.  This allows a development team to on-board to the platform with a few initial product differentiating values (e.g. product name, unique product key, organization, etc.), and access to the source code to be built.  The CoStar configuration Ansible playbooks update integrated tools with these values, including generation of the initial CI and CD pipelines.  The development team is then ready to use the build system and respond to any discovered issues/vulnerabilities.

Consolidated Dashboard – Integrated observability into all pipeline quality measures and security findings

The next, and perhaps most useful, aspect of the CoStar Framework is an integrated observability into all pipeline execution results.  This permits teams to respond to the majority of findings without needing to access additional tooling interfaces.  Moreover, if additional specific information/findings are needed, then a direct access link is provided as part of the detail drill-down.  This is a key advantage in that development teams are not required to navigate multiple system interfaces for each of the different tools and instead focus on correcting discovered issues.

Deliver

The only true measure of value in software development is the secure release of capability into production.  The CoStar Framework is designed to provide continuous delivery (CD) through automation of key workflows and readiness states (for a more detailed treatment of release coordination please refer to the Perficient blog series on Release Coordination).  The combination of governed workflow automation, infrastructure-as-code target environment management, and built-in tracking metrics allows the CoStar Framework to scale across the enterprise.

Workflow Automation – Secure SDLC definition, governance, and execution

A critical part of repeatable, reliable product release is a set of well-governed automated workflows.  The intent of this automation is allow “presumptive release“, where every system candidate release is assumed ready for production.  The consequence of this approach is the need to unambiguously  define what is required to go into production with confidence (e.g. “readiness states“).  As part of the framework, once a production candidate release is identified it is automatically placed onto the production release schedule.  By analogy, the candidate release is like a rocket on the pad – all of the preparation steps are automated on the way to launch; only intervention by the launch director will prevent lift-off.

In the CoStar framework the four critical process areas (Intake, Construction, Release, Maintenance) are designed to automate as much of the release coordination activities as possible.  For steps that absolutely require manual intervention, the framework provides task management and automated notifications.

Metrics and Measures – DevSecOps historical metric gathering and reporting

A process is only as good as the feedback that allows it to grow.  Metrics and measures in DevSecOps are often difficult to collect due to the large number of integrated systems typically found in automated CI/CD pipelines.  In the CoStar framework the metrics are collected as a consequence of build results being captured and summarized in a common “single-pane” dashboard.  By analyzing the history of build results over time, each system development team will be able to track their improvements in release consistency – fewer uncaught issues/vulnerabilities.

Continuous Delivery – Configuration control and automated target environment compliance

The final piece to releasing a system candidate to production is to automate the configuration of the target environments.  Given the size and complexity of modern software deployments, this is a task that best left to specifically designed release management and deployment tools (e.g. CloudBees CD/RO, UrbanCode, Octopus, etc.).  The CoStar framework creates and manages a set of “deployable units” that can be targeted to specific deployment/release target environments.  In a future version of the CoStar Framework continuous compliance will be enabled to verify that deployment target environments meet pre-defined organizational security policies, such as operating system hardening or network whitelisting.

Maintain

Cloud Resource Management – Update or replace any automation tooling with direct redeployment

The architecture of the CoStar framework is based on modularity; the tooling is built into independent containers and deployed separately.  This allows for update/replacement of any given set of tools with minimal disruption to the remaining systems.  The automated playbooks are built to allow for on-the-fly configuration of running systems, including the generation of new security access tokens.  This approach minimizes the need to involve other teams in the organization, simplifies the process for framework maintenance, and places the authority over DevSecOps tooling with the team responsible for pipeline stability.

DevSecOps Platform Governance – Ensure platform configuration and consistency across the enterprise

Automation has proven to be one of the best, most efficient ways to establish and maintain consistency for enterprise systems.  A platform for DevSecOps is no exception.  By utilizing the aforementioned automation to build the CoStar Framework, governance is established around system tooling, user access, application pipelines, and the other critical aspects of DevSecOps best-practices.  As changes are introduced to improve the framework, they can be tested in stable on-demand environments prior to roll-out to the larger enterprise.  This provides a means for training environment creation as well, so that development teams are aware of improvements to the overall automation frameworks.

Organizational Personas – Clearly define organizational roles and responsibilities

Finally, it is important to remember that people are always going to be part of any development organization.  No amount of automation can replace the need for talented, creative, and goal driven teams.  In order to build these teams, and to show how each individual works within the processes and tools defined in the CoStar Framework, we are leveraging Organizational Personas.  These role and responsibility descriptions detail each individual’s responsibility area.  The also provide detailed tool expectations/guides, typical deliverable artifacts, decision rights, expected skills, and workflow assignments.

Figure 3. Organizational Persona – CI/CD Engineer

Essentially an Organizational Persona is similar but opposite of a marketing persona; the individual takes on the predefined qualities of a specific role rather than representing a diverse marketing target audience.  In essence, the individual is “putting on the badge” and assuming the well-defined responsibilities.  The value to this approach is to unambiguously define role boundaries directly understood by each individual.  If someone wants to perform a task assigned to another persona, that individual is now responsible for the entirety of the second role!  This strongly discourages blurring responsibility ownership and enables better collaboration between individuals and teams.

Conclusion

Most organizations have recognized the importance in hosting a DevSecOps program to automate their software and infrastructure development.  However, there are significant challenges faced by DevSecOps teams as they work to support larger and more complex corporate systems.  Many of these issues can be solved by treating as a product the DevSecOps environment itself.  The CoStar Framework represents one such approach to providing “automation for automation”.