Today, technology organizations typically have people with two distinct skill sets: Software Engineers and DevOps Engineers. Further, some may have DevOps and compliance functions managed within the same or separate teams. In startups and smaller companies, there may just be the same engineers wearing all three hats.
Software engineers come up with the high level application architecture. The business provides compliance requirements. These two are passed on to the DevOps team who use their subject matter expertise to realize what needs to be done for the cloud infrastructure. There are other elements of operations in scope, such as CI/CD and diagnostics that include central logging, monitoring and alerting.
The application engineers start off by giving a set of requirements to the operations or DevOps team. This typically includes:
High level architecture. Like the AWS example shown in the figure below which depicts the following: - A set of docker containers to be deployed connected to a SQL database along with a Redis instance and an S3 bucket. - Part of the containers needs to be behind a public ELB, part behind an internal LB. - Data science team may want a Spark cluster connected to ES - Lambda functions behind API gateway are to be deployed. One could draw similar examples for other cloud providers
2. Multiple environments might be required: Dev, Stage, QA and Production. In some case there may be a need to deploy a unique copy of the application for each customer (Single Tenant Application).
3. Diagnostics. Central logging, monitoring and alerting must be established.
4. Compliance standards. Specific standards are to be met like PCI, HIPAA, SOC 2 etc.
5. CI/CD is to be established.
For organizations operating in regulated industries, the infrastructure needs to follow strict compliance guidelines. Some are stricter than others which is typically measured by the number of compliance controls that need to be satisfied. NIST, PCI, HITRUST, and SOC 2 are examples of such compliance standards. It could take companies 6 months to a year to make a 50-node infrastructure compliant with these standards.
The AWS PCI guide is 3400 pages long! Operational Best Practices for PCI DSS 3.2.1 - AWS Config (amazon.com) Even if one were to scope it to 20 commonly used services, the control set is overwhelming!
The high level application and compliance requirements are passed onto a DevOps team that is the subject matter expert for the Cloud. This team would accept the requirements and translate them into hundreds or thousands of lower level configurations, best practices and compliance controls. These include IAM Roles, Instance profiles, KMS Keys, PEM key, vulnerability scanning system, virus scanners, VPC, Security Groups, Intrusion detection, etc. This translation is usually done based on human knowledge and subject matter expertise in the area. Further, DevOps engineers are usually required to write thousands of lines of code to implement these requirements using programming languages like Terraform, Python and Bash.
A common misconception is that Terraform automates the DevOps workflow. In fact, Terraform is only a programming language. One needs substantial infrastructure know-how to build automation using Terraform. Typically, DevOps engineers are not aware of compliance nuances that go beyond best practices and have to redo a lot of the work on an ongoing basis.
DevOps essentially is a skill that requires one to be a programmer (to write IaC), an operator, and a compliance expert. These are three distinct skill sets that have never traditionally co-existed in the IT industry. This is the #1 challenge in the DevOps space.