Ensure Compliance in Engineering to Help Avoid Errors

Lessons Learned from Halt and Catch Fire Scenarios

Halt and Catch Fire (HCF) refers to an early machine command that causes the CPU to “cease meaningful operation, typically requiring a restart,” according to Wikipedia. Created to poke fun at the many abbreviations for commands used at the time, the name became so popular that it eventually became officially recognized as a real instruction. Aside from delivering a clear visual to Engineering teams when there is no way for the system to recover without a restart, Halt and Catch Fire is also a suitable moniker for the relationship Security Compliance sometimes has with Engineering. 

In general, Engineering and related departments are aware of the need to ensure Compliance. Unfortunately, awareness does not always translate into understanding. When the reasons (the WHY) for Compliance are not clearly understood, Engineering may view Compliance-related tasks and practices as obstacles to completing their deliverables. Engineering tasks are often critical, and getting to the finish line quickly is necessary, but disregarding Security Compliance requirements can end in disaster.

Having been involved in the GRC world for decades, I have encountered several instances when Engineering would have done well to pay better attention to Security Compliance and not dismiss these activities as inconsequential. 

3 Stories of Engineering Errors That Could Have Been Avoided by Ensuring Compliance

Here are just a few stories of how Compliance helps a company minimize and avoid engineering pitfalls. 

Don’t Assume, Know for Sure

One multinational enterprise I worked with was looking to have a newly-developed product audited for SOX Compliance. I called a meeting with the DBA team to review the Engineering Compliance requirements. The team felt the meeting was unnecessary as they had significant database expertise and insisted everything was up to par. Not convinced, I asked them to pull the logs for a specific tech stack to provide evidence that database monitoring controls are in effect. The team pulled up a script to show me it was running. However, I noticed that the script was named “SOX Database Monitoring Script,” which made me suspicious. Further investigation revealed that the script last ran six months ago!  The team was very embarrassed. It turns out that they had turned off all monitoring scripts during a database upgrade and then slowly brought the scripts back online. However, the SOX script was of so little value to them that they never turned it back on. They simply did not understand the WHY, the context, or the risk. 

Avoid a One-Off Mindset

At one of my previous Compliance engagements, I encountered a common security vulnerability where a storage bucket was named something very simplistic and sequential, like 12345, 12346. This type of enumeration vulnerability (IDOR) is critical because hackers can easily uncover additional bucket names using the following sequential number. When the DevOps team received a report of this type of vulnerability, they viewed it as a one-off activity. Instead of the DevOps and Compliance teams working together to address the security issue, the DevOps team did not try to look for additional instances in their code where the same vulnerability reappeared. Each time a similar vulnerability was uncovered, a separate ticket was created, resulting in a longer cycle time and unnecessary effort expended on communication and tracking. Aside from the obvious need for a layer of obfuscation, Engineers should not treat vulnerabilities as one-offs; rather, they should build a new process or apply knowledge to the whole environment. 

Wall of Fame and Shame

At one fast-moving enterprise, Devs were pressured to get their code into production as soon as possible. Unfortunately, the teams engaged in a horrible Engineering process known as “ship it, then fix it,” which means they would throw the code into production and fix any issues on the go. I learned about this harmful practice when audit consultants tasked the platform storage team to perform a weekly validation of the changes that went into production. The DevOps team found thousands of changes every week, making it impossible for the team to validate the changes and ensure Compliance. I knew I had to reduce direct production changes fast. Instead of taking a mandate-based top-down approach, we built a monitoring dashboard to track changes in production and sent this to the managers and directors. This way, they could see for themselves the ‘Wall of Fame and Shame’ regarding the number of direct changes to production that had reduced product quality. 

Ensure Compliance, Communicate Effectively & Yield Benefits for Engineering Teams

To paraphrase a quote from the TV series Halt and Catch Fire about the early days of PCs and the Internet, “Security Compliance isn’t the thing; they’re the thing that gets you to the thing.” An excellent GRC team and Engineer relationship is crucial in ensuring a company’s security is up to par. It is imperative that Engineering understand that Security Compliance is not the enemy or the barrier to getting their work done or the reason they need to log in on a Saturday afternoon. They need to understand that as they shift left, Security Compliance in Engineering is there to get them to “the thing” – the security of the company’s network, systems, and products. When Engineering ensures Compliance, they are ensuring that no breaches occur on their watch.