It has been a few years since automation was introduced into the world of security Compliance. Automation promised to do away with unreliable and time-consuming manual collection of screenshots and replace it with automated Compliance evidence collection. And in many ways, that promise has been delivered, with large numbers of companies onboarding one of several tools on the market (such as the one offered by Anecdotes). However, with all of the (justified!) excitement over the newly introduced automation, many companies have overlooked the details of exactly what evidence these tools will automatically be providing.
In this blog, we will dive into the three main types of evidence created by Compliance automation tools, and discuss their main advantages and disadvantages. Our goal is to cut through the noise and confusion so that when you look for an automation tool, you’ll know what to expect.
Test-Based Evidence:
This first type of evidence is pretty straightforward. The tools that provide you with test-based evidence are prescriptive, in other words, they define exactly what needs to be checked and then they run simple pass/fail tests. If what needs to be in place is indeed there, the tool will tell you that you have passed the test. If it is not, the tool will tell you that you have failed. The test result saying you “passed” is the evidence you present to the auditor.
Let’s take an example. Say, you need to check that all buckets are encrypted. If they are, it will tell you that you passed. Great, you can move on. If they are not, you will have failed the test and you will need to go back to fix the problem. So far it seems pretty easy, right? Well, it is, and that is the main advantage of this type of evidence, but it is also where we hit the first bump in the “test-based evidence” road. Since this is a simple pass/fail test, all you get is the result. Meaning if you fail, you don’t know where your mistake is. This will, most likely, require you to open a ticket for the infrastructure team, asking them to determine which buckets are not encrypted, (since you do not have the data yourself) and remedy the situation so that the next time you run the test, you can pass. (Let’s not forget that going back to the relevant system owners and manually sifting through the data for mistakes is exactly what you wanted to avoid by onboarding an automation tool…)
Now, say that the infrastructure team informs you that there are indeed unencrypted buckets, but you realize that they are out of scope for this specific audit. In other words - you got a false negative. What now? Frustration aside, how do you fix the error if it is all prescriptive? You can’t give the auditor the “fail” as evidence. Unfortunately, there is no straightforward solution for this.
When you are a smaller company, using the tests defined by the tools may be good enough. But once your organization becomes a little more mature, you will need to configure some of the tests. For example, your organization may require all passwords to meet strict criteria – they must be at least 12 characters, and require upper and lower case letters, a special character, and a number. The test, however, is only able to check whether the password is more than eight characters in length since that is the control requirement. Therefore, although the password may “pass” the test and be fit for an audit, in terms of your own defined policies, this is a false positive.
The last thing to note about this kind of evidence is whether all auditors will accept it even if you get a “pass.” Will they trust the test results? Remember those false negatives, what about false positives? Will they be willing to sign off on that?
To Summarize:
- Test-based evidence IPE: The date and time the test was conducted by the vendor as well as the pass/fail result, and the API call used.
- Biggest advantage: Very easy and quick automated Compliance evidence. Prescriptive, test-based evidence means you don’t need prior Compliance knowledge or expertise and can pass that audit you so desperately need in order to sell quickly.
- Biggest disadvantage: If your tech stack or Compliance program is even a little complicated, the pass/ fail tests will haunt you in your dreams.
{{banner-image}}
Test-Based Evidence 2.0:
Many of the problems with test-based evidence arise because no context is provided. The next generation of test-based evidence learned from its predecessor's mistakes and provides the data the test is based on in a PDF file. So now you can show the auditor that you passed the test, and provide the data your “grade” is based on. Remember the trouble we had with finding the cause of our failing grade with test-based evidence 1.0? Eliminated as well. You can look through the data, find what needs to be fixed and fix it. Sounds like the perfect solution, no? Ummm… not quite.
While a PDF is great and all, it can’t be filtered or edited. Yes, we know you can technically edit a PDF, but as it is a standalone document, even if you can edit it, it won’t impact the result of your test. “Well that’s good, isn’t it? We don’t want people falsifying evidence, right?” you may ask. Of course, we don’t. When we say edit, we mean work with the data.
The data in the PDF is static and therefore can’t be customized to meet your needs. As opposed to working with live data which you can slice and dice and choose to present only relevant fields, with a PDF, you are limited in that you can only present the entire data set. For example, when out-of-scope data appears in the PDF, it is not only confusing, it also impacts the results of the test. With live data, you can scope that data out and the result will change accordingly.
To Summarize:
- Test-based evidence 2.0 IPE: The date and time the test and data collection was conducted by the vendor, the API call used, and a PDF with the data that supports the test result.
- Biggest advantage: You get all of the advantages of the first generation, and the PDF solves many of its problems – both your organization and the auditor can see what data the test is based on.
- Biggest disadvantage: You are obligated to work with – and present to the auditor – a full data set that may not be truly reflective of the scope of your Compliance program.
Dataset Evidence:
The final form of evidence is automatically collected raw data. Not raw as in a JSON file. Think more of a JSON file that has been put into a table to form easy-to-use datasets. What, no tests? Nope. There is no automatic pass/fail in this form of evidence. Before you dismiss it out of hand, hear it out.
Instead of tests, you have rules. Rules that are predefined by the tool, or, and this is key, by you. Because this evidence is basically structured datasets, you can decide what rules you want to apply, and if something isn’t met, it will be flagged for you. Looking for 12-character passwords? Great, make a rule and see if any of the data points don’t meet the rule. In other words, YOU get to decide if the evidence passed or failed YOUR test.
You can work with the datasets in all kinds of ways; they are flexible. If some of the data isn’t in scope for the audit, you can just filter it out and avoid a false negative. You can analyze dataset evidence, search for specific information, review certain data points, and well, do anything really. Think of it like cooking your own meal rather than ordering a standard dish at a restaurant. If you get the ingredients yourself, you can ensure they are all fresh, you can decide what to put in and what to leave out. You can make sure that everything is cooked to your liking, not to the (prescriptive) chef’s.
Oh, and auditors can benefit from technology and continue to work in the same way that they’ve always worked. They can see the data in full (albeit in an easier-to-consume form), and draw their own conclusions.
To Summarize:
- Dataset evidence IPE: The date and time the test and data collection was conducted by the vendor, the API call used, the JSON file, the preview table with an item count for validation against the JSON, and a hashing mechanism (used for integrity validation).
- Biggest advantage: You get all of the advantages of automation with the flexibility to tailor the evidence to meet your specific requirements.
- Biggest disadvantage: There isn’t an automatic pass/fail, you have to do some of the work yourself.
Automation That Works For You
Depending on where you are in your journey toward Compliance maturity, each one of these types of Compliance evidence might be right for you. If you are new to Compliance or are a small company, test-based evidence from a prescriptive solution could be just what you are looking for. If you are managing a mature Compliance program, you might find that dataset evidence is the way forward. Don’t allow yourself to be blinded by the “idea” of automation, do your research and make sure you fully understand what type of automated Compliance evidence you can expect from each tool.
About the author
