Automating Compliance Evidence Collection

The two weeks before an assessment look the same everywhere. Somebody is taking screenshots. Somebody else is trying to remember whether the account review happened in March or April. A third person is exporting a report that proves the control is on today and says nothing about the eleven months before today.

That scramble is not a documentation failure. It is a data collection failure, and data collection is a solved problem. You already automate things far harder than this.

Why a screenshot from Tuesday is weak evidence

NIST SP 800-171A assesses each requirement by some combination of three methods: examine a document or artifact, interview the people responsible, and test that the mechanism actually operates. An assessor is generally looking for more than one of the three before calling something met.

Two words govern whether what you hand over is any good.

• Adequacy. The artifact addresses the objective. A policy saying accounts are reviewed quarterly does not demonstrate that accounts were reviewed quarterly.
• Sufficiency. There is enough of it, covering the whole scope. One screenshot of one server does not cover fourteen endpoints, and one capture from last Tuesday does not cover a control that is supposed to operate every quarter.

Sufficiency is where point-in-time collection dies. For any control that operates on a cycle — log review, account review, vulnerability scanning, backup testing, training — the evidence has to span the period. You cannot generate that retroactively. Either you were collecting it or you were not.

Which gives you the whole design rule in one sentence: collect at the cadence the control operates, not at the cadence of the assessment.

Three kinds of evidence, and only two are worth automating

Kind	Examples	Automate?
State — what the configuration is right now	Conditional access policies, firewall rules, group membership, encryption settings, patch levels, installed software	Yes. This is an API call on a schedule.
Event — records produced as things happen	Authentication logs, ticket history, scan results, backup job outcomes, training completions, alert dispositions	Yes. Export on a schedule and keep the exports.
Judgment — a human decided something	Access review sign-off, risk decisions, incident after-action notes, the SSP itself	No. But you can automate the prompt, the form, and the filing.

The third row is where people either give up or, worse, get clever. Do not automate judgment. Automate everything around it so the judgment takes four minutes and leaves a record.

The architecture, which is boring on purpose

Scheduled trigger, authenticated API call, normalise, write to a dated immutable location, index it, alert on failure. Five steps, and the same five for every source.

I run this on n8n because I already run n8n, and because a workflow engine gives you the retries, the credential store and the failure notifications without writing them. Cron and a shell script would work. A scheduled function in your cloud tenant would work. The engine is not the interesting decision.

The interesting decisions are these three:

• Pull from the API, not the console. A screenshot proves what a rendered page looked like. A JSON export from Microsoft Graph or the Google Admin SDK proves what the tenant actually returned, is diffable against last month, and takes no human time. Where a console is genuinely the only source, script the capture so at least it happens on schedule.
• Write once, never edit. Dated path, no overwrites. evidence/2026-09-28/3.5.3-mfa-policy-export.json. An evidence store where files get updated in place has destroyed the only thing that made it evidence.
• Hash on write. A SHA-256 alongside each artifact, and a manifest per collection run. It costs nothing and it answers the “how do we know this was not edited later” question before anyone asks it.

Metadata is what turns a folder into evidence

A directory of ten thousand JSON files is not better than the screenshot scramble; it is the same scramble with more files. Every artifact needs a small header record, and this is the part people leave out.

• Which requirement or requirements it supports — 3.5.3, 3.1.1
• Collection timestamp, in UTC, and the period it covers
• Source system and the exact API endpoint or command that produced it
• Which assets or population it covers, and whether that is the full in-scope set
• Who or what collected it — the workflow name and version
• The hash

Get that right and “show me evidence of multi-factor enforcement across the last year” becomes a query rather than an archaeology project. Get it wrong and you have automated the production of material nobody can find.

A worked example: the quarterly account review

This one is worth walking through because it touches all three evidence kinds and because it is the control small shops most often fail on paper while doing it perfectly well in practice.

• Day 1 of the quarter, automatically. Pull every account from the identity provider with last-sign-in date, group memberships, privileged role assignments and MFA registration state. Pull the current employee roster from HR or payroll. Diff the two.
• Generate the review packet. A short document listing accounts with no sign-in in 90 days, accounts not matching the roster, privileged accounts, and anything changed since last quarter. This is the artifact that makes the human step fast.
• Route it for the judgment step. A task to the responsible person with the packet attached and three possible dispositions per exception: keep, disable, investigate. Four minutes of actual work.
• Capture the decision. Their responses, with name and timestamp, written back into the evidence store next to the input data.
• Verify the follow-through. Thirty days later, pull the accounts again and confirm the ones marked “disable” are disabled. That closing loop is what turns a review into a control.
• Alert on silence. If the review is not completed within two weeks, that is a notification, not a quiet gap discovered eleven months later.

Four quarters of that is an unarguable answer to an assessor, and the total human cost is under half an hour a year.

Where to start, if you start today

Do not try to cover 110 requirements. Pick five where the evidence is both API-accessible and time-sensitive, get them running, and add one a week after that.

• Identity state — accounts, roles, MFA registration, conditional access policies. Weekly. Feeds a large block of the access control and identification families.
• Endpoint state — inventory, encryption status, patch level, endpoint protection status and definition age. Weekly.
• Vulnerability scan output — the raw result file, every scan, kept. Not the summary dashboard.
• Backup job results — success and failure records, plus the restore test when you run one.
• Training completions — the export from wherever training lives, monthly.

Five workflows, maybe two days of work, and they run for years. Compare that against two weeks of screenshots every assessment cycle, done under time pressure by people who have other jobs.

Four traps

The evidence store is in scope. A repository of configuration exports, asset inventories and network detail about your CUI enclave is exactly the material an attacker would want, and it may itself contain CUI. Put it inside the boundary, apply the same 110 requirements to it, and describe it in the SSP. Do not stand up a compliance evidence bucket in a random cloud account.

Silent failure is worse than no automation. A collection job that stops working in February and is noticed in November has produced a nine-month hole in exactly the record you were relying on. Every workflow needs a failure alert, and something should verify weekly that each expected artifact actually arrived. I have been bitten by a job that reported success while the underlying command did nothing at all — the exit code was never checked. Check the exit code.

Collection is not implementation. Automating the evidence for a control you have not implemented gets you a very well-organised record of non-compliance. Useful, honestly — but do not confuse the two, and do not let a green dashboard convince anyone the control is on.

Never let a model generate evidence. Use AI to index, summarise, cross-reference against 800-171A objectives, and draft the narrative that explains an artifact. Do not let it produce the artifact. An assessment package containing a plausible-looking log excerpt that no system ever emitted is not a compliance problem, it is a False Claims Act problem, and the distinction between summarising evidence and manufacturing it needs to be a bright line in whatever you build.

Keep it for six years

The CMMC rule requires retaining assessment-related documentation for six years from the assessment date. Size your storage for that from the start — it is trivially cheap for text and JSON, and it is annoying to retrofit a retention policy onto a store that has been quietly rotating things out.

It also means the evidence store outlives the tooling that fills it. Keep the artifacts in formats that will still open in 2032. JSON, CSV and PDF will. A proprietary dashboard’s internal database will not.

The reason to do this now

Third-party certification is suspended, so there is no assessment on the calendar to panic about. That is exactly why this is the right month to build it. The work is unglamorous, it takes a couple of days, and its entire value is that it has been running for a year by the time anyone asks.

And it pays before then. A self-assessment built on collected evidence produces an SPRS score you can actually defend, and a System Security Plan whose control descriptions point at something real rather than at an intention. That is worth more than the assessment it eventually supports.

---

Where to go next

• Not sure which controls you would even be collecting for? The CMMC readiness quiz takes about five minutes.
• Our CMMC guide for small contractors walks the 110 requirements in the order that makes sense to implement them.
• Want the evidence index and control-to-artifact mapping already built? That is part of our CMMC packets.