- Published Sep 28, 2026
- 6 min read
Five things to write down before collecting data
Before public pages become a table, write down whether the source is allowed, what the fields look like, where failures are logged, where the data lives, and how someone else re-runs it. Then you can tell whether a collection job can be handed over — and whether we can take it.
1. Is this source allowed to be collected?
This is for people about to collect public data. By the end you have a start list: sources, fields, failure records, where the data lives, and how acceptance works. Miss any one and the handoff is a script that ran once on someone else's machine. VAV Agent's data collection delivery is those five items as a re-runnable program and a runbook.
Before starting, write the source list and note the basis for each: the page is public, or the client has authorization. Targets requiring logins, captchas, or bypassing access controls are out of scope. We never treat 'technically scrapeable' as 'deliverable'.
Personal sensitive data is excluded by default. If the business truly requires it, confirm authorization and retention first, then decide whether to start.
2. Fields are defined by samples, not by verbal agreement
Lock the field dictionary with a small batch of real pages: field name, meaning, example values, and when nulls are allowed. Expand collection only after the samples are confirmed. Otherwise the script reveals 'this date is sometimes relative' at record two thousand.
3. Failures leave records — they don't vanish quietly
Write down update frequency, retry counts, and which log failed sources appear in. At acceptance, a missing source should be visible. A script that swallows failures hands over a full-looking table that cannot be audited.
4. Where the data lives, and for how long
Agree on storage before delivery: the client's repository, drive, or database. Production data doesn't stay with the team by default; if retention is needed for maintenance, its scope and duration are written down.
5. Acceptance means reproduction, not 'it ran once'
At handover the client should be able to re-run everything by following the runbook. Three checks: fields match the samples, agreed sources all have records, and no failures are hidden. Miss any of the three and it's still just a successful demo, not a delivery.
How we can help with this
If you already have public pages but no field dictionary and no way to re-run the job, send two or three sample links and the fields you need. We first decide whether the source is in scope: public or already authorized can start; anything that needs a login, a captcha, or a bypass of access controls cannot.
When it can, the delivery is a source list, a field dictionary, a scheduled job, a failure log, and a runbook. Acceptance means your colleague can re-run it from the runbook, not that it succeeded once on our machine.