Data provenance explained

What data provenance means for AI training data, which records buyers need, and how provenance supports training data disclosure under AB 2013 and the EU AI Act.

By Gareth HoyleUpdated For both
In short
  • Data provenance is the documented history of a dataset: who created it, how it was collected, under what terms, and what was done to it.
  • Buyers need provenance to defend their use of data and to meet training data disclosure rules.
  • A dataset card or manifest records provenance in a format both sides can check.
  • Researchers auditing public AI datasets have found widespread gaps in license and source information.

What provenance covers

  • Origin: the systems and people that created the data
  • Collection: when, how and under what notice or consent
  • Rights: ownership and license basis for each part
  • Processing: cleaning, de-identification, filtering and labeling steps
  • Chain of custody: who has held copies since

Why it matters now

Training data transparency rules require developers to describe their datasets. California AB 2013 requires disclosure of training data sources, including whether data was purchased or licensed (LegiScan). The EU AI Act sets transparency duties for general-purpose AI models (European Commission). The Data Provenance Initiative has audited licenses and sources of widely used AI datasets (Data Provenance Initiative).

How to document it

The "datasheets for datasets" approach proposes that every dataset ship with documentation of motivation, composition, collection, processing, uses and distribution (Gebru et al.). A data manifest with per-record rights flags is the practical version for licensing. See how to build a data manifest.

Questions

Who is responsible for provenance, buyer or seller?

Both. The seller documents origin and rights; the buyer keeps records of what they acquired and how it was used.

Sources

  1. LegiScan: California AB 2013, Generative artificial intelligence training data transparency
  2. European Commission: AI Act regulatory framework
  3. Data Provenance Initiative
  4. Gebru et al.: Datasheets for Datasets (Communications of the ACM)

Read next

Tell us what you need

Book a free 30-minute consultation. Tell us what data you want to buy or license, and we'll tell you honestly whether we can help and what it involves.

  • We reply to every enquiry within two working days.
  • No upfront fees for sellers
  • We never hold your raw data