- A sample is only useful if it was drawn at random from the full set, not hand-picked.
- Check duplication and overlap with public data first, because they reduce value fastest.
- Scan for personal information that de-identification missed before the sample goes further.
- The best test is a small training or evaluation run that shows whether the data moves your metrics.
Checks to run
- Representativeness. Ask how the sample was drawn. Random sampling across the full set is the standard.
- Duplication. Exact and near-duplicate rates within the sample.
- Overlap. How much already appears in public crawls or datasets you hold.
- Quality. Language detection, encoding, resolution, audio clarity, truncation.
- Labels. Accuracy against a hand-checked subset; agreement rates for human labels.
- Personal information. Automated scans plus manual review for missed identifiers.
- Machine-generated content. Screen for AI-generated text, which adds little.
- Model impact. A small fine-tune or evaluation run comparing with and without the data.
For sellers
Prepare a random sample, document how it was drawn, and run your own personal information scan before sharing. A sample that turns out unrepresentative will cost trust in the full deal.
Questions
How big should a sample be?
Big enough to measure the checks you care about, typically hundreds to a few thousand records, or a few hours of audio or video.