How the work is done.
Every label, transcript, translation and judgement in our data is made by a person and checked by a person. This page sets out how, step by step.
Each rule below is enforced by the platform itself, most of them in the database, so they hold whoever is working and however the system is reached.
Four commitments behind every dataset.
Made by people
Every label, transcript, translation and judgement is produced by a trained person. No model writes answers on this platform.
Independent
Each person answers without seeing anyone else’s answer, so agreement between them measures something real.
On the record
A submission can never be edited afterwards. Decisions about it are recorded beside it, by name.
Checked
Work is reviewed by appointed experts, sent back when it falls short, and measured against known answers.
From applicant to delivered dataset.
- 01
Recruit and verify
Applications on the contributor and professional calls are read by a person. Professional background is checked by an admin and recorded separately from the profile, so nobody can mark themselves verified.
- Applications reviewed individually
- Qualifications and licences verified by staff
- Only approved members can train or work
- 02
Train and assess
A structured programme of 4 core levels and 51 lessons, free to every approved member. Each lesson ends in an assessment graded by the platform against answer keys no learner can read: a score cannot be claimed, only earned.
- 90% to pass every lesson, unlimited retakes
- Levels open in order, after the one before
- Badges and certificates verifiable by link or QR code
- 03
Match people to work
Rather than picking familiar names, every eligible person is scored for each piece of work, and the parts of the score are shown so any choice can be explained.
- Language 40, quality 25, reliability 15
- Experience 10, fit 10
- Quality means accuracy on known answers and agreement with others
- 04
Work independently
Work goes only to the named people chosen for it. Anyone not chosen cannot see that it exists. Each person answers on their own, with source material that stays on the platform.
- Assignment-only: no open pool
- No one sees another person’s answer
- Task material cannot be downloaded
- 05
Review and rework
Submissions are write-once; reviews are kept beside them. A reviewer approves, sends back with a note, or rejects. Rework is a new attempt, and both attempts are kept, which shows who improved and who guessed again.
- Reviewers appointed by name, with a record of who appointed them
- Every decision traceable to what was judged
- A disputed decision can be shown exactly as it was made
- 06
Measure quality
Quality is measured, not assumed: accuracy on items with known answers, and agreement between independent annotators, including how far boxes, outlines and spans overlap. Automatic checks are arithmetic, such as audio judged faster than it plays, and they only raise a flag for a person to read.
- Known-answer (gold) items
- Agreement scores, including overlap for shapes
- Flags for a person, never automatic penalties
Who can see what, and why.
Least access
Submitted work is visible only to admin, operations and reviewers appointed by name. The rules live in the database, not only on the pages, so they hold for any way in.
Material stays put
Recordings, images and documents are shown inside the task and are not offered for download.
Answers are final
Once submitted, an answer belongs to the dataset. It cannot be edited or withdrawn later, which is what makes the record trustworthy.
Versioned delivery
Datasets are cut as numbered versions, so what a client received can always be reproduced.
Fair to the people doing the work.
- Training, certificates and badges are free, and the standard is the same for everyone
- Unlimited retakes: a lesson can be attempted again until it is passed
- Members see their own lesson marks, and work sent back comes with the reviewer’s note
- Credentials belong to the member and can be shared and verified by anyone
- Terms for commissioned work are agreed with the member before it begins
Personal information is handled as set out in our privacy notice, and the platform’s protections are described on Security and trust.
Questions about the method?
We are glad to walk funders and research partners through any part of it, including the platform itself.