AI Data Readiness Score: Label Noise Hurts 5x More Than Missing Data
DOI: 10.5281/zenodo.24367405[1] · View on Zenodo (CERN)
| Badge | Metric | Value | Status | Description |
|---|---|---|---|---|
| [s] | Reviewed Sources | 50% | ○ | ≥80% from editorially reviewed sources |
| [t] | Trusted | 100% | ✓ | ≥80% from verified, high-quality sources |
| [a] | DOI | 86% | ✓ | ≥80% have a Digital Object Identifier |
| [b] | CrossRef | 50% | ○ | ≥80% indexed in CrossRef |
| [i] | Indexed | 75% | ○ | ≥80% have metadata indexed |
| [l] | Academic | 89% | ✓ | ≥80% from journals/conferences/preprints |
| [f] | Free Access | 93% | ✓ | ≥80% are freely accessible |
| [r] | References | 28 refs | ✓ | Minimum 10 references required |
| [w] | Words [REQ] | 4,885 | ✓ | Minimum 1,000 words of article body. Current: 4,885 |
| [u] | Original [REQ] | 100% | ✓ | ≥85% unique sentences (no padding or repeated blocks) and no leaked drafting notes. Current: 100% unique |
| [e] | Evidence [REQ] | 23 | ✓ | Cites at least 1 external source(s): links, DOIs or a reference list. Current: 23 |
| [d] | DOI [REQ] | ✓ | ✓ | Zenodo DOI registered for persistent citation. DOI: 10.5281/zenodo.24367405 |
| [o] | ORCID | ✓ | ✓ | Author ORCID verified for academic identity |
| [p] | Peer Reviewed | — | ○ | Peer reviewed by an assigned reviewer |
| [h] | Freshness | 71% | ✓ | ≥60% of references from 2025–2026. Current: 71% |
| [c] | Data Charts | 6 | ✓ | Original data charts from reproducible analysis (min 2). Current: 6 |
| [g] | Code | ✓ | ✓ | Source code available on GitHub |
| [m] | Diagrams | 3 | ✓ | Mermaid architecture/flow diagrams. Current: 3 |
| [x] | Cited by | 0 | ○ | Referenced by 0 other hub article(s) |
Abstract #
Most AI data readiness assessments are checklists: each data problem gets a tick box, and every box counts the same. We tested whether that weighting matches what actually happens to a model. We injected six training-data defects at five graded severities into 10 public tabular datasets: missing values, label noise, duplicate rows, class imbalance, target leakage and reduced sample size. We then trained logistic regression, random forest and gradient-boosted trees with three seeds each, 5,490 model fits in total. Every model was scored on an untouched clean test set and on the team’s own validation split. At the same 20% dose, label noise removed 12.9% of the above-chance balanced accuracy, against 2.5% for missing values, about five times as much. Duplicates did no measurable harm to deployment performance (-0.3%). They did inflate the random forest’s self-reported balanced accuracy by 9.9 percentage points at a 40% duplicate rate. A leaked proxy feature made logistic regression report 97.7% balanced accuracy while it delivered 58.6%. From the single-defect curves we built an empirically weighted AI Data Readiness Score. We tested it on 300 random defect combinations, each on a dataset whose curves were left out of the fitting. Its Spearman correlation with actual loss was 0.839, against 0.658 for an equal-weight checklist (bootstrap 95% CI of the difference 0.124 to 0.242) and 0.471 for a count of defects present.
1. Introduction #
The series’ friction taxonomy found that data quality is the most-cited barrier between AI capability and deployment, named by 52% of organisations in the surveys it pooled [1][2]. That finding raises a practical question. If data quality blocks adoption, which data problems should a team fix first? Field interviews show that data problems compound quietly. Sambasivan and colleagues interviewed 53 practitioners and reported a 92% prevalence of “data cascades”, compounding downstream failures caused by data issues [2][3].
Organisations answer the question with an AI readiness assessment. The hub’s own AI Data Readiness Index Assessment is a typical example [3][4]. It scores five dimensions (data quality, infrastructure, governance, technical and organisational) with five questions each and sums the 25 answers into a percentage that maps to a readiness level from 1 to 5. Like most instruments of this kind, it weights every question equally. That is a reasonable default when nothing is known about the relative damage each problem does. It is no longer reasonable once the damage can be measured.
This article measures that damage. We take six defects that appear in almost every data readiness checklist, inject each one at controlled severity into real public datasets, and record what happens to model performance on clean deployment data. We then turn the measured damage into weights for a data readiness score and test that score on datasets it has never seen. This is the base version of a study we plan to extend (Section 5.4), so the design favours reproducibility over scale. Everything runs on six CPU cores in 179.5 s.
RQ1: At a matched 20% dose, which of six training-data defects cause the largest loss of performance on clean deployment data, and how does that loss grow with severity? RQ2: Which defects leave deployment performance intact but distort the performance a team reports from its own validation split, and by how much? RQ3: Does an empirically weighted AI Data Readiness Score predict the performance loss of unseen datasets with random defect combinations better than an equal-weight checklist?
2. Existing Approaches: what an AI data readiness assessment measures, and why checklists fail #
2.1 Readiness levels and readiness reports #
The idea of grading data before modelling goes back to Lawrence’s Data Readiness Levels. This position paper proposed three stages of data preparedness as a common language for project management, because overruns often come from underestimating curation work [4][5]. Afzal and colleagues turned the idea into a document, the Data Readiness Report. It records quality issues and every transformation applied to a dataset, and is meant to sit alongside datasheets and model cards [5][6].
Work from 2025 and 2026 has formalised and automated the idea. A 2025 ACM Computing Surveys review of more than 140 papers proposes a taxonomy of data readiness for AI (DRAI) metrics for structured and unstructured data. It also concludes that standardised metrics are “still evolving” [6][7]. The same group’s AIDRIN 2.0 tool scores quality, bias, fairness and privacy through a user interface [7][8]. CADRE lets users define their own readiness metrics, rules and remedies for federated learning, and was tested on six datasets and seven readiness issues [8][9]. For scientific foundation models, Brewer and colleagues pair Data Readiness Levels (raw to AI-ready) with data processing stages in a maturity matrix [9][10]. In automated driving, a 2026 study condenses 46 image metrics and 2 point-cloud metrics, computed over nearly 5 TB of data, into a single readiness level from 1 to 9 [10][11].
These tools are good at measuring defects: how many values are missing, how imbalanced the classes are, how many rows repeat. None of them reports how much each defect costs a model. When the measures are rolled up into a single score, the weights come from convention or from the user.
2.2 Documentation standards #
Datasheets for Datasets made documentation the main route to data accountability. Each dataset is accompanied by a structured account of its motivation, composition, collection and recommended uses [11][12]. A 2025 analysis of 4,041 scientific data papers checked how far published documentation meets the needs of the ML community and regulators. It compared them with NeurIPS Datasets and Benchmarks papers and proposed guidelines to close the gaps [12][13]. Documentation tells a reader that a defect exists. It does not say whether the defect matters.
2.3 Defect-by-defect evidence #
The closest prior empirical work is by Mohammed and colleagues. They polluted training data, test data or both along six data quality dimensions and measured 19 algorithms on classification, regression and clustering [13][14]. Their focus is explaining algorithm behaviour, not building a readiness score. They also do not separate reported from deployed performance, which turns out to matter for two of our defects.
Label noise has its own literature. Confident learning estimates the joint distribution of noisy and true labels and finds label errors without knowing them in advance [14][15]. The same method found at least 3.3% label errors on average across the test sets of 10 widely used benchmarks [15][16]. A 2025 benchmark breaks label-error detectors down by components and recommends in-sample gathering with a logit-margin disagreement measure [16][17]. Label noise also degrades the quality and utility of synthetic tabular data generated from it [17][18].
Leakage and contamination are mostly studied as evaluation problems. Kapoor and Narayanan found leakage in 17 scientific fields, affecting 294 papers, and proposed a taxonomy of eight leakage types [18][19]. A 2026 controlled benchmark measured the ROC-AUC inflation caused by five leakage mechanisms. Duplicate contamination added 1.52 points (95% CI 0.40 to 2.80), while global scaling and global median imputation had near-zero effects [19][20]. A leakage-aware evaluation framework published in 2026 condenses such risks into a Leakage Risk Score [20][21]. Train-test overlap is now a central concern in large language model evaluation as well [21][22].
Class imbalance has benchmark suites. CLIMB covers 73 datasets and 29 algorithms and reports that naive rebalancing has clear limits [22][23]. TILBench tests more than 40 methods on 57 datasets and finds that no single method dominates [23][24].
flowchart TD
A[AI data readiness assessment] --> B[Readiness levels and reports]
A --> C[Documentation: datasheets]
A --> D[Defect-specific research]
B --> B1[Measure defects, weight by convention]
C --> C1[Record defects, no weights]
D --> D1[Label noise, leakage, imbalance studied one at a time]
B1 --> G[Gap: no measured cost per defect]
C1 --> G
D1 --> G
G --> S[Empirically weighted readiness score]
The gap is the step between these strands. Readiness tools measure defects, and defect studies measure damage one defect at a time. No one has calibrated a readiness score against measured damage and then checked whether the score predicts loss when several defects occur together.
3. Method and Data: controlled defect injection #
3.1 Datasets #
We used 10 public binary classification datasets: two built into scikit-learn and eight from OpenML. Large datasets were subsampled once, deterministically, to 4,000 rows. That keeps the full experiment within a few CPU minutes and avoids large downloads.
| Dataset | Source | Rows | Features (categorical) | Positive rate | Clean BA, LR / RF / HGB |
|---|---|---|---|---|---|
| breast-cancer | scikit-learn | 569 | 30 (0) | 0.373 | 0.972 / 0.956 / 0.964 |
| california-housing | scikit-learn, above-median value | 4,000 | 8 (0) | 0.492 | 0.825 / 0.844 / 0.871 |
| adult | OpenML 1590 | 4,000 | 14 (8) | 0.242 | 0.768 / 0.781 / 0.783 |
| credit-g | OpenML 31 | 1,000 | 20 (13) | 0.300 | 0.689 / 0.641 / 0.685 |
| bank-marketing | OpenML 1461 | 4,000 | 16 (9) | 0.119 | 0.665 / 0.637 / 0.679 |
| spambase | OpenML 44 | 4,000 | 57 (0) | 0.391 | 0.918 / 0.938 / 0.943 |
| diabetes | OpenML 37 | 768 | 8 (0) | 0.349 | 0.725 / 0.721 / 0.733 |
| phoneme | OpenML 1489 | 4,000 | 5 (0) | 0.294 | 0.655 / 0.862 / 0.844 |
| magic-telescope | OpenML 1120 | 4,000 | 10 (0) | 0.363 | 0.748 / 0.843 / 0.837 |
| blood-transfusion | OpenML 1464 | 748 | 4 (0) | 0.238 | 0.526 / 0.615 / 0.628 |
Source: our analysis; clean balanced accuracy (BA) is the mean over three seeds on the untouched test set. LR = logistic regression, RF = random forest, HGB = HistGradientBoosting.
3.2 Defects and severities #
For each seed, 30% of every dataset is set aside as a clean test set and never modified. It stands in for deployment data. All defects are injected into the remaining 70%, each at five severities:
- Missing values (MCAR): 5%, 10%, 20%, 30% or 40% of feature cells are set to missing completely at random. The preprocessing step imputes them (median for numeric, most frequent for categorical).
- Label noise: 5% to 40% of training labels are flipped at random.
- Duplicates: copies of 5% to 40% of the rows are appended before the team’s own train/validation split, so copies can land on both sides. This is how duplicates usually cause contamination in practice.
- Class imbalance: 20%, 40%, 60%, 80% or 90% of the minority-class rows are removed.
- Target leakage: an extra column equals the recorded label on a share q of rows (20%, 40%, 60%, 80% or 95%) and is a random draw on the rest. It mimics a field written after the outcome, such as a closure code. At deployment the outcome is not yet known, so on the clean test set the column is a random draw with the same base rate.
- Reduced sample size: 20%, 50%, 75%, 90% or 98% of the training rows are removed, stratified by class.
The severity scales differ by defect because each defect has its own natural unit. All six grids include 20%, which gives a matched dose for comparison.
3.3 Models, estimates and the loss metric #
Each condition trains three models behind standard scikit-learn preprocessing (imputation, scaling, one-hot encoding). The models are L2-regularised logistic regression, a random forest with 100 trees and HistGradientBoosting with 100 iterations, each with seeds 0, 1 and 2. The data left after injection is split 80/20 into a training part and the team’s own validation part. We record two numbers:
- Reported: balanced accuracy on the team’s validation split, which carries the defect. This is the number a team would put in its report.
- True: balanced accuracy on the clean test set, which is what deployment delivers.
Datasets differ in how far above chance (0.5) a model can get, so we express damage as the share of above-chance balanced accuracy lost: 100 × Σ[BA(clean) − BA(defect)] / Σ[BA(clean) − 0.5]. Runs are paired by dataset, model and seed. We pool over runs as a ratio of sums, not a mean of ratios. The reason is logistic regression on blood-transfusion, which reaches only 0.526 clean balanced accuracy. Dividing each run by its own headroom would let small differences there swamp everything else. We also report the plain drop in balanced-accuracy points and in ROC-AUC points.
3.4 Score construction and validation #
The AI Data Readiness Score is
Readiness = 100 − Σ P(d, s), floored at 0,
where s is the measured severity of defect d and the sum runs over the six defects. P(d, s) is the mean single-defect loss curve of defect d, interpolated linearly between grid points, with negative values set to 0. A penalty point therefore means one expected percent of above-chance performance lost.
To test the score, we drew 30 random defect combinations per dataset, 300 in total. In each combination every defect is present with probability 0.5 and has a uniform random severity. Each combination is applied in full and trained with all three models and three seeds. The penalty curves used to predict a dataset’s combinations are refitted on the other nine datasets (leave-one-dataset-out), so the score never sees the dataset it is scoring. We compare it with two baselines:
- an equal-weight graded checklist, in which each defect contributes (severity / maximum severity) × 100/6;
- a count of defects present, the tick-box version.
Our main statistic is Spearman’s rank correlation with actual loss. We also report a 2,000-sample bootstrap of the difference between the score and each baseline.
flowchart LR
D[10 public datasets] --> T[30 percent clean test set]
D --> P[70 percent pool]
P --> I[Inject one defect or a random combination]
I --> S[Team split 80/20]
S --> M[LR, RF, HGB x 3 seeds]
M --> R[Reported BA on team split]
M --> U[True BA on clean test]
U --> C[Single-defect loss curves]
C --> W[Readiness score weights]
W --> V[Leave-one-dataset-out test on 300 combinations]
The full run covers 1,830 conditions and 5,490 model fits and takes 179.5 s with six parallel workers. Code, data, results and charts: stabilarity/hub/research/ai-data-readiness-score.
4. Results #
4.1 RQ1: Which defects actually break models #
Chart 1 shows loss against severity for each defect and model. Chart 2 compares all six defects at the same 20% dose.


| Defect | Loss at 20% (% of above-chance BA) | Loss at max severity | BA drop at max (points) | ROC-AUC drop at max (points) | Most damaging at 20% (cells of 30) | Cells with loss above 5% at 20% |
|---|---|---|---|---|---|---|
| Label noise | 12.9 | 51.5 (40%) | 14.2 | 19.7 | 20 | 26 |
| Class imbalance | 7.6 | 56.8 (90%) | 15.7 | 4.7 | 9 | 17 |
| Missing values (MCAR) | 2.5 | 3.9 (40%) | 1.1 | 1.4 | 1 | 8 |
| Reduced sample size | 2.3 | 45.9 (98%) | 12.7 | 14.2 | 0 | 6 |
| Target leakage | 2.1 | 65.6 (95%) | 18.1 | 14.6 | 0 | 6 |
| Duplicates | -0.3 | 0.5 (40%) | 0.1 | 0.4 | 0 | 4 |
Source: our analysis, results.json. A cell is one dataset and model pooled over three seeds (10 datasets × 3 models = 30 cells).
Label noise is the defect that breaks models. Flipping 20% of labels removed 12.9% of above-chance performance, about five times the 2.5% lost to 20% missing values. It was the most damaging defect in 20 of the 30 dataset-model cells and cost more than 5% in 26 of them. The curve is convex: 2.9% at 5% noise, 5.0% at 10%, 26.3% at 30% and 51.5% at 40%. The flexible learners suffer most. At 20% noise HistGradientBoosting lost 17.8% and logistic regression 9.5%, because boosting fits the flipped labels it is given. Label noise also hurts ranking, not just the decision threshold: at 40% noise ROC-AUC fell by 19.7 points.
Class imbalance is mostly a threshold problem. Removing 20% of the minority class cost 7.6%, and removing 90% cost 56.8%. ROC-AUC tells a different story. At 90% removal balanced accuracy fell 15.7 points but ROC-AUC only 4.7. The model still ranks cases well, and the default 0.5 threshold is what fails. The defect is real at the deployed threshold, but recalibrating or moving the threshold recovers most of it. That is consistent with the benchmark finding that rebalancing methods help unevenly [22][23][23][24].
Leakage and sample size are threshold defects. Both are harmless at low doses (2.1% and 2.3% at 20%) and severe at high ones. Leakage cost 20.1% at q = 60%, 40.5% at 80% and 65.6% at 95%. Cutting training data cost at most 7.0% until three quarters of it was gone (379 training rows on average). It then cost 21.0% at 152 rows and 45.9% at about 30 rows.
Missing values and duplicates barely matter for deployment. With standard imputation, even 40% MCAR missingness cost only 3.9% (1.1 balanced-accuracy points), and logistic regression lost nothing (-1.1%). Duplicates cost nothing at any rate we tested (-0.3% at 20%). Copies add no new information, and the clean test set contains none of them.
Chart 4 shows how much these averages hide. At 20% label noise the loss ranges from 8.1% (bank-marketing) to 26.1% (credit-g). Imbalance costs 0.7% on breast-cancer and 37.4% on blood-transfusion. Small datasets with weak signal (credit-g, diabetes, blood-transfusion) are hurt most by label noise, imbalance and sample size.

4.2 RQ2: Defects that corrupt the readiness assessment itself #
Two of the defects that do little harm on clean data do a lot of harm to the evaluation. Chart 3 compares what the team’s own validation split reports with what the clean test set measures.

Duplicates inflate tree models. At a 20% duplicate rate, 26.8% of the team’s validation rows have an exact copy in its training rows. At 40% the share is 45.3%. The random forest memorises those copies. Its reported balanced accuracy rose from 79.9% at a 5% duplicate rate to 88.0% at 40%, while its true performance stayed between 78.1% and 78.9%. The gap was 4.3 points at 20% duplicates and 9.9 points at 40%. For HistGradientBoosting it was 3.2 and 7.7 points. Logistic regression, which cannot memorise individual rows, stayed within 1.1 points. This is a larger effect than the 1.52-point ROC-AUC inflation reported for duplicate contamination in a recent controlled benchmark [19][20]. Our duplicate rates are higher and our metric is thresholded, so the two numbers are not directly comparable.
Leakage is the most dangerous defect for decisions. At q = 95%, logistic regression reported 97.7% balanced accuracy on its own validation split and delivered 58.6% on clean data, an overstatement of 39.1 points. The random forest and boosting models overstated by 37.6 and 37.5 points. Even at q = 60%, the overstatement was 12.0 to 16.0 points across the three models. A team that relies on its own split would ship a nearly useless model with the best score it has ever measured. This is the failure pattern Kapoor and Narayanan documented across hundreds of published studies [18][19].
Label noise does the opposite. Noisy validation labels make a model look worse than it is. At 20% noise, logistic regression reported 62.0% but delivered 72.6%, an understatement of 10.5 points. The random forest and boosting models understated by 11.4 and 10.4 points. A team could reject a usable model, or underestimate how much cleaning its labels would help. Auditing a sample of validation labels by hand fixes the estimate before it fixes the model [15][16].
An AI readiness assessment therefore needs two sub-scores. One predicts the loss at deployment. The other flags how far the team’s own numbers can be trusted. In the combination experiments, adding up the single-defect inflation curves predicted the reported-minus-true gap with Spearman 0.828. Actual gaps ranged from -24.0 to +49.1 points.
4.3 RQ3: An empirically weighted AI Data Readiness Score #
Chart 6 and the table below give the penalty curves fitted on all 10 datasets. These are the published weights.

| Defect (severity unit) | Penalty points at each severity level |
|---|---|
| Label noise (share of labels wrong) | 5%: 2.9 · 10%: 5.0 · 20%: 12.9 · 30%: 26.3 · 40%: 51.5 |
| Class imbalance (share of minority rows missing) | 20%: 7.6 · 40%: 13.1 · 60%: 24.3 · 80%: 41.1 · 90%: 56.8 |
| Target leakage (share of rows where a proxy reveals the label) | 20%: 2.1 · 40%: 8.5 · 60%: 20.1 · 80%: 40.5 · 95%: 65.6 |
| Reduced sample size (share of rows missing) | 20%: 2.3 · 50%: 3.6 · 75%: 7.0 · 90%: 21.0 · 98%: 45.9 |
| Missing values (share of cells missing) | 5%: 0.0 · 10%: 0.9 · 20%: 2.5 · 30%: 3.8 · 40%: 3.9 |
| Duplicates (deployment sub-score) | 5%: 0.4 · 10%: 0.2 · 20%: 0.0 · 30%: 0.0 · 40%: 0.5 |
Source: our analysis. A penalty point is one expected percent of above-chance balanced accuracy lost. Interpolate linearly between levels; Readiness = 100 minus the sum of penalties, floored at 0. Duplicates and leakage also carry the evaluation-integrity flags of Section 4.2.
Validation. On the 300 held-out combinations, the score ranked the actual loss with Spearman 0.839 (Pearson 0.803). The equal-weight checklist reached 0.658 and the count of defects present 0.471 (Chart 5). The bootstrap 95% interval for the score’s advantage is 0.124 to 0.242 over the equal-weight checklist and 0.283 to 0.457 over the count. Neither interval includes zero.

| Predictor | Spearman, 300 combinations | Pearson | Spearman, 900 combination × model pairs | Spearman vs BA drop in points |
|---|---|---|---|---|
| Empirical readiness score (curves) | 0.839 | 0.803 | 0.793 | 0.854 |
| Empirical score, one linear weight per defect | 0.816 | 0.792 | 0.772 | not computed |
| Equal-weight graded checklist | 0.658 | 0.658 | 0.621 | 0.665 |
| Count of defects present | 0.471 | 0.468 | 0.449 | 0.492 |
Source: our analysis, leave-one-dataset-out. All predictors receive the same measured severities.
The advantage holds across the board. The score beat the equal-weight checklist on every one of the 10 held-out datasets. Its Spearman ranged from 0.748 (blood-transfusion) to 0.972 (adult), against 0.511 (magic-telescope) to 0.875 (phoneme) for the checklist. It held for every model: 0.750 for logistic regression, 0.819 for random forest and 0.820 for HistGradientBoosting. The score also separates risk bands. The 93 combinations scored 85 to 100 lost 8.3% of above-chance performance on average (2.0 balanced-accuracy points). The 79 combinations scored below 50 lost 74.9% (18.9 points). Bands 70 to 85 and 50 to 70 lost 28.4% and 49.7%.
The score is calibrated in rank, less so in level. Its mean absolute error is 14.5 penalty points. On average actual loss exceeds the prediction by 4.8 points, so defects that occur together do slightly more harm than the sum of their parts. A simple linear version, with one weight per defect per 10% of severity (label noise 10.5, leakage 5.3, imbalance 5.2, sample size 2.7, missing values 1.1, duplicates 0.0), keeps most of the ranking power (0.816) and fits on a single line of a spreadsheet.
Which weights matter. Dropping one defect at a time from the score shows where its predictive power comes from. Without leakage, Spearman falls from 0.839 to 0.623. Without imbalance it falls to 0.710, without label noise to 0.721 and without sample size to 0.793. Without missing values or duplicates it stays at 0.839. For predicting deployment loss, those two checklist items carry no information.
5. Discussion: data readiness for AI in practice #
5.1 A short data readiness assessment derived from the weights #
The results turn into a short assessment. Severity is measured in the units of the table in Section 4.3, so each step needs one number.
| Step | What to measure | How | Weight class |
|---|---|---|---|
| 1 | Label error rate | Hand-audit a random sample of a few hundred labels; or flag suspects with confident learning [14][15] and audit those | High: 12.9 points at 20% |
| 2 | Leakage | For every feature, check whether it exists at prediction time; investigate any single feature with near-perfect validation scores | Low below q = 20%, then up to 65.6 points; largest effect on reported accuracy |
| 3 | Minority under-representation | Compare training and deployment class shares; check ROC-AUC alongside thresholded metrics | Medium: 7.6 points at 20%, mostly recoverable by threshold choice |
| 4 | Training set size | Count rows per class | Low until about three quarters of the data is gone, then 21.0 to 45.9 points |
| 5 | Missing values | Share of missing cells; check that missingness is random | Low: at most 3.9 points under MCAR |
| 6 | Duplicates | Exact-hash rows across the training and validation sets | Deployment 0; evaluation inflation up to 9.9 points for tree models |
Source: our analysis. Weights are from the table in Section 4.3.
flowchart TD
S[Dataset ready for AI?] --> L{Label error rate above 10 percent?}
L -->|Yes| L1[Clean labels first: largest deployment loss]
L -->|No| K{Any feature recorded after the outcome?}
K -->|Yes| K1[Remove it: reported accuracy cannot be trusted]
K -->|No| D{Duplicates across train and validation?}
D -->|Yes| D1[Deduplicate before splitting, then re-evaluate]
D -->|No| I{Minority class under-represented?}
I -->|Yes| I1[Tune threshold, report ROC-AUC too]
I -->|No| M[Missing values and size: fix only if extreme]
Three points matter for practice. First, the order of work should follow the weights. An equal-weight checklist sends the same effort to imputing missing values as to auditing labels. Our measurements show a five-fold difference in damage at the same dose. Second, a readiness assessment must judge the evaluation as well as the data. Duplicates and leakage are the two defects most likely to be “fixed” late, because they make the numbers look good. Third, the series friction taxonomy puts data quality among the barriers that organisations can reduce through their own action [1][2]. Weights tell them where that action pays off first.
5.2 Limitations and threats to validity #
- Synthetic defects. All defects are injected: random label flips, missingness completely at random, exact duplicates and an idealised leaked proxy. Real label noise depends on the class and on the features. Real missingness is often informative. Real duplicates are often near-duplicates. Each of these could change the weights, most likely raising the penalties for missing values and duplicates.
- Data scope. Ten small tabular binary classification datasets of at most 4,000 rows. Text, images, time series, regression and large datasets are not covered. The penalty for sample size in particular depends on how much data there was to begin with.
- Models. Three standard models with default hyperparameters and no tuning. Noise-robust losses, class weights or threshold tuning would reduce some penalties, the imbalance penalty most of all. Tabular foundation models, which do well on small datasets [24][25][25][26], may respond differently.
- Metric. Balanced accuracy at a 0.5 threshold. Under ROC-AUC, the imbalance weight would be much lower (a 4.7-point AUC drop at 90% removal against 15.7 points of balanced accuracy).
- Additivity. The score adds up single-defect penalties. Combined defects did 4.8 points more harm than predicted on average, and individual combinations deviated by 14.5 points on average.
- Statistics. Three seeds per condition. Loss curves are pooled over datasets, and per-dataset values vary widely (Chart 4). Correlations are computed over 300 combinations from 10 datasets, so they are not fully independent.
5.3 What would change the conclusion #
Our central claim is that label noise and leakage deserve far more weight than missing values and duplicates. Three findings would overturn it. One would be real-world defect profiles (systematic label noise, missing-not-at-random values, near-duplicates) under which missing values or duplicates hurt clean performance substantially. Another would be a model family that is robust to label noise but sensitive to missingness. The third would be an equal-weight checklist matching the empirical score on a larger, independent set of datasets.
5.4 Path to the journal version #
This article is the base version of a planned journal paper. The extension will:
- Grow the benchmark to 50 or more datasets from a curated suite such as TabArena [24][25], including multiclass and regression tasks.
- Add realistic defect models: class-conditional and feature-dependent label noise, missing-not-at-random patterns, near-duplicates and temporal leakage.
- Add noise-robust training, class weighting, threshold tuning and a tabular foundation model [25][26].
- Model interactions between defects explicitly, with a pairwise interaction term or a learned meta-model, and report calibration of score levels as well as ranks.
- Test the score on datasets with documented real defects, for example benchmarks with confirmed label errors [15][16], and compare it with existing readiness tools [7][8].
6. Conclusion #
RQ1 Finding: Label noise is the training-data defect that actually breaks models. Measured by the share of above-chance balanced accuracy lost on clean data at a matched 20% dose, it cost 12.9%, against 7.6% for class imbalance, 2.5% for missing values, 2.3% for reduced sample size, 2.1% for leakage and -0.3% for duplicates. It was the most damaging defect in 20 of 30 dataset-model cells. This matters for the series because data quality is the most-cited barrier to adoption, and the first unit of data-quality effort is best spent on labels.
RQ2 Finding: Duplicates and leakage do little or no harm at low doses on clean data but distort what the team reports. Measured by reported minus true balanced accuracy, a 40% duplicate rate inflated the random forest by 9.9 points, and a leaked proxy at q = 95% inflated logistic regression by 39.1 points (97.7% reported, 58.6% delivered). Label noise works the other way and understated performance by 10.5 points at 20% noise. This matters for the series because a readiness assessment that trusts the team’s own validation numbers can approve a model that will not work, or reject one that would.
RQ3 Finding: An empirically weighted AI Data Readiness Score predicts performance loss on unseen datasets far better than a checklist. Measured by Spearman correlation with actual loss over 300 held-out defect combinations, it scored 0.839, against 0.658 for an equal-weight checklist and 0.471 for a count of defects present. It beat the checklist on all 10 held-out datasets. This matters for the series because readiness indices are only useful for adoption decisions if their weights reflect measured consequences rather than convention.
The next question is whether these weights survive real-world defects: the same protocol applied to datasets with documented label errors, informative missingness and temporal leakage.
References (26) #
- Stabilarity Research Hub. (2026). AI Data Readiness Score: Label Noise Hurts 5x More Than Missing Data. doi.org. dtl
- Stabilarity Research Hub. Adoption Friction Taxonomy: Categorizing the Barriers Between AI Capability and Enterprise Deployment. tb
- Sambasivan, Nithya; Kapania, Shivani; Highfill, Hannah; Akrong, Diana; Paritosh, Praveen; Aroyo, Lora M. (2021). “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. doi.org. dcrtil
- Stabilarity Research Hub. AI Data Readiness Index Assessment. tb
- Lawrence, Neil D.. (2017). Data Readiness Levels. doi.org. dtil
- Afzal, Shazia, C, Rajmohan, Kesarwani, Manish, Mehta, Sameep, et al.. (2020). Data Readiness Report. doi.org. dtil
- Kaveen Hiniduma, Suren Byna, Jean Luca Bez. (2025). Data Readiness for AI: A 360-Degree Survey. doi.org. dcrtil
- Hiniduma, Kaveen, Ryan, Dylan, Byna, Suren, Bez, Jean Luca, et al.. (2025). AIDRIN 2.0: A Framework to Assess Data Readiness for AI. doi.org. dtil
- Hiniduma, Kaveen; Li, Zilinghan; Sinha, Aditya; Madduri, Ravi; Byna, Suren. (2025). CADRE: Customizable Assurance of Data Readiness in Privacy-Preserving Federated Learning. doi.org. dcrtil
- Wesley Brewer, Patrick Widener, Valentine Anantharaj, Feiyi Wang, et al.. (2025). Data Readiness for Scientific AI at Scale. doi.org. dcrtil
- Marsh, Ian; Stenmark, Victor; Poledna, Yuri; Hyyti, Heikki; Sanfridson, Martin; Aksoy, Eren. (2026). Data Readiness Levels for Automated Driving. doi.org. dcrtil
- Gebru, Timnit; Morgenstern, Jamie; Vecchione, Briana; Vaughan, Jennifer Wortman; Wallach, Hanna. (2021). Datasheets for datasets. doi.org. dcrtl
- Joan Giner-Miguelez, Abel Gómez, Jordi Cabot. (2025). On the Readiness of Scientific Data Papers for a Fair and Transparent Use in Machine Learning. doi.org. dcrtil
- Mohammed, Sedir; Budach, Lukas; Feuerpfeil, Moritz; Ihde, Nina; Nathansen, Andrea; Noack, Nele; Patzlaff, Hendrik; Naumann, Felix; Harmouch, Hazar. (2025). The effects of data quality on machine learning performance on tabular data. doi.org. dcrtil
- Northcutt, Curtis G., Jiang, Lu, Chuang, Isaac L.. Confident Learning: Estimating Uncertainty in Dataset Labels. doi.org. dtil
- (2021). [2103.14749] Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. doi.org. dti
- Pickler, Henrique, Kamassury, Jorge K. S., Silva, Danilo. (2025). Benchmarking noisy label detection methods. doi.org. dtil
- Jeonghoon Kim, Chao Huang, Xin Liu. (2025). An empirical study on impact of label noise on synthetic tabular data generation. doi.org. dcrtil
- Kapoor, Sayash; Narayanan, Arvind. (2023). Leakage and the reproducibility crisis in machine-learning-based science. doi.org. dcrtil
- Sharma, Ankit. (2026). Not All Data Leakage Is Equal: A Controlled Benchmark of Evaluation Inflation in Tabular Classification. doi.org. dtil
- Julio Suárez Albanchez. (2026). Beyond Accuracy: A Reproducible Framework for Detecting Data Leakage and Shortcut Learning in Machine Learning Pipelines Toward Trustworthy, Reproducible, and Sustainable AI Systems. doi.org. dcrtil
- Cheng, Yuxing, Chang, Yi, Wu, Yuan. (2025). A Survey on Data Contamination for Large Language Models. doi.org. dtil
- Zhining Liu, Zihao Li, Ze Yang, Tianxin Wei, et al.. (2025). CLIMB: Class-imbalanced Learning Benchmark on Tabular Data. doi.org. dcrtil
- Liu, Ruizhe, Luo, Jiaqi. (2026). Tabular Imbalanced Learning: A Survey, Benchmark, and Practical Guide. doi.org. dtil
- Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, et al.. (2025). TabArena: A Living Benchmark for Machine Learning on Tabular Data. doi.org. dcrtil
- Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, et al.. (2025). Accurate predictions on small data with a tabular foundation model. doi.org. dcrtil