---

Abstract #

Open-weight models let anyone compute token probabilities, but their training corpora are rarely public, so claims about training data are hard to check. We ask what a third party holding only the weights can verify. We scored 2,400 matched member and non-member documents from the Pile train and test splits, 400 Wikipedia articles created in October 2026 and the WikiMIA benchmark with three Pythia checkpoints (70M to 410M parameters), using four standard membership signals: loss, zlib ratio, Min-K% Prob and Min-K%++. We compared them with model-free blind baselines and ran 500-trial dataset-inference tests at set sizes of 25 to 200 documents. Three results stand out. First, single documents cannot be verified: on matched data the best attack on Pythia-410M reaches an AUC of 0.5153, and no attack exceeds 0.53 on any model or domain. Second, the common shortcut of using post-cutoff text as non-members inflates Min-K%++ to 0.6675, yet a control with no members at all scores 0.6669, and a bag-of-words classifier that never sees the model reaches 0.8971 to 0.9832. Third, set-level dataset inference with a proper held-out set keeps false positives near 5% but has power of only 0.02 to 0.118 at 200 documents, while a date-shifted held-out set produces a false-positive rate of 0.996. Verifiable training-data claims need held-out data, content marked before training or a published corpus, not stronger attacks.

1. Introduction #

Our previous article in this series, Vulnerability Disclosure in AI: Coordinated Disclosure Norms and Responsible Reporting Practices[1]^[2], looked at how outsiders report problems they find in AI systems. This article asks a prior question: what can an outsider find out at all about the data a model was trained on? The weights are public, so anyone can compute token probabilities, yet the training corpus usually is not, and the claims developers make about it (“trained on licensed and public data”, “no user data”, “benchmarks decontaminated”) are hard to check.

Transparency is moving in the wrong direction. The 2025 Foundation Model Transparency Index reports that the average developer score fell from 58 out of 100 in 2024 to 40 in 2025, and names training data and training compute among the areas where companies are most opaque[2]^[3]. Regulation is pushing the other way. Since 2 August 2025 the EU AI Act requires providers of general-purpose AI (GPAI) models to publish a public summary of their training content following a Commission template, but the Commission states that its AI Office will supervise that obligation “without performing a work-by-work assessment or checks whether specific content has been used or not” for training[3]^[4]. The Open Source Initiative’s Open Source AI Definition 1.0 asks for “sufficiently detailed information about the data used to train the system so that a skilled person can build a substantially equivalent system”, not for the data itself[4]^[5]. Both regimes therefore produce documents about data. Whether a document matches the weights is left to third parties, and the tool third parties reach for is membership inference.

Membership inference attacks (MIAs) on language models have a credibility problem. Large evaluations on Pythia models trained on the Pile found that attacks barely beat random guessing when members and non-members come from the same distribution, and that apparent successes came from temporal shift between the two sets[5]^[6]. Model-free “blind” classifiers beat published attacks on eight evaluation datasets[6]^[7]. We test which of these results hold for an auditor working on an open model with a public, split training corpus.

RQ1: With only the weights of an open model, how well can a third party tell whether a single document was in its training data, when members and non-members come from the same distribution? RQ2: How much does the common shortcut of using text published after the training cutoff as known non-members inflate measured detection, compared with model-free baselines? RQ3: How many documents does a set-level test (dataset inference) need to detect training use reliably, and what happens to its false-positive rate when the auditor’s reference set is drawn from a later date?

For the Trusted Open Source series, the answers decide which training-data claims about open models count as evidence and which remain statements of trust.

2. Existing Approaches (2025–2026 state of the art) #

2.1 Document-level membership inference from token probabilities #

Reference-free attacks score a text by how the target model assigns probability to it. The baseline is the average token loss; the zlib ratio divides that loss by the text’s compressed length to discount text that is simply predictable[5]^[6]. Min-K% Prob averages only the k% least likely tokens, on the hypothesis that unseen text contains a few outlier tokens with very low probability; its authors report a 7.4% improvement on their WikiMIA benchmark over earlier methods[7]^[8]. Min-K%++ normalises each token’s log-probability by the mean and spread of the model’s predictive distribution at that position, treating training samples as local maxima of the modelled distribution, and reports AUROC gains of 6.2% to 10.5% on WikiMIA over the runner-up[8]^[9]. New variants keep arriving: Gap-K% (2026) scores the log-probability gap between the model’s top-1 prediction and the actual token and reports state-of-the-art results on WikiMIA and MIMIR[9]^[10]. They cost one forward pass per document, which makes them the default tool for auditing an open model.

2.2 The distribution-shift critique and blind baselines #

Most published MIA benchmarks for language models are built post hoc: members are texts that probably existed before the training cutoff and non-members are texts created after it. WikiMIA, for example, takes members from Wikipedia event pages before 2017 and non-members from after 2023[6]^[7]. Duan et al. showed on the Pile train and test splits that the same attacks drop to near chance once the time gap is removed[5]^[6]. Maini et al. found the same confound and observed that most MIAs perform no better than random guessing between same-period members and non-members[10]^[11]. Meeus et al. quantified the shift in six post-hoc datasets with a bag-of-words classifier that never sees a model and concluded that all of them suffer from strong distribution shifts[11]^[12]. Das et al. went furthest: on WikiMIA a blind bag-of-words classifier reaches an AUC ROC of 99.0% against 83.9% for the best published attack (Min-K%++), and a TPR of 94.7% at 5% false positives against 43.2%[6]^[7]. The 2026 OLMo-Detect benchmark, built on the fully open OLMo 2 pipeline with non-members filtered against the corpus, finds that the best unsupervised and supervised attacks both reach an AUC of only 0.68 and that no unsupervised attack is robust to distribution shift, with AUCs moving by up to 0.42[12]^[13]. Even strong, reference-model attacks scaled to 1B-parameter models stay below an AUC of about 0.7 in practical settings, with many per-sample decisions statistically indistinguishable from a coin flip[13]^[14].

2.3 Set-level tests: dataset inference #

Copyright claims are usually about collections, not one paragraph. Dataset inference combines several MIA signals with a learned weighting and runs a statistical test of “suspect set scores higher than a known non-member set”. Maini et al. distinguish the train and test splits of Pile subsets with p-values below 0.1 and no false positives, and state that the method needs only 1,000 text sequences[10]^[11]. Puerto et al. aggregate paragraph-level signals and show that weak signals compound: on Pythia 6.9B, a paragraph-level AUROC of 0.528 on arXiv becomes 0.943 for collections of 500 documents[14]^[15]. Newer variants relax the access model: a 2025 black-box method compares the suspect model to locally trained reference models with and without the dataset[15]^[16], and CatShift infers membership from how much fine-tuning on the suspect set shifts the model’s outputs, without logits[16]^[17]. The weak point is the held-out set. Dataset inference needs private non-member data from the same distribution as the suspect set, which a 2026 paper calls “often unavailable or difficult to construct for real-world cases”; it proposes natural identifiers such as hashes and short URLs, whose format lets an auditor generate unlimited same-distribution non-members[17]^[18].

2.4 Proof by design: canaries, watermarks and the null hypothesis #

Zhang et al. argue that MIAs cannot prove training use at all, because convincing evidence requires a low false-positive rate under the null hypothesis that the model was not trained on the data, and that null cannot be sampled without knowing the training set or retraining the model[18]^[19]. Their two sound alternatives are data extraction and membership inference on canaries inserted in advance. STAMP implements the second idea for publishers: release one watermarked rephrasing of a text, keep others private, and compare model likelihoods with a paired test; it detects benchmarks that appear once in training and make up less than 0.001% of tokens[19]^[20]. Watermark-style methods are covered in more depth in our earlier article on model watermarking[20]^[21]. The Hubble suite (2025) makes the idea measurable at scale, releasing paired standard and perturbed models where book passages, biographies and test sets were inserted at controlled frequencies, and shows that the same sequence is memorised better in a smaller corpus than in a larger one[21]^[22]. Canaries work only for data marked before training, so they help future rights holders, not audits of models already released.

2.5 Provenance and disclosure infrastructure #

The other path is documentation that can be checked against a released corpus. The Data Provenance Initiative audited more than 1,800 text datasets and found licence omission rates above 70% and error rates above 50% on popular hosting sites[22]^[23]. Where the corpus itself is public, verification becomes a lookup: OLMoTrace finds verbatim matches between model output and OLMo’s multi-trillion-token training data within seconds[23]^[24], and the Common Pile v0.1 offers 8 TB of openly licensed text from 30 sources, with two 7B models trained on it that are competitive with Llama 1 and 2 7B[24]^[25]. On the regulatory side, the EU template requires providers to list the internet domain names in the top 10% by size of content scraped (top 5% or 1,000 domains for small and medium-sized enterprises), and non-compliance can be fined up to 3% of worldwide turnover or EUR 15 million[3]^[4]. The GPAI Code of Practice, published on 10 July 2025, adds a Model Documentation Form in its Transparency chapter[25]^[26]. A 2026 survey of 95 publications maps provenance, transparency and traceability work for LLMs into one taxonomy[26]^[27].

flowchart TD
    A[Training-data claim about an open model] --> B[Statistical tests on weights]
    A --> C[Marks inserted before training]
    A --> D[Disclosure and provenance]
    B --> B1[Document MIA: loss, zlib, Min-K, Min-K++]
    B --> B2[Dataset inference: aggregate plus t-test]
    B --> B3[Black-box and token-only variants]
    C --> C1[Canaries and copyright traps]
    C --> C2[Watermarked rephrasings]
    D --> D1[EU AI Act training-content summary]
    D --> D2[OSI data information]
    D --> D3[Open corpora and output tracing]

3. Method and Data #

Target models. We audit three checkpoints of Pythia, a suite released with its training data and data order so that training membership is known exactly[27]^[28]: Pythia-70M, -160M and -410M (70.4M, 162.3M and 405.3M parameters). Pythia was trained on the Pile, an 825 GiB English corpus built from 22 subsets, distributed with separate train and test splits[28]^[29]. The models are small enough to score on a 6-core CPU without a GPU, which is the point: this is the audit any third party can run on open weights today.

Matched member and non-member sets (RQ1). Following the evaluation design of Duan et al.[5]^[6] and Maini et al.[10]^[11], members are Pile training documents and non-members are Pile test documents from the same source. We took the first 400 documents of at least 1,500 characters from each of three sources, English Wikipedia, ArXiv and Pile-CC (a Common Crawl extract), in the first training shard and in the test split of the publicly mirrored Pile. That is 2,400 documents, 400 per cell.

Date-shifted sets (RQ2). We built the shortcut that many audits use: 400 English Wikipedia articles created between 13 and 21 October 2026, years after Pythia’s training data was fixed, pulled through the MediaWiki API (new pages of at least 5,000 bytes, plain-text extracts). Section-heading lines were removed so that the fresh articles differ from Pile Wikipedia by date and not by markup. We compare Pile train Wikipedia with the fresh set, and, as a negative control, Pile test Wikipedia with the fresh set. Neither side of the control contains a single training document, so any detection there is shift, not membership. We also score the two WikiMIA splits of 64 and 128 words (542 and 250 texts)[7]^[8].

Signals and metrics. Texts are truncated to 2,500 characters and the first 256 tokens are scored in fp32. For every document we record four reference-free signals: mean token loss, zlib ratio, Min-K% Prob and Min-K%++ with k = 20%. We report ROC AUC with 1,000-sample bootstrap 95% confidence intervals and the true-positive rate at 1% false-positive rate. Two model-free blind baselines in the spirit of Das et al.[6]^[7] never touch a model: a 5-fold cross-validated TF-IDF bag-of-words classifier and the latest four-digit year mentioned in the text.

Dataset inference (RQ3). We implement the Maini et al. procedure on Pythia-410M: a logistic regression learns a weighting of the four signals on one half of the suspect and validation sets, and a one-sided Welch t-test asks whether the suspect half scores higher than the validation half on the held-out data. For set sizes of 25 to 200 documents we run 500 random trials each. Power is the share of trials with p < 0.05 when the suspect set is made of members; the false-positive rate is the same share when the suspect set is made of non-members. We run it twice: once with a validation set from the Pile test split (matched), and once with the fresh October 2026 Wikipedia articles as validation set, which is what an auditor without private same-period data would have to use.

All data, scores and code are in stabilarity/hub/research/open-source-model-auditing-infrastructure-third-party-verification-of-training-data-claims. Scoring took 494, 923 and 2,100 seconds for the three models on a 6-core CPU; the statistical analysis is offline and deterministic.

flowchart LR
    P1[Pile train: 1,200 docs, members] --> S[Score with Pythia 70M, 160M, 410M]
    P2[Pile test: 1,200 docs, non-members] --> S
    F[Wikipedia created Oct 2026: 400 docs] --> S
    W[WikiMIA 64 and 128 words] --> S
    S --> M[Loss, zlib, Min-K, Min-K++]
    M --> R1[RQ1 matched AUC]
    M --> R2[RQ2 shifted AUC]
    B[Blind baselines, no model] --> R2
    M --> R3[RQ3 dataset inference t-test]

4. Results #

4.1 RQ1: Single documents cannot be verified from the weights #

On matched sets, every signal is at chance. For Pythia-410M the best attack reaches an AUC of 0.5153 on Wikipedia (Min-K%), 0.503 on ArXiv and 0.5059 on Pile-CC (both zlib ratio), and every one of the twelve attack-domain confidence intervals includes 0.5. Averaged over the three domains, the attacks score 0.4965 (loss), 0.5063 (zlib), 0.4954 (Min-K%) and 0.4841 (Min-K%++). The single highest AUC across all three models, four attacks and three domains is 0.53. At a false-positive rate of 1%, which is the operating point that matters for an accusation, no attack on Pythia-410M identifies more than 1.25% of members.

Model size does not rescue the attacks in this range. The mean of the best attack per domain is 0.5056 for Pythia-70M, 0.512 for 160M and 0.5081 for 410M. This replicates the near-chance finding that Duan et al. reported for Pythia models up to 12B parameters[5]^[6]. It is also consistent with OLMo-Detect’s ceiling of 0.68 on a much larger, corpus-filtered benchmark[12]^[13].

One detail is worth noting for auditors. The blind bag-of-words classifier scores 0.5338, 0.5379 and 0.5511 on the three matched comparisons, slightly above the attacks, and on Pile-CC its interval (0.5135 to 0.591) excludes 0.5. Even the Pile’s own train and test splits are not perfectly exchangeable, so a small “membership” signal of 0.51 to 0.55 cannot be attributed to the model without a blind baseline next to it.

4.2 RQ2: Date-shifted non-members inflate AUC, and the model is not needed #

Replacing Pile test documents with Wikipedia articles created in October 2026 changes the picture. Pile train Wikipedia against the fresh set gives Min-K%++ an AUC of 0.6675 (95% CI 0.6293 to 0.7038), 0.1522 above its matched counterpart. The negative control explains where that gain comes from: Pile test Wikipedia, which Pythia never trained on, against the same fresh set gives Min-K%++ an AUC of 0.6669 (95% CI 0.6294 to 0.7066). The two confidence intervals almost coincide. The attack detects the date, not the training run.

Blind baselines do far better than the model-based attack on every shifted comparison. The bag-of-words classifier reaches 0.9224 on Pile train against fresh Wikipedia and 0.8971 on the member-free control; simply reading the latest year mentioned in the text gives 0.6943 and 0.6987, on a par with Min-K%++. On WikiMIA the gap is wider: with 64-word texts the best attack on Pythia-410M scores 0.6119 while the bag-of-words baseline scores 0.9832 and the year heuristic 0.9105; with 128-word texts the figures are 0.6258, 0.9559 and 0.899. These match the direction and size of the Das et al. result on WikiMIA, a blind AUC of 99.0% against 83.9% for the best published attack[6]^[7].

The shifted curves track each other across model sizes (0.6235, 0.6161 and 0.6675 with members; 0.6304, 0.6046 and 0.6669 without): a larger model makes the shift more visible, not membership.

Comparison (Pythia-410M)	Members in set	Best MIA AUC	Bag-of-words AUC (no model)	Latest-year AUC (no model)
Wikipedia, Pile train vs Pile test	400 of 800	0.5153	0.5338	0.5044
Wikipedia, Pile train vs created Oct 2026	400 of 800	0.6675	0.9224	0.6943
Wikipedia, Pile test vs created Oct 2026	0 of 800	0.6669	0.8971	0.6987
WikiMIA, 64 words	284 of 542	0.6119	0.9832	0.9105
WikiMIA, 128 words	139 of 250	0.6258	0.9559	0.899

Source: our analysis, results.json. AUC is dimensionless; 0.5 is chance. Bag-of-words AUC is 5-fold cross-validated.

4.3 RQ3: Set-level tests either miss or accuse falsely, depending on the reference set #

With a matched validation set, dataset inference controls false positives as it should: across all three domains and five set sizes the false-positive rate stays between 0.026 and 0.066, around the nominal 0.05. But it finds almost nothing. At 200 suspect and 200 validation documents, power is 0.02 on Wikipedia, 0.018 on ArXiv and 0.118 on Pile-CC. Pile-CC is the only domain where the median p-value falls with set size, from 0.4486 at 25 documents to 0.2861 at 200, so a signal may exist there at larger scale.

Swap the validation set for October 2026 Wikipedia and the test appears to work: power rises from 0.312 at 25 documents to 0.884 at 100 and 1.0 at 200. The false-positive rate rises with it, from 0.276 at 25 documents to 0.808 at 100 and 0.996 at 200. With 200 documents of Pile test Wikipedia as the suspect set, a dataset Pythia never saw, the test reports training use in 99.6% of trials. More data makes the false accusation more certain, not less.

The matched null result most likely reflects scale rather than the method. Maini et al. work with 1,000 sequences per set on Pythia models from 410M to 12B[10]^[11], and Puerto et al. report that a paragraph-level AUROC of 0.51 yields collection-level AUROCs of only 0.5 to 0.65, while 0.53 lifts them to between 0.6 and 0.9[14]^[15]. Our per-document AUCs on 256 tokens sit at 0.50 to 0.515, below that threshold. Our estimate, from these two sources and our measurements, is that a third party auditing a sub-1B open model for the use of a corpus of a few hundred short documents should expect a non-result, and that a positive result with a date-shifted reference set is evidence of nothing.

Verification route	What it can establish	Access needed	Key requirement	Reported performance
Document MIA (loss, Min-K%, Min-K%++)	Nothing reliable for one document	Weights or log-probs	Same-distribution non-members	AUC 0.5153 at best on matched Pile (ours); 0.68 ceiling on OLMo-Detect[12][13]
Dataset inference	Use of a collection	Weights or log-probs	Private IID held-out set	p < 0.1, no false positives, 1,000 sequences[10][11]; power 0.118 or less at 200 documents (ours)
Natural-identifier dataset inference	Use of a collection with hashes or short URLs	Weights or log-probs	Data must contain identifiers	Removes the held-out requirement[17][18]
Watermarked rephrasings (STAMP)	Use of content marked before release	Log-probs	Publisher acts before training	Detects sets under 0.001% of tokens[19][20]
Verbatim tracing (OLMoTrace)	Exact overlap of output and corpus	Released corpus and index	Developer publishes data	Results within seconds[23][24]
EU training-content summary	Domains and dataset categories	Public document	Applies to GPAI providers in the EU	Top 10% of domains by size, no work-level checks[3][4]

Source: cited works and our analysis (results.json).

5. Discussion #

What a third party can verify today. For an open-weight model with an unpublished corpus, tests on the weights cannot settle a claim about one document, nor about a modest collection unless the auditor holds same-period text the developer could not have used. Zhang et al. put the formal reason in one sentence: a sound proof needs a false-positive rate under a null hypothesis that cannot be sampled[18]^[19]. The commonly used substitute for the null, text created after the cutoff, yields a test that convicts a model of training on data it never saw in 99.6% of trials at 200 documents. An audit that uses post-cutoff text as its reference set measures distribution shift.

What does verify is evidence designed in advance or data disclosed in full. Rights holders who want future leverage should mark content before release (canaries, watermarked rephrasings) and keep private, same-period copies as held-out sets; the STAMP and natural-identifier results show this is technically mature[19]^[20][17]^[18]. Developers who want their claims trusted can publish the corpus or an index of it, as OLMo and the Common Pile do, which turns the question into a lookup[23]^[24][24]^[25].

What regulation and definitions add. The EU summary template and the OSI definition both require a description of data rather than the data. Under the template, a provider lists the largest domains and the dataset categories, and the AI Office checks the form, not the content[3]^[4]. Under OSAID 1.0 a model can qualify as open source with unshareable data, provided the data is described well enough that a skilled person can build a substantially equivalent system[4]^[5]. Neither description can be checked against the weights by an outsider. Licence metadata adds its own error: the Data Provenance Initiative’s omission rates above 70% suggest that even honest summaries inherit mislabelled sources[22]^[23]. The FMTI’s finding that training data is among the most opaque areas[2]^[3] describes the starting point, and the CheckMIABench argument that any open model with intermediate checkpoints and public training data can be turned into a clean testbed[29]^[30] describes the most useful thing open developers can release next.

flowchart TD
    Q[Claim about training data] --> A{Corpus or index published?}
    A -->|Yes| L[Look up overlap, verifiable]
    A -->|No| B{Content marked before training?}
    B -->|Yes| C[Canary or watermark test, sound]
    B -->|No| D{Private same-period held-out set?}
    D -->|Yes| E[Dataset inference, needs about 1,000 docs]
    D -->|No| F{Reference set from after cutoff?}
    F -->|Yes| G[Result measures date shift, reject]
    F -->|No| H[Disclosure only, unverifiable]

Limitations and threats to validity. We tested small models: 70M to 410M parameters. Larger models memorise more, and Hubble shows that memorisation depends on how often a sequence appears relative to corpus size[21]^[22], so a 7B model audited for a duplicated corpus could look different. We scored only the first 256 tokens of each document and used four reference-free signals, while stronger reference-model attacks exist; even those stay below an AUC of about 0.7 in practical settings[13]^[14]. Dataset inference was capped at 200 documents per set by our sample of 400 per cell. The blind baseline’s small signal on matched Pile data means our matched sets are themselves not perfectly exchangeable. The fresh 2026 articles are new pages, many of them on older subjects such as a 1962 state senate election, so they are a milder shift than WikiMIA’s post-2023 events, and stronger shifts would only widen the gap we report.

What would change the conclusion. A same-distribution test on a 1B-plus open model that detects a few hundred documents at a false-positive rate near 1% would weaken RQ3. A reference-free attack that beats a blind baseline on a member-free control would weaken RQ2. Rules requiring corpus indexes or attested training pipelines would change what is verifiable without changing the statistics.

6. Conclusion #

RQ1 Finding: From the weights alone, a third party cannot tell whether a single document was in an open model’s training data when members and non-members come from the same distribution. Measured by the best ROC AUC on matched Pile train and test documents for Pythia-410M = 0.5153 (Wikipedia), 0.503 (ArXiv) and 0.5059 (Pile-CC), with a maximum of 0.53 over all models, attacks and domains, and a true-positive rate of at most 1.25% at 1% false positives. This matters for the series because claims about individual works in open-model training data remain trust-based unless the corpus is published.

RQ2 Finding: Post-cutoff non-member sets inflate detection by about 15 AUC points, and the inflation comes from the date, not the training run. Measured by Min-K%++ AUC = 0.6675 with members against 0.6669 on a control with no members, and by blind bag-of-words AUCs of 0.8971 to 0.9832 that beat every model-based attack on shifted data. This matters for the series because a trustworthy open-model audit must report a blind baseline and a member-free control next to any membership result.

RQ3 Finding: At the scale we tested, set-level dataset inference needs more than 200 documents: with a proper held-out set it controls false positives but detects little, and with a date-shifted held-out set it accuses falsely. Measured by power at 200 documents = 0.02 to 0.118 with matched validation, and by a false-positive rate = 0.996 at 200 documents with an October 2026 validation set. This matters for the series because the infrastructure that makes training-data claims verifiable is held-out data, content marking and corpus disclosure, not better attacks.

The next question for the series is which verification artefacts, from corpus indexes to attested training runs, open developers can publish without exposing data they are not allowed to share.