Copies Are Not Confirmations: One in Five News Articles Is a Near-Copy, and LLM Judges Count Reprints as Proof
AI EVALUATION
A claim repeated by three outlets looks confirmed; often it is one wire story wearing three mastheads. Measuring 200,000 articles and four LLM judges, a new paper shows text duplication moves small judges from 6% to 71% lie acceptance, and tests what deduplication can and cannot fix.
By Ossam Hashim
KEY FINDINGS
- In a 200,000-article sample of English CC-News (2017-2019, 6,877 outlets), a MinHash/LSH detector links 18.9% of articles to a near-copy from another outlet, in 12,243 chains - a lower bound, since the sample holds about 28% of the corpus. Hand-labelling 150 detected pairs against newsroom taxonomy: 82% reflect real dependence (52 wire copy, 40 same-owner sharing, 22 syndication, 8 press releases); stripping boilerplate lines before hashing raises precision to 0.97.
- Three identical reprints of a claim about an invented person move the local judges sharply: Llama 3.2 3B accepts 6% of such lies shown one report and 71% shown three copies (adjusted p < 10^-13); phi4 14B goes from 21% to 90% (p < 10^-14). Both ignore a note stating the reports are one source. Gemini 2.5 Flash already believes a single report (93%); Gemini 2.5 Pro accepts 0% when the provenance note is present. Copies also move judges from Unsure to True when abstention is allowed, so the effect is not a forced-binary artefact.
- In a simulated continual learner (a LoRA-adapted Qwen2.5-1.5B student fed a 30-day report stream), showing the judge one report per detected chain instead of every copy cuts the copied lies accepted by 0.10 to 0.24 across all four judges, at a mean cost of no more than 4.2 percentage points of accepted truths.
- Text similarity is blind to reworded copies: the detector links 92% of lightly edited reprints to their original but 0% of LLM rewrites by phi4, Llama 3.2 or Gemini 2.5 Flash. A new-information test - the share of a report's named entities and quotations absent from its earlier look-alike - separates rewrites from 100 labelled independent reports of the same event with AUC 0.98-0.99 and cuts accepted reworded-copy lies from 0.67 to 0.20 under realistic detector-miss patterns, or to 0.40 when the rewriter is instructed to invent plausible details.
- The authors' bottom line is defensive humility: counting reports or chains alone fails in both directions, the effect of provenance information depends entirely on the judge, dependence detection does not replace claim verification, and colluding sources - coordinated but non-copying - evade every signal evaluated.
One Story, a Dozen Mastheads
The measurement half of the paper quantifies what media critics have long asserted. A uniform sample of 200,000 CC-News articles is reduced to 5-word shingles, hashed with 64 MinHash permutations, and locality-sensitive hashing surfaces cross-outlet pairs at estimated Jaccard similarity 0.6 or better, joined into chains. Most chains are small - 7,550 of 12,243 span exactly two outlets, 165 span eleven to fifty - and the earliest articles in detected chains sit most often on wire-service sites or on client sites carrying AP and Canadian Press copy. The share is stable however the comparison window is cut (18.5% on the first 1,000 characters, 19.4% on full text, 20.0% if either end may match), and all 27 false matches in the labelled sample trace to shared boilerplate - login pages, paywalls, report templates - which is why the boilerplate-stripped variant reaches 0.97 precision. The taxonomy matters for what follows. Wire copy, same-owner sharing, syndication and press-release reproduction are ordinary, legitimate newsroom economics; none of it is deception. The problem is downstream: any system - a human skimming search results, an LLM judge asked to verify a claim, a continually updated model - that silently converts "reported in three places" into "confirmed by three sources" is doing arithmetic on an illusion of independence.
Judges That Count Reprints
The judge experiment is deliberately artificial in order to be clean: 320 claims - lies about invented people, true facts about invented people, known-fact counterfactuals, legitimate updates - shown to four judges (Llama-3.2-3B-Instruct and phi4 as 4-bit local builds, Gemini 2.5 Flash and Pro via API) in nine conditions from zero to ten reports. No judge accepts a claim about an unknown entity without any report (at most 1%). The dose-response to copies is where the local judges fail: three identical reprints flip 52 of Llama 3.2's 80 lie verdicts to True with none flipping back. The effect is not monotone - five and ten copies persuade less than three, for reasons the paper does not resolve - and differently worded reports move Gemini Pro further than identical ones (1.00 versus 0.76), sensitivity to diversity of evidence rather than proof the judge infers provenance. Provenance information splits the field. Told that all reports reprint one wire story, Gemini 2.5 Pro accepts none of the lies - and, shown the same note with an Unsure option, abstains on every one; the local judges ignore the note entirely, and Flash accepts nearly everything regardless. A wrong provenance note is its own hazard: Gemini Pro obeys a false one-source note even against its trust in a single report, so metadata is a defence only if it is correct. The paper's five findings are backed by exact McNemar tests with Holm adjustment over 112 contrasts - unusually careful statistics for this genre.
One Report per Chain
The continual-learning simulation puts the failure where it would actually operate: a small student model absorbing a news stream over 30 simulated days, with claims replayed from a buffer and acceptance rules deciding what enters training. Lies arrive via three attacks - verbatim copy networks of ten, paraphrased copies, and colluding sources that coordinate without copying. Counting rules fail as expected: accept-at-three-reports is beaten by copy networks, and accept-after-two-chains is beaten by the detector's own misses, while blocking true news only one outlet carries. The result that survives is deduplication at the evidence layer: when an LLM judge sees one report per detected chain rather than every copy, accepted copy-attack lies fall by 0.10 (Gemini Flash) to 0.24 (Llama 3.2), and the truth cost stays at or under 4.2 points on average. Copy removal fixes exactly one failure - counting copies as sources - and helps least the judge that needs it most, since Gemini Flash already believes a single unverified report.
A Novelty Test for Rewrites
The second defence targets what MinHash cannot see. Independent reporting of the same event usually adds entities - another witness, a figure, a quote - while a rewrite keeps them; so for a report resembling an earlier one, the test measures the share of its named entities (spaCy, 16 types) and quoted passages absent from the earlier text, and flags a copy below 0.4 novelty. Against 200 phi4 rewrites it reaches AUC 0.98-0.99, with none of the 11 manually confirmed independent reports in the held-out pairs wrongly flagged (Wilson interval [0, 0.26] on that small sample); entity overlap alone performs as well, with TF-IDF and a current embedding model close behind. The weaknesses are stated plainly: an adaptive attacker told to invent three or four plausible details weakens it, the score reads entities rather than propositions so a contradiction counts as novelty, and it catches only 17% of natural same-source derivatives such as retold interviews. In the stream it still cuts accepted reworded-copy lies from 0.67 to 0.20 with realistic miss patterns - and to only 0.40 against the detail-inventing attacker.
Assumptions and limitations
All findings are the authors' own, on their sample and setup. The 18.9% copy rate is a detector lower bound on a 28% sample of one corpus (CC-News 2017-2019, English, articles over 300 characters); the dependence taxonomy labels came from an AI-assisted draft process, and the new-information test's false-alarm estimate rests on 100 pairs labelled by the first two authors, who are not independent of the study. The judge results cover four models at temperature 0 with short answer budgets on invented claims about invented people - a controlled instrument, not deployment traffic - and the five-to-ten-copy non-monotonicity is unexplained. The learning simulation feeds measured detector rates into a scripted stream rather than running detectors on generated text, main results use three seeds, and the stream evaluation of the novelty test used phi4 only. Nothing here detects colluding sources that coordinate without copying, and removing copies does not make an unreliable judge reliable - Gemini Flash's trust in a single report is untouched by deduplication. The worked example is an invented toy implementing the paper's novelty formula illustratively; all statistics above are the authors' measurements, not this blog's.
Worked Example: Entity Novelty in Twelve Lines
The new-information test reduces to a set difference: extract a report's named items, drop the ones its look-alike already contains, and ask what fraction is new. Below is a minimal proxy - capitalised words as stand-in entities - scored against the paper's 0.4 flag threshold. A reworded copy scores 0.25 and is flagged; an independent report adding a witness scores 0.50 and passes. Invented sentences, illustrative only.
import re
def ents(t):
w=re.findall(r'[A-Z][a-z]+',t)
return set(x.lower() for x in w)
def novelty(old, new):
eo, en = ents(old), ents(new)
return len(en - eo) / len(en)
orig = ('Ana Reyes opened '
'the new Lima plant')
rw = ('The Lima plant was '
'opened by Ana Reyes')
ind = ('Ana Reyes opened the '
'Lima plant; Worker Musa '
'Diop gave a short tour')
pairs = [('rewrite', rw),
('report', ind)]
for tag, r in pairs:
n = novelty(orig, r)
print(tag, round(n, 2), n < 0.4)
Sources & Provenance
- Copies Are Not Confirmations: How News Duplication Fools LLM Judges and Continual Learners (Metales, Grabenheinrich, Metales; Djilali Liabes University of Sidi Bel Abbes, 9 Nov 2026) - arXiv abstract
- Full paper (PDF): detector design, judge tables, stream simulation, novelty-test ablations - arXiv full text
- Code and labels, released by the authors - experiment repository
All figures are attributed to the preprint as published. Sources accessed 9 November 2026. This analysis is independent editorial content for GlobalTechDigest.
Comments
Post a Comment