GrayTell

Citation Integrity LLM — AI skill for Claude Code

AI community

Empirical evaluation of citation accuracy in frontier LLMs (DeepSeek, Claude, ChatGPT) under research-agent conditions.

How to install Citation Integrity LLM

This entry records only its repository, not the path inside it, so there is no exact command to give. Open GrayTell/citation-integrity-llm and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Citation Integrity LLM does

Empirical evaluation of citation accuracy in frontier LLMs (DeepSeek, Claude, ChatGPT) under research-agent conditions.

Alternatives in AI

Repomix — 📦 Repomix is a powerful tool that packs your entire repository into a single, AI-friendly file 22.7k ★
Welcome — AI Research Skills — You now have access to 86 production-ready skills covering the entire AI research lifecycle: literature survey 5.4k ★
Academic Commercialization Agent — Turn any research paper into a commercialization report — 6 AI agents, TRL/MRL scoring, patent landscape, mark 581 ★

README

Citation Integrity in Frontier Language Models

**Author:** Anubhav Sapkota **Affiliation:** GrayTell Labs **Date:** September 18, 2026 **License:** Apache 2.0

Overview

This repository contains the paper, data, prompts, and scoring rubric for an empirical evaluation of **citation accuracy in three frontier language models** — Claude Sonnet 5 Medium, ChatGPT 5.6 Terra, and DeepSeek V4.1 Flash — under research-agent conditions.

Each model was asked five research questions across three domains (AI/ML, biology/medicine, physics) and **forced to provide formal academic citations** in a strict format. Every citation was then manually verified against Google Scholar, PubMed, DOI resolution, and publisher websites.

Key Findings

Metric	Count	Percentage
Total citations evaluated	45	100%
Real & Accurate	32	71%
Real but Distorted	12	27%
Fully Fabricated	1	2%

**The dominant failure mode is author misattribution, not fabrication.**

Models frequently produced citations with a correct DOI, correct title, and correct journal — but attributed to the wrong authors. This error type is dangerous because it survives superficial verification (DOI resolution confirms the paper exists; a researcher may not check the authors).

Per-Model Accuracy

Model	Accurate	Distorted	Fabricated	Accuracy Rate
Claude Sonnet 5 Medium	12	3	0	80%
ChatGPT 5.6 Terra	11	3	1	73%
DeepSeek V4.1 Flash	9	6	0	60%

Claude was the most accurate. ChatGPT produced the only fabrication. DeepSeek had the highest distortion rate (40%), almost all through wrong author attribution.

**Note on statistical significance:** With n=15 per model, 95% confidence intervals overlap. These per-model differences should not be interpreted as statistically significant. See the Threats to Validit