Attribution: This article was based on content by @thomasfuchs on mastodon.
Original: https://hachyderm.io/@thomasfuchs/115549398807745536

Introduction

A viral social post recently captured a raw public reaction to a technical idea: train a large language model (LLM) on nuclear licensing documents and use it to generate parts of new license applications. The poster’s profanity underscored a reasonable instinct: this sounds risky, even reckless. But beneath the shock value there’s a subtler story about how AI tools can support — not replace — highly regulated, safety-critical workflows like nuclear licensing. In this article I unpack what a realistic, responsible AI-assisted licensing workflow would look like, what technical building blocks are required, and where the real risks and safeguards lie.

Credit: original Mastodon post by @thomasfuchs and reporting at 404Media (link: https://www.404media.co/power-companies-are-using-ai-to-build-nuclear-power-plants/).

Key Takeaways

  • AI can speed drafting, extraction, and compliance mapping but must be integrated with strict provenance and human verification.
  • Use retrieval-augmented generation (RAG), knowledge graphs, and human-in-the-loop workflows to anchor outputs to sources and reduce hallucination.
  • Data governance, secure deployment, and auditable workflows are essential; responsibility for any submitted document remains with licensed engineers and licensees.
  • Pilot, validate, and measure AI outputs against regulatory criteria and safety-focused QA before production use.

Background: The U.S. Nuclear Regulatory Commission (NRC) oversees nuclear licensing and requires detailed, traceable safety and environmental documentation.

Main concepts: what people mean by “train an LLM on licensing documents”

Large language models (LLMs) are neural networks trained to predict and generate text conditioned on input. In the context of licensing, two high-level approaches are often discussed:

  • Fine-tuning a domain-specific LLM on a corpus of past licensing documents and technical reports so the model “speaks the language” of the domain.
  • Using a base LLM combined with retrieval — commonly called retrieval-augmented generation (RAG) — where the model is given relevant documents at inference time so it can ground responses on actual sources (Lewis et al., 2020).

Neither approach magically automates compliance. Licensing packages submitted to the NRC include highly structured artifacts — safety analysis reports, environmental impact statements, site-specific design bases, quality assurance (QA) programs, and technical specifications — all tied to regulations such as 10 CFR Parts 50 and 52. Any AI output must therefore be auditable, source-linked, and verified by qualified humans before it is relied on for regulatory submission (U.S. Nuclear Regulatory Commission, 2020).

Practical building blocks and techniques

  1. Retrieval-augmented generation (RAG)
  • How it helps: RAG combines a document search/index with an LLM so generated text cites and reflects retrieved passages rather than only internal model weights. This reduces “hallucination” — the model inventing facts — and improves traceability (Lewis et al., 2020).
  • Practical tip: Index NRC public documents, past license applications, and applicable regulations. At query time, retrieve the most relevant passages and prompt the LLM to generate sections that explicitly reference those passages.
  1. Knowledge graphs and requirement tracing
  • How it helps: A knowledge graph can map regulations to specific document sections, test results, and design assumptions. This enables automated requirement tracing and impact analysis when a regulation changes (Hogan et al., 2021).
  • Practical tip: Build a lightweight graph that links 10 CFR clauses to template sections and to evidence artifacts (e.g., test reports). Use automated extraction to connect new documents into the graph, but retain human curation.
  1. Document understanding and info extraction
  • How it helps: Natural language processing (NLP) tools can extract structured data (e.g., design parameters, failure modes, QA commitments) from PDFs and past filings. These structured facts feed templates and support consistency checks.
  • Practical tip: Combine rule-based parsers with supervised models and validate extractions against human-labeled samples during pilot phases.
  1. Human-in-the-loop workflows and auditable outputs
  • How it helps: Humans remain responsible for content; AI accelerates drafting, not final approval. Maintain versioned artifacts, review logs, and explicit source citations for every generated paragraph (Amershi et al., 2019).
  • Practical tip: Integrate approval gates: draft → SME review → regulatory counsel review → traceability verification → submission.

Concrete examples / use cases

  1. Drafting and templating a Safety Analysis Report (SAR) subsection
  • What AI could do: Given a retrieval vector of prior SAR subsections, site data, and applicable code clauses, a RAG pipeline can produce a first-draft subsection (e.g., “reactor coolant system description”) that includes inline citations to prior filings and standards.
  • Human role: A licensed engineer verifies calculations, checks assumptions, and signs off on regulatory claims before submission.
  1. Requirement-to-evidence tracing for inspections
  • What AI could do: Extract regulatory requirements from 10 CFR and automatically link them to evidence documents (test reports, inspections, QA records). The system could surface gaps where evidence is missing.
  • Human role: Compliance personnel triage gaps and assign remedial actions; auditors verify trace links.
  1. Rapid regulatory Q&A and review support
  • What AI could do: During NRC pre-application interactions, an assistant can summarize past NRC decisions on similar issues, propose draft responses, and cite relevant NUREG guidance.
  • Human role: Regulatory affairs leads revise language to reflect site-specific constraints and legal strategy.

Risks, mitigations, and governance

  • Hallucination and unsupported claims: LLMs may generate plausible-sounding but false statements. Mitigation: require evidence citations for any factual claim, use RAG with high-recall retrieval, and run independent verification checks against source documents (Lewis et al., 2020; Bender et al., 2021).
  • Traceability and provenance: Regulatory reviewers require traceable links from claims to test data and standards. Mitigation: store provenance metadata for every generated token/section, and integrate outputs with a document management system that logs the chain of custody.
  • Data rights and sensitive information: Operators must avoid exposing sensitive site data during model training or retrieval. Mitigation: prefer on-premises models or confidential compute environments, segregate public vs. internal corpora, and apply strict access controls.
  • Liability and responsibility: The legal responsibility for submissions stays with the licensee and signing engineers. Mitigation: clearly document AI’s role, require sign-off by credentialed personnel, and treat AI as a drafting aid rather than an authorizing party.
  • Model drift and regulatory changes: Regulations and guidance evolve. Mitigation: implement continuous update pipelines for regulatory corpora and periodic model re-evaluation; attach timestamps to citations and require cross-checks against current rules.

Evaluation frameworks and QA checkpoints

Before any production use, AI outputs must be validated against measurable criteria:

  • Citation coverage: fraction of factual claims with an explicit source citation.
  • Traceability score: percentage of regulatory requirements linked to evidence artifacts.
  • Human correction rate: proportion of AI-generated text requiring substantive edits.
  • Red-team tests: adversarial prompts to surface failure modes and hallucination patterns.
  • Safety-critical verification: for any content that affects safety analysis, require independent, reproducible calculation checks rather than linguistic review alone (Amershi et al., 2019).

Regulatory and cultural context

Nuclear licensing is conservative for good reasons: public safety, legal liability, and political scrutiny. Adoption will therefore be incremental and focused on augmentation rather than automation. Pilots that limit AI to low-risk drafting, extraction, and review assistance are the likeliest path forward. Regulators and operators will emphasize deterministic behavior, transparent source attribution, and rigorous validation workflows (U.S. Nuclear Regulatory Commission, 2020).

Background: “Hallucination” in LLMs refers to confident but ungrounded or incorrect model outputs; it is a well-known failure mode requiring mitigation.

Best practices and recommendations (actionable)

  • Start small with pilot projects: focus on high-value, low-risk tasks like document summarization, indexing, and evidence linking.
  • Use RAG, not blind fine-tuning: anchor generated text to retrieved passages and show those passages inline for reviewers (Lewis et al., 2020).
  • Build a requirements-to-evidence knowledge graph: automate traceability and impact analysis on regulation changes (Hogan et al., 2021).
  • Maintain human-in-the-loop and strict approval gates (Amershi et al., 2019): AI drafts; humans validate and sign.
  • Deploy in secure environments: use on-premises or confidential computing, strict role-based access control, and data-redaction pipelines for sensitive items.
  • Establish measurement and red-teaming: define acceptance criteria, run adversarial tests, and log correction metrics.
  • Document AI provenance and role in the workflow: regulatory reviewers and auditors must be able to reconstruct why a sentence appears in a filing and who approved it (Bender et al., 2021).

Implications & insights

AI can materially reduce the time spent on repetitive drafting and on cross-referencing past decisions and regulations. In a resource-constrained industry this could lower costs and accelerate review cycles. But the critical constraint is not model capability alone; it’s trust — demonstrable, auditable trust in outputs that affect public safety. That trust requires engineering for provenance, organizational processes for verification, and regulatory engagement so that regulators and licensees jointly develop safe paths to adoption.

Conclusion & next steps

The sensory shock expressed in the original post reflects healthy skepticism. Training an LLM and letting it author nuclear license documents autonomously would be reckless. However, thoughtfully designed AI-assisted systems — combining retrieval, knowledge graphs, document understanding, and strong human oversight — can speed many parts of the licensing workflow while preserving safety and accountability. The right next steps are small, measurable pilots that focus on traceability and verification, coupled with clear governance and regulator engagement.

Further reading and citations

  • Lewis et al. (2020) — Retrieval-augmented generation techniques for knowledge-intensive tasks.
  • Amershi et al. (2019) — Guidelines for human-AI interaction and human-in-the-loop design.
  • Bender et al. (2021) — Risks and ethical considerations for large language models (“stochastic parrots” critique).
  • Hogan et al. (2021) — Knowledge graph methods for linking heterogeneous information.
  • U.S. Nuclear Regulatory Commission (2020) — NRC rules and guidance on licensing packages and submission requirements.

If you’d like, I can draft a small pilot plan (6–8 weeks) that shows exactly how to prototype a RAG-based assistant for one licensing subsection, including data needs, evaluation metrics, and human approval gates.

References