Briefing note
Key takeaways
- The approved manual, technical order, or procedure remains the source of truth.
- A useful citation must open the correct revision, location, and supporting passage—not merely name a document.
- Revision, applicability, warnings, tables, and figure relationships must survive ingestion and retrieval.
- The best evaluation questions come from real maintainer workflows and known failure modes.
The problem is evidence retrieval, not document chat
A maintainer often begins with a symptom, fault code, component, or configuration—not the publication number and exact phrase used by the author. The useful system translates that field language into a search across the approved library, returns the most relevant procedure and context, and keeps the evidence attached to the answer.
The Army's AI-Assisted Maintenance program describes workflows for operations, maintenance, troubleshooting, general questions, and preventive maintenance checks and services. Its public guidance is explicit that AI assists Soldiers in using authoritative technical manuals and that the technical manual remains the source of truth. That is the right boundary for a maintenance knowledge assistant.
Design objective
Help the person reach and understand the governing evidence faster. Do not convert a generated answer into a substitute publication or an unreviewed maintenance authority.
Preserve technical structure during ingestion
Generic PDF extraction can flatten the very structure a maintainer needs. Headers detach from steps, tables become scrambled text, notes lose their targets, and page numbers drift from the rendered file. Ingestion should preserve both searchable text and a stable path back to the original source view.
- Publication identifier, title, revision, effective date, status, and owning authority.
- Equipment model, variant, serial range, configuration, subsystem, and other applicability metadata.
- Section hierarchy, task identifier, page or digital location, paragraph, and figure or table references.
- Warnings, cautions, notes, prerequisites, tools, consumables, limits, and follow-on conditions.
- Supersedence and cancellation relationships so an obsolete source is not presented as current.
- Distribution and access attributes that can be enforced before retrieval, not after generation.
Scanned pages require OCR quality checks, but OCR text should not replace the preserved page image. When a symbol, diagram, or layout matters, the operator needs a route back to the rendered source and surrounding context.
Retrieve the procedure with its qualifiers
Semantic similarity alone is not enough for technical material. Retrieval should combine exact identifiers and fault codes, lexical matches, semantic matches, metadata filters, and reranking. The query may need expansion from a local term to the manual's terminology, but that expansion should never broaden access beyond the user's authorized pack.
Chunking must respect procedure boundaries. If a step is retrieved without the warning immediately above it, or a limit without its unit and applicability, the returned context is incomplete. Systems should retrieve neighboring sections when they qualify the selected passage and should surface conflicting or multiple applicable sources rather than silently choosing one.
A citation is a verification path
A document title at the bottom of an answer is attribution, but it may not be enough for verification. A strong citation identifies the publication and revision, points to the page or structured location, shows the supporting passage, and opens the local source. The operator should be able to decide whether the cited evidence actually supports the statement without repeating the entire search.
- Statement-level support: material claims map to one or more retrieved passages.
- Source identity: publication number, title, revision, and authority are visible.
- Stable location: page, section, task, or paragraph opens correctly in the preserved source.
- Context: warnings, prerequisites, exceptions, and applicability are available with the cited passage.
- Separation: the interface distinguishes source language from generated explanation or summary.
Cited does not automatically mean correct
A model can attach a real citation that only partially supports its sentence, miss a controlling exception, or combine two passages incorrectly. Citation correctness must be evaluated, and users must be able to inspect the source.
Build around the maintainer's workflow
The interface should begin with the equipment and situation known to the user. Ask for configuration details when they affect applicability. Return a concise orientation, the governing references, and safe next navigation steps. If the evidence does not support a definitive answer, say so and identify the information or escalation required.
- Confirm equipment, configuration, symptom, and operating state.
- Retrieve authorized evidence and identify the governing publication revision.
- Present a short sourced orientation with explicit uncertainty or conflicts.
- Let the maintainer open the cited task, warning, table, or figure in context.
- Keep execution, sign-off, and return-to-service authority in the approved maintenance process.
- Capture unresolved questions and useful feedback without silently rewriting the approved source.
Support knowledge continuity without inventing doctrine
Formal publications are only part of operational knowledge. Unit SOPs, approved troubleshooting cards, lessons learned, and escalation contacts help people apply them. The Army's VICTOR initiative is aimed at reducing the burden of searching disconnected knowledge silos and bringing authoritative Army knowledge and lessons learned to the point of need. DARPA's KMASS program likewise highlights knowledge loss during rotations and the need to deliver locally contextualized information to technical experts.
A knowledge assistant should label these source classes and preserve their precedence. A local troubleshooting card can help a technician collect evidence or understand an escalation threshold, but it should not masquerade as an official procedure or overwrite a controlling manual.
Evaluate against real maintenance questions
A useful test set includes routine questions, ambiguous symptoms, terminology mismatches, configuration-specific procedures, superseded publications, cross-page warnings, tables, scanned pages, and questions that the approved corpus cannot answer. Subject-matter experts should establish the expected source and assess both retrieval and the final response.
- Did retrieval include the governing source and all controlling qualifiers?
- Did the answer remain within the evidence and distinguish inference from source content?
- Did citations open the right location in the right revision?
- Did access controls operate before documents entered model context?
- Did the system abstain or request clarification when equipment applicability was unknown?
- Could the operator complete verification faster without skipping required review?
DoD's AI ethical principles place responsibility and judgment with personnel and call for traceable, reliable, and governable systems. Testing should therefore include the human workflow: whether a user understands the system's limits, can reach the evidence, and can stop or disregard an unsupported recommendation.
What to require from a technical manual assistant
- A documented source-authority and revision model.
- Local source viewing with stable page or task citations.
- Retrieval filters for equipment applicability and user authorization.
- Evidence that warnings, tables, figures, and cross-page procedures survive ingestion.
- A defined abstention and escalation behavior.
- Representative evaluation results reviewed by qualified subject-matter experts.
- A controlled process for approving, replacing, and rolling back knowledge packs.
- Clear language that the assistant supports people and does not replace approved maintenance authority.
Primary references
Official sources
These sources support the technical and policy context in this guide. Product-specific statements describe design principles, not a certification or authorization claim.
- 01AI-Assisted MaintenanceU.S. Army Communications-Electronics Command Integrated Logistics Support Center
- 02U.S. Army Combined Arms Command advances AI-powered knowledge platform, VICTORU.S. Army
- 03Don't just ask a chatbot. Have it push useful info right when it's needed.Defense Advanced Research Projects Agency
- 04Artificial Intelligence Risk Management Framework (AI RMF 1.0)National Institute of Standards and Technology
- 05DoD Ethical Principles for Artificial IntelligenceU.S. Department of Defense Chief Digital and Artificial Intelligence Office