Which metadata fields matter for AI retrieval

What is metadata in an AI knowledge base?

Metadata is structured information about a source: what it is, where it came from, who owns it, whether it is approved and where in it a passage sits. The document text is the content. Metadata is the context around it that a retrieval system uses to filter, and that a reviewer uses to check.

Take an evaluation report in a programme library. Its metadata might read: source ID SRC-042; title "2025 Mid-term Evaluation of a Youth Skills Programme"; document type evaluation report; organisation the commissioning foundation; date published 14 March 2025; project Youth Skills Programme; status approved; access level internal; source link the file's address in the shared drive.

Those fields tell the system when this report is a valid source for a question. They also help the reviewer. If an AI answer cites "SRC-042, page 18", the reviewer can open the right file and check the claim.

Metadata is one part of the wider preparation work in how to get your documents ready for AI retrieval. This guide goes field by field.

It also helps to keep metadata apart from similar terms:

• File name: a human-readable label, which changes when someone renames the file.
• Folder path: where the file is stored, not what it is.
• Tags: short labels for grouping, one kind of metadata.
• Embeddings: numerical representations of meaning used for semantic search.
• Source ID: a stable identifier that does not change when the file is renamed or moved.

Why is semantic search not enough on its own?

Semantic search finds text that means something similar to the question, even when the wording differs. It does not know whether that text is current, approved, confidential, from the right project or fit for a report.

Suppose a policy team asks: what does the 2025 material say about service delivery barriers in rural districts? A system relying only on similarity might return a passage from an old report, an unapproved draft, a different province, a confidential interview transcript or a summary with no source attached. Each of those reads as a plausible answer.

Metadata narrows the search before the model drafts anything. The system can restrict retrieval to sources where the project, geography and reporting period match, the status is approved, the access level suits the user, and a locator exists for checking.

How do retrieval tools use metadata?

Most retrieval platforms store metadata as key-value pairs alongside each file or chunk and let you filter on them at query time. The details differ by platform, so check the documentation of the one you use.

Three examples, checked against current vendor documentation:

• OpenAI file search and Retrieval API. Each file in a vector store can carry attributes, up to 16 keys, and searches can be filtered on them with comparison and compound filters (OpenAI, Retrieval guide, 2026).
• Amazon Bedrock Knowledge Bases. Documents can carry metadata attributes, supplied for files in Amazon S3 through a companion metadata file, and queries can filter on them with operators such as equals, greater than and in (Amazon Web Services, Bedrock user guide, 2026).
• Pinecone. Records can carry metadata key-value pairs, and queries can include a metadata filter that limits results to matching records (Pinecone, Filter by metadata, 2026).

For a non-technical team, the point is simple. Metadata lets the system search inside the right evidence set, not just across text that sounds similar. The fields have to exist, and be filled in consistently, before the tool can use them.

Which core fields should every AI knowledge base start with?

Start with the fields that help the team retrieve, filter, cite, protect and review the material. Most libraries need the baseline below before adding specialist fields.

What you add depends on use. An internal policy library may need effective date, policy owner and department. A research project may need stakeholder group, geography, research question and consent status. A public submissions analysis may need respondent category, policy section and coding status.

What is the difference between document-level and chunk-level metadata?

Document-level metadata describes the whole file and helps the system choose which sources to search. Chunk-level metadata describes the exact excerpt retrieved and helps a reviewer find and cite it. Evidence-heavy work needs both.

Retrieval tools split long documents into smaller pieces, called chunks, before searching. A summary of the whole document may be enough for light internal search. It is not enough when a report writer needs to verify a claim, insert a quote or show which source supports a recommendation.

Chunk-level fields differ by format:

• Transcripts: speaker, timestamp, stakeholder group, question number, theme, quote ID.
• Spreadsheets: sheet name, row ID, column name, indicator, reporting period, geography.
• Policy documents: section heading, clause number, page number, version.

Which fields do evidence-heavy reports need?

Research, evaluation, donor reporting and public-sector teams need fields that connect a source to the output it supports. They are rarely looking for a file. They are looking for the evidence behind a finding.

These fields let a public submissions team filter comments by stakeholder group, issue and policy section. They let a programme team show which findings rest on field interviews, which on administrative data, and which on a handful of observations.

Which fields protect drafts, confidential and sensitive material?

Status, version, confidentiality, consent and intended-use fields stop the system treating every file as equally safe or equally reliable. They are what keeps a draft out of a published answer and a confidential transcript out of a shared one.

With these fields, a team can let the system use only approved sources for outward-facing outputs, while internal users still search raw material in a separate space. Access fields belong in the source register from the start. Adding them after the knowledge base is live means re-checking every file.

How do you start without overbuilding?

Start with a source register, not a technical build. A spreadsheet, shared list or simple database that records the core fields for each source is enough to make a messy library searchable and checkable, and it becomes the design blueprint for any later AI knowledge base.

We cover the register's layout and upkeep in how to build a source register for an evidence-heavy report. For evidence libraries, add a second table for extracted evidence items, linked to the register by source ID. This starter schema is a reasonable first version:

SOURCES table
source_id | file_name | title | document_type | organisation | owner
date_published | date_added | version | project | geography | theme
status | access_level | source_link | locator_available
review_owner | last_reviewed

EVIDENCE table
evidence_id | source_id | quote_id | locator (page / row / timestamp)
stakeholder_group | geography | method | theme | subtheme
research_question | finding_id | evidence_strength
limitation_note | consent_or_use_restriction | coding_status

Then work through the set in order:

1. List the source material.
2. Assign source IDs.
3. Define document types from a fixed list.
4. Add dates, owners and project fields.
5. Add theme fields from a controlled vocabulary.
6. Add status and confidentiality fields.
7. Add locators for evidence-heavy material.
8. Write a short data dictionary that defines each field and its allowed values.
9. Test retrieval with real questions.
10. Refine the fields based on searches that failed.

Use questions the team actually asks: which approved sources support this finding; what district-level interviews say about implementation barriers; which 2025 reports mention safeguarding concerns. When the system returns the wrong material, look for the cause in the metadata first: a missing field, an inconsistent tag, a vague document type or no locator. Our guide on how to test an AI knowledge base sets out a full test plan.

The register also gives you a quality check before ingestion: required fields complete, sensitive files labelled, no draft marked as approved. The same discipline of every record linked to its source underpins the TRC Evidence Archive. If you want the register, schema and retrieval set up for you, we offer a source-linked evidence system, built and handed over.

Common mistakes

• No stable source IDs. Without them, answers cannot be traced and records cannot be updated cleanly. The ID should not change when the file name does.
• Relying on folder names. If "Approved" or "2025" exists only in the folder path, it may be lost or applied inconsistently at indexing.
• Inconsistent tags. "Schools", "education" and "learning" may mean the same thing to people but behave as three different filters. Use a controlled vocabulary.
• Mixing several things in one field. Source type, project, theme and output type each need their own field.
• No status or confidentiality field. The system then has no way to keep drafts and restricted material out.
• No locators. Without a page, row or timestamp, a reviewer cannot check the answer quickly.
• Too many fields too soon. Sixty fields with twelve filled in is harder to maintain than fifteen filled in well.
• No data dictionary. Fields drift when nobody has written down what each value means.