One quarter, 263,466 documents: the enterprise QA problem built around time
CorporateBench turns changing enterprise relationships into a large-scale QA test, giving PMs a concrete pilot for deciding whether time-aware structured retrieval beats vector RAG.
An enterprise assistant can retrieve the right company policy and still answer the wrong question when the answer depends on a date. A team assignment, reporting line, or project owner may have changed between two documents. CorporateBench, a paper posted on August 27, 2026, turns that failure mode into a large-scale benchmark for enterprise document question answering. 1
Quick view
| Trend | Fresh signal | Problem it addresses | Evidence boundary | Action window |
|---|---|---|---|---|
| Time-aware, structured enterprise QA | CorporateBench was posted on August 27, 2026. 1 | Answers that require joining documents, relationships, and changing states over time. 2 | The benchmark uses synthetic companies and documents; it gives a controlled stress test, with production performance on confidential corpora still open. 2 | Run a bounded comparison this sprint if your assistant answers questions about ownership, access, staffing, or policy history. |
The product question is precise: when enterprise QA fails at scale, will a larger context window help, or does the product need a structured knowledge layer that records relationships and their time ranges?
The benchmark makes the hidden state visible
CorporateBench simulates four companies ranging from 12 to 10,210 employees. Each company evolves through one 90-day quarter. At the quarter boundary, the simulation adds about 10% new hires, splits teams, and gives earlier relationships end dates. The resulting corpus contains 263,466 generated documents; the largest company, Pound, contributes 232,693 documents, 120,789 entities, and 371,905 relationships. 2
The benchmark covers five tasks: extracting a knowledge base, classifying topics, answering knowledge-base questions, answering topic questions, and answering integrated questions that combine both. The questions come from 250 manually written templates filled with values returned by verified SPARQL queries. This construction lets the authors test whether an answer matches a known state across documents, rather than relying on a loose human impression of relevance. 2
The scale matters because a question such as "Who worked on this project after the reorganization?" can require several pieces of evidence. One message names a project, another links an employee to a team, and a later update closes the earlier team relationship. A top-k vector search ranks passages by semantic similarity; the answer needs a consistent state assembled across those passages.
Why the date belongs in the data model
A standard document index can store the sentence that says Alice reports to Bob. A temporal knowledge layer also stores when that relationship became true, when it ended, when a source stated it, and when the system recorded it. The distinction separates "Who reports to Bob now?" from "Who reported to Bob on March 15?" 3
CorporateBench makes this distinction part of the task. The quarter-end update creates cases in which two documents are needed to recover a person's employment status. The benchmark therefore tests a capability that ordinary retrieval can hide: selecting the relationship whose validity window contains the question's date. 2
A practical implementation can use two clocks for each fact: valid time, when the fact held in the business world, and transaction time, when the product learned it. When a new document contradicts an old fact, the product closes the old validity window and records the new fact with its source. The history stays available for audit while current queries use the open window. Zep describes this pattern as a bi-temporal knowledge graph. 3
The results separate retrieval from representation
CorporateBench evaluates Claude Haiku 4.5, Claude Sonnet 4.5, GPT-5 Nano, GPT-5.1, and Gemini 2.5 F-L across its QA settings. The paper reports F1 scores, where higher values mean more exact matches to the benchmark answers. Each comparison below uses the small company, S, and the extra-large company, XL. 2
- Extraction loses temporal detail first. Claude Haiku 4.5's relationship-extraction F1 falls from 0.845 to 0.494 as the corpus moves from S to XL. Its temporal-relationship F1 falls from 0.470 to 0.173. The drop is larger for the time-qualified relation than for the relation in general. 2
- Vector retrieval weakens with the corpus. GPT-5 Nano's KB QA F1 in the paper's RAG setting falls from 0.527 to 0.244 from S to XL. RAG retrieves text passages before the model forms an answer. 2
- Structured access helps, while scale still matters. Claude Sonnet 4.5's KB QA F1 with structured KB access is 0.767 on S and 0.642 on XL, higher than its RAG scores of 0.322 and 0.215 on the same sizes. The structured setting still declines with scale, and the paper identifies text-to-SQL as a difficult task in its own right. 2
The result supports a product hypothesis: representation and retrieval deserve separate tests. A graph or SQL layer may preserve entities, relationships, dates, and provenance that passage ranking leaves scattered. The model still has to translate a natural-language question into the right structured query and select the right evidence.
The authors also ran 1,000 human judgments on generated emails and negative examples. Reviewers identified the intended relationship correctly in 76.2% of judgments. That check supports the benchmark's controlled construction; the synthetic companies remain a boundary on how directly the scores transfer to a real employer's private corpus. 2
A product path for the next experiment
A PM team can test the hypothesis without replacing its entire search stack.
- Choose one workflow with time-sensitive answers. Start with project ownership, access approval, staffing, or policy history. Write 30 to 50 current and as-of-date questions whose answers a domain owner can verify.
- Preserve evidence at ingestion. Split each document into addressable text units. Extract entities, relationships, and claims, then retain the source passage and ingestion event beside every extracted fact. Microsoft Research's GraphRAG documentation describes this pattern, including text units, entity and relationship extraction, hierarchical community summaries, and global, local, DRIFT, and basic search modes. 4
- Add validity windows before adding more model context. Store
valid_from,valid_to,observed_at, andrecorded_atfor facts whose business meaning can change. Keep old states queryable so an answer can distinguish current from historical truth. 3 - Route questions by the answer shape. Use vector or full-text search for a passage-specific question, a structured query for an entity-and-date question, and a hybrid path for questions that need both. Microsoft GraphRAG documents separate global, local, DRIFT, and basic retrieval modes; the routing rule here is a pilot design, not a CorporateBench result. 4
- Gate answers on evidence and time. Log the selected facts, validity windows, source passages, generated query, latency, and cost. Page a human when the query returns conflicting open facts, an empty validity window, or evidence that fails the domain owner's check.
For a plain-language orientation to the building blocks, SurrealDB's April 27, 2026 guide covers entity and relationship extraction, deduplication, ontology alignment, and combining vector, graph, and full-text search. It is an implementation explainer, not evidence about CorporateBench's scores. 5
Loading content card…
A recent practitioner framing from Jing Wang uses "context graph" for a layer that combines business context with runtime state, evidence, decisions, and governance. That vocabulary is useful for product scoping; it is a practitioner view rather than an evaluation of CorporateBench. 6
Loading content card…
The decision gate
CorporateBench makes one capability worth testing now: an enterprise assistant should be able to answer over a changing business state, with evidence that a user can inspect. The paper gives a controlled stress test for that capability, while the synthetic setting leaves the production transfer question open.
The first gate can be small. Compare the current vector-RAG path with a time-aware structured layer on the same workflow and the same question set. Require improvement in current-answer accuracy, as-of-date accuracy, and evidence recall. Track latency, model and storage cost, unanswered conflicts, and the rate of facts whose validity window was wrong. Expand traffic only when the structured path beats the existing path on held-out questions and a domain owner can trace each accepted answer to the right source and date.
That experiment answers the PM question more directly than a larger context window benchmark: does the product need more text, or does it need a model of how enterprise facts change?
References
- 1
- 2CorporateBench HTML paper
arxiv.org
- 3What Is a Temporal Knowledge Graph?
getzep.com
- 4Microsoft GraphRAG documentation
microsoft.github.io
- 5
- 6
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.