
Read Notion rows past the 10k cap
Use Notion SDK v5.23.0’s row-iteration helpers to stream large data sources safely, avoid silent 10k-row under-reads, and build more reliable PM automation reports.
Notion SDK v5.23.0 added two helpers for large data sources:
iterateAllDataSourceRows(client, args) and collectAllDataSourceRows(client, args). The version shipped on July 8, 2026, and the npm package page lists v5.23.0 for @notionhq/client. 1 2Use this when a PM workflow can under-read a large Notion data source: roadmap inventories, launch-risk registers, experiment backlogs, or customer-feedback databases where the automation needs every row before it can produce a clean weekly artifact.
The workflow risk is an incomplete read, not just inconvenient pagination. A single
dataSources.query call can stop at 10,000 rows; at that point has_more becomes false and the response carries request_status.type === "incomplete", so plain pagination can stop without returning the full data source. 2 The new helpers work around that cap by partitioning queries on created_time and de-duplicating rows by row ID. 3Prerequisites
| Requirement | Detail |
|---|---|
| SDK version | Install or upgrade @notionhq/client to v5.23.0 with npm install @notionhq/client. 2 |
| Runtime | The package requires Node 18 or later; TypeScript 5.9 or later is optional. 2 |
| Target object | The helper call must include data_source_id, and the helpers accept the same arguments as dataSources.query. 2 |
| Data shape | The helper manages start_cursor and sorts internally, and it combines a caller-provided filter with the lower bound of each time window using and. 4 |
Do not start with
collectAllDataSourceRows just because the name is simpler. The README warns users to confirm that the full result can fit in memory before using collectAllDataSourceRows, and it recommends iterateAllDataSourceRows for very large data sources. 2The pattern: stream first, summarize second
Treat the helper as the ingestion stage for a PM-facing automation. The automation reads every row from the Notion data source, reduces the stream into the few fields the PM actually needs, and writes the output somewhere else: a weekly stakeholder page, a launch-risk digest, a Slack-ready summary, or a CSV for deeper analysis.
The minimal streaming loop is the new default:
import { Client, iterateAllDataSourceRows } from "@notionhq/client";
const notion = new Client({ auth: process.env.NOTION_TOKEN });
const dataSourceId = process.env.NOTION_DATA_SOURCE_ID!;
const ownerCounts = new Map<string, number>();
for await (const row of iterateAllDataSourceRows(notion, {
data_source_id: dataSourceId,
filter: {
property: "Status",
status: { does_not_equal: "Archived" },
},
})) {
const owner = extractOwner(row) ?? "Unassigned";
ownerCounts.set(owner, (ownerCounts.get(owner) ?? 0) + 1);
}
await writeWeeklyRiskDigest(ownerCounts);That loop is enough when the automation can reduce each row as it arrives. A roadmap owner count, launch-risk scan, stale-request detector, or duplicate-candidate finder does not need to hold the whole database in memory.
Use
collectAllDataSourceRows only when the next step genuinely needs an array. For example, a scoring pass that sorts every candidate by priority and then keeps the top 200 can be acceptable if the database size is known and the memory budget is comfortable.import { collectAllDataSourceRows } from "@notionhq/client";
const rows = await collectAllDataSourceRows(notion, {
data_source_id: dataSourceId,
filter: {
property: "Team",
select: { equals: "Growth" },
},
});
const topRisks = rankLaunchRisks(rows).slice(0, 200);
await updateLeadershipBrief(topRisks);Implementation steps
- Replace generic pagination in the read phase. Use
iterateAllDataSourceRowsanywhere the existing script loops throughdataSources.querypages and expects to cover the whole data source. The helper exists because plain pagination can miss rows after the 10,000-row cap. 2 - Keep the PM logic outside the helper. The helper should only deliver rows. Your script should still own field extraction, normalization, scoring, and the final writeback.
- Pass the business filter up front. If the PM artifact only needs active launches, open P0/P1 customer issues, or the current half's roadmap rows, encode that as the
filterargument. The helper will combine that filter with its internal time-window lower bound. 4 - Reduce during iteration where possible. Count, group, score, or emit rows as the stream arrives. This keeps the automation useful for larger databases and follows the SDK guidance to prefer streaming for very large data sources. 2
- Write a bounded output. The article, page, or alert should contain the decision surface, not a mirror of the database. For a weekly PM workflow, that usually means top risks, unowned items, stale rows, or the owners with the highest open load.
Expected outcome
The workflow stops depending on the 10,000-row behavior of a single query. The SDK helper advances through
created_time windows and handles row de-duplication, while the PM-facing code focuses on the artifact: a risk digest, cleanup queue, owner load summary, or roadmap exception report. 3A good smoke test is simple: run the same reducer with
iterateAllDataSourceRows, log the final row count, and compare it with your old pagination script. If the old script stopped at 10,000 and the new script continues, the automation was previously under-reading the source data.Gotchas
The helper is not magic if one minute is too dense. If more than 10,000 rows share the same
created_time value at minute precision, the time window cannot shrink further and the helper can throw. 4 Split that workload with an extra filter, such as status, team, category, or another stable property.Do not mix helper-managed sorting with custom cursor logic. The helpers manage
start_cursor and sorts, so leave the windowing mechanics to the SDK and keep your code at the business-filter layer. 4Treat edge runtimes carefully. A very large scan can still outlive a short execution window. For a large database, prefer a job runner that can stream for long enough, persist checkpoints around the reducer, or split the work by filter before deployment.
Default to streaming for PM automations.
collectAllDataSourceRows is comfortable for scripts that need an array, but it moves the failure mode from pagination to memory. The README's guidance is clear: use the iterator for very large data sources. 2Smallest test
Pick one large Notion data source where a weekly PM automation needs full coverage. Replace the current pagination loop with
iterateAllDataSourceRows, keep the reducer intentionally small, and emit one bounded artifact: owner counts, stale rows, or top launch risks.If that test finds rows beyond the old stopping point, move the helper into the production read phase. If it throws on a dense time window, add a business filter and run the same reducer per segment.
Cover image: AI-generated illustration.
Contenido relacionado
- Inicia sesión para comentar.
