Simple definition
A method APIs use to return large data sets in smaller pages or batches instead of sending everything in one response.
Pagination is fundamental to security connectors because alerts, incidents, findings, assets, users, indicators, and audit events can easily exceed one API response. The connector must understand exactly how the provider signals the next page and how result ordering behaves while data changes.
Common API pagination models
Page number
Requests explicit pages such as page 1, page 2, and page 3.
Offset and limit
Requests a defined number of records beginning at a numeric offset.
Cursor
Uses an opaque or ordered cursor representing the current position.
Continuation token
The server returns a token that must be supplied with the next request.
Next link
The response provides a full or relative URL for the next page.
Keyset pagination
Uses the last returned ordered key or value as the boundary for the next request.
Why pagination matters in security connectors
- Security APIs often contain more records than a single response can safely return.
- Ignoring pagination silently produces incomplete alert, incident, asset, or log collection.
- Changing data during a long query can cause some pagination methods to skip or repeat records.
- Each additional page consumes rate-limit and processing capacity.
- Pagination failures need retry behavior that does not restart an entire large sync unnecessarily.
- Checkpointing should advance only after the required pages in the current sync window are processed safely.
How API pagination works
Define the query window
Choose filters, ordering, start time, checkpoint, page size, and other source parameters.
Request the first page
Send the initial call and validate the returned records and pagination metadata.
Capture continuation state
Read the next cursor, token, offset, page number, or link exactly as the API defines it.
Process and continue
Handle each page while preserving query semantics and retrying transient failures when appropriate.
Detect completion
Stop when the provider's documented end condition is reached.
Commit sync progress
Advance durable checkpoint state only after the full required window has been safely processed.
Pagination failure modes to design for
Missed pages
The connector fails to follow the next token or stops before the API's end condition.
Duplicate records
Changing result sets or overlap cause the same object to appear on more than one page.
Expired token
Provider continuation tokens can have lifetimes or session assumptions.
Unstable ordering
Records inserted during collection can shift offset-based page boundaries.
Rate limits
Large page counts increase API request consumption and throttling risk.
Mid-sync failure
A later page fails after many earlier pages have already been processed.
Common security API pagination use cases
Alerts
Retrieve large alert histories or new-alert windows from SIEM, XDR, email, and security APIs.
Incidents
Collect incident and case records across many pages.
Vulnerabilities
Retrieve findings, assets, CVEs, and scanner results from high-volume APIs.
Cloud assets
Enumerate accounts, resources, findings, policies, and configurations.
Identity objects
Collect users, groups, sign-ins, sessions, devices, and directory changes.
Threat intelligence
Retrieve indicator feeds, observables, campaigns, and enrichment data incrementally.
Pagination vs Checkpointing vs Polling
| Concept | Primary purpose | Example |
|---|---|---|
| Pagination | Retrieve all pages in one large result set. | Follow a next cursor until the current alert query returns no more pages. |
| Checkpointing | Persist progress across sync runs and failures. | Store the last safely processed timestamp and record ID. |
| Polling | Repeatedly check the source for new or changed data. | Run the alerts query every few minutes using the last checkpoint. |
Key technical considerations
Follow provider semantics exactly
Do not assume page, offset, token, cursor, and next-link models are interchangeable.
Retry the failed page safely
Avoid unnecessarily restarting a large sync when only one later request failed.
Deduplicate where needed
Changing datasets and overlap can produce repeated objects across boundaries.
Respect rate limits
Choose page size and request pacing with provider quotas in mind.
Use stable query windows
Time filters and ordering reduce movement while paging through changing data.
Observe page progress
Record page count, records retrieved, continuation state, latency, retries, and incomplete-sync failures.
Benefits and common problems
Benefits
- Complete retrieval of large API datasets.
- Controlled memory and response sizes.
- Scalable incremental connector processing.
- Compatibility with API limits.
- Clear progress across large sync windows.
Common problems
- Stopping after the first page.
- Assuming offset pagination is stable while data changes.
- Losing the continuation token during retry.
- Advancing the checkpoint before all pages finish.
- Page sizes that create rate-limit or timeout problems.
How ForshTec builds pagination into security connectors
ForshTec engineers provider-specific pagination together with checkpointing, retries, rate-limit handling, deduplication, and state recovery so large security datasets are collected completely.
- Cursor, token, page, offset, and next-link pagination
- Checkpoint-safe page processing
- Retry and backoff for mid-sync failure
- Rate-limit-aware page sizing
- Duplicate-boundary handling
- Pagination diagnostics and regression tests
Have an API that returns only part of the dataset?
ForshTec can implement and test the pagination model all the way through completion, retries, state persistence, and changing-data edge cases.
Talk to ForshTecFAQs
What is API pagination?
API pagination divides a large result set into smaller pages or batches that the client retrieves through multiple requests.
What are common pagination types?
Page-number, offset/limit, cursor, continuation-token, next-link, and keyset pagination are common models.
Why can offset pagination create duplicates or gaps?
If records are inserted, deleted, or reordered while paging, later numeric offsets can refer to a different set of records.
How does pagination work with checkpointing?
Pagination retrieves all pages in the current query window; checkpointing stores the safe position from which a later sync should resume.
Should a connector retry an entire paginated sync after one page fails?
Not necessarily. Good designs preserve enough state to retry safely without repeating more work than required.
Why does page size matter?
Larger pages reduce request count but can increase latency, payload size, memory use, timeout risk, and per-request processing cost.