Simple definition
Saving the last successfully processed point in a data sync so the integration can continue without duplicates or data loss.
The checkpoint may be a timestamp, cursor, sequence number, object ID, page token, or composite state value. The important part is that it represents progress the connector can safely trust after a restart.
What does checkpointing do?
Resume safely
Continues after crashes, deployments, token failures, or source outages.
Reduce data gaps
Keeps a durable reference to the last safely completed source position.
Limit duplicates
Avoids re-reading the full history while allowing controlled overlap.
Support polling
Persists state between scheduled collection cycles.
Scale large syncs
Works with pagination and batching for high-volume security data.
Improve diagnostics
Makes sync lag, last success, and recovery position observable.
Why checkpointing matters
- Security integrations often run continuously for months or years and must survive routine restarts.
- Without durable state, a connector may restart too far ahead and lose data or too far behind and create large duplicate volumes.
- Checkpoint state helps separate source collection progress from transient worker or deployment state.
- Reliable checkpoints make retry and recovery behavior predictable for support teams.
- Per-tenant or per-source checkpoint isolation prevents one failed stream from corrupting another.
- Checkpoint age is a useful operational signal for detecting stalled ingestion.
How checkpointing works
Load current state
Read the durable checkpoint for the source, tenant, object type, or stream.
Request the next window
Use the state with source filters, pagination, and any required overlap.
Process records
Validate, normalize, deduplicate, and deliver the returned records.
Confirm durable success
Ensure the batch is accepted by the next stage before claiming progress.
Advance state
Persist the new position atomically or with equivalent failure-safe logic.
Resume later
The next run starts from the saved state and repeats the process.
Common checkpoint types and pitfalls
Timestamp
Simple and common, but boundary events and late arrivals need care.
Cursor
An opaque provider-issued position in a result set or change feed.
Sequence or offset
Tracks ordered stream or event positions.
Record ID
Useful when IDs are stable and monotonically ordered.
Composite state
Combines timestamp plus ID or multiple positions to avoid ambiguity.
Overlap window
Re-reads a small window to protect against late or same-time events.
Common checkpointing use cases
SIEM ingestion
Remember the last event retrieved from a product or cloud API.
Cloud findings
Track changes or finding updates across repeated security posture syncs.
Vulnerability sync
Resume vulnerability collection without replaying the full scanner history.
Identity events
Collect sign-ins, audit events, or directory changes incrementally.
ServiceNow sync
Track bidirectional record positions or update windows safely.
Threat intelligence
Resume indicator or feed collection across pagination and polling cycles.
Checkpointing vs Pagination vs Polling
| Concept | Primary purpose | Example |
|---|---|---|
| Checkpointing | Persist progress across runs and failures. | Store the last processed timestamp plus event ID. |
| Pagination | Break one large result set into manageable pages. | Follow a next-page token until the current query is complete. |
| Polling | Request new or changed data repeatedly on a schedule. | Call an alerts endpoint every few minutes using the checkpoint as the starting point. |
Key technical considerations
Durable storage
Keep state outside ephemeral worker memory so redeployments do not lose progress.
Atomic advancement
Never save state beyond records that have actually completed processing.
Per-stream isolation
Use separate state when tenants, sources, or object types progress independently.
Retry compatibility
Failed batches must be safe to repeat without corrupting the checkpoint.
State versioning
Checkpoint formats may need migration when connector logic or APIs change.
Observability
Expose checkpoint age, lag, last success, retries, and processed record counts.
Benefits and common problems
Benefits
- Reliable resume after failures and deployments.
- Lower risk of permanent data gaps.
- Reduced unnecessary reprocessing.
- Better ingestion-lag visibility.
- Scalable incremental collection.
Common problems
- Advancing state before delivery succeeds.
- Timestamp-only logic that misses boundary events.
- One global checkpoint for independent sources.
- No overlap for late-arriving records.
- Checkpoint state stored only in ephemeral memory.
How ForshTec helps with connector checkpointing
ForshTec builds security connectors with durable sync state, overlap handling, pagination, retries, idempotency, and production observability.
- Timestamp, cursor, sequence, and composite checkpoint design
- Polling and pagination architecture
- Deduplication and controlled overlap windows
- Retry-safe and idempotent processing
- Multi-tenant state isolation
- Connector health, lag, and support diagnostics
Need a connector that resumes cleanly after failure?
ForshTec can design checkpointing around the actual consistency, pagination, and rate-limit behavior of your source APIs.
Talk to ForshTecFAQs
What is checkpointing in a security data integration?
Checkpointing saves the last successfully processed position so the integration can resume from a known safe point.
What values can be used as checkpoints?
Common values include timestamps, cursors, IDs, sequence numbers, stream offsets, page tokens, and composite combinations.
When should a connector update its checkpoint?
Normally only after the corresponding data has been processed or durably accepted by the next stage.
Does checkpointing eliminate duplicates?
No. Reliable connectors may intentionally overlap collection windows and use deduplication or idempotent writes.
Why can timestamp checkpointing miss events?
Events can share timestamps, arrive late, differ in precision, or be returned out of strict order.
How is checkpointing different from pagination?
Pagination manages pages inside one result set; checkpointing persists progress across runs, restarts, and failures.