API Pagination: Pull Large Datasets Reliably

No API returns a million records in one response. Instead, results come in pages, and a connector has to fetch them one after another. API pagination looks simple until a sync silently misses records, copies the same record twice, or runs for hours and fails at page 812 with no way to resume. These problems usually come from how pages are requested and how the connector reacts when data changes mid-sync. This guide explains the main pagination styles, where connectors lose or repeat records, and how to pull large datasets reliably.

The main API pagination styles

APIs use a few common approaches:

• Offset pagination: request records with a limit and an offset, such as "50 records starting at record 1,000"
• Page number pagination: request page 1, page 2 and so on, with a fixed page size
• Cursor pagination: each response includes a token or record ID marking where to continue, and the next request passes it back
• Link-based pagination: each response includes the URL of the next page, often in a Link header

Stripe's pagination documentation, for example, describes cursor-based pagination using starting_after and ending_before parameters, with a has_more field that tells clients whether more results exist. GitHub's guide to using pagination in the REST API uses link headers, with URLs labelled rel="next", rel="prev", rel="first" and rel="last".

Where offset pagination goes wrong

Offset and page number pagination are easy to use but fragile when data changes during a sync:

• If a record is deleted from an earlier page while you are fetching, every later record shifts back by one, and one record is skipped
• If a record is inserted earlier in the sort order, records shift forward, and one record appears twice
• Large offsets can be slow on some systems, because the server still has to count past every skipped record

Cursor pagination avoids most of these problems because each request continues from a specific record rather than a position.

Follow the API's own pagination

The safest rule is to follow exactly what the API provides:

• If the response gives a next link or cursor, use it as-is rather than building the next URL yourself
• Stop when the API says there are no more pages, such as a missing next link or has_more being false, not when a page happens to be short
• Use a stable sort order, ideally by a unique ID or by creation time plus ID
• Use the largest page size the API allows, to reduce the number of calls

Worked example: following next links

The simplified, untested Python function below follows link-based pagination with the requests library, which exposes parsed Link headers through the response's links property. It assumes each page body is a JSON list. URLs and parameters are placeholders.

import requests


def fetch_all_pages(start_url, params, session: requests.Session):
    url = start_url
    while url:
        response = session.get(url, params=params, timeout=30)
        response.raise_for_status()
        for record in response.json():
            yield record
        # Follow the server-provided next link; stop when there is none
        url = response.links.get("next", {}).get("url")
        params = None  # the next link already contains the query string

A production version would add rate limit handling, retries for temporary errors and checkpointing, described below.

Checkpoint progress

Long syncs fail. A network drop or a deployment at page 812 should not mean starting again from page 1. Store progress as you go:

• After each page is processed successfully, save the cursor or next link
• On restart, resume from the last saved checkpoint
• Make processing idempotent, so a page processed twice, once before a crash and once after, does not create duplicates

Cursors sometimes expire. If a saved cursor is rejected, fall back to an incremental query based on the last record you processed.

Sync incrementally by updated time

Full extracts are rarely needed after the first load. Most connectors should fetch only records changed since the last successful sync:

• Filter by a "modified since" field, where the API supports it
• Sort by modified time plus a unique ID, so records with the same timestamp are handled consistently
• Store the highest modified time and ID processed, and start the next run from there
• Overlap each run by a short window, such as a few minutes, to catch records updated while the previous run was in progress

Deletions need separate handling. Many APIs do not return deleted records in normal queries, so look for a deleted-records endpoint, a change feed or webhooks for deletions.

Verify totals

Pagination bugs are silent, so check the results:

• Compare the number of records fetched with the total count the API reports, if it offers one
• Run periodic reconciliation that compares record IDs or counts between source and target
• Track records per run over time; a sudden drop often means pages are being skipped

Our guide to Deluge pagination shows these principles in Zoho's scripting language, and our guide to the Zoho CRM bulk read API explains when a bulk export is better than paging through records one request at a time.

Respect rate limits while paging

A full sync of a large object can use thousands of requests in a short time. Pace page requests below the provider's limits, read any remaining-allowance headers, and pause when a limit response arrives instead of retrying immediately. If a sync regularly runs into limits, schedule it outside busy hours, raise the page size if the API allows, or switch to a bulk export for the heavy part of the work.

Log the number of pages, records and requests for each run. These figures make it easy to spot when a sync starts taking longer or using more of the allowance than usual, which is often the first sign of data growth or a pagination problem.

When to use bulk export instead

Some APIs offer bulk or export endpoints that produce a file of all matching records asynchronously. They are usually better than paging when:

• You need hundreds of thousands of records or more
• You are doing an initial load or a full reconciliation
• Page-by-page calls would use too much of your rate limit

Use normal pagination for ongoing incremental syncs, and bulk exports for large one-off or periodic loads.

Common API pagination mistakes

• Building next-page URLs manually instead of using the link or cursor provided
• Stopping when a page is shorter than expected, rather than when the API says there are no more
• Using offset pagination on data that changes during the sync
• Not saving progress, so failures mean restarting from the beginning
• Advancing the sync checkpoint before the records were actually written to the target

Frequently Asked Questions

What is API pagination?

It is how APIs split large result sets into smaller pages. Clients request one page at a time using offsets, page numbers, cursors or next links until all results are retrieved.

Why is cursor pagination more reliable than offset pagination?

Cursors continue from a specific record, so inserts and deletes elsewhere in the data do not shift results. Offsets can skip or repeat records when data changes mid-sync.

How do you resume a failed paginated sync?

Save the cursor or next link after each successfully processed page and restart from it. Make processing idempotent so a repeated page does not create duplicates.

How do incremental syncs work with pagination?

Filter by a modified time, sort by modified time plus ID, page through the changes, store the last value processed and start the next run from there with a small overlap.

Pull large datasets without losing records

We build connectors that page reliably, checkpoint progress, sync incrementally and reconcile totals. See our CRM and API integration services, or talk to us about API pagination.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.