Pagination and partitioning¶

spark-rest-api-reader separates two independent concerns:

Pagination (pagination_type) — how a single partition walks through multiple pages of results until the API says "no more."
Partitioning (partitions / pages) — how the overall pull is split into independent units of work that Spark can fan out across executors.

You can combine them: partition by region and have each region's partition separately paginate through everything on its own, or auto-generate one partition per page number for maximum parallelism when the API doesn't need per-partition pagination at all. This page covers both, and how they compose.

For the record-list/format side of things (records_path, response_format) that pagination stop conditions read from, see Response formats and records.

Pagination (`pagination_type`)¶

Pagination happens within one partition: a partition keeps requesting the next page until its stop condition is met, or until max_pages_per_partition is hit.

`pagination_type`	Extra options
`none` (default)	One request per partition.
`page_number`	`page_param` (default `page`), `start_page` (default `1`), `page_size_param`, `page_size`
`offset_limit`	`offset_param` (default `offset`), `limit_param` (default `limit`), `limit` (default `100`), `start_offset` (default `0`)
`cursor`	`cursor_param` (default `cursor`), `next_cursor_path` (required)
`link_header`	— (follows the `Link` response header)

`none`¶

The default. One request is made per partition and the result is used as-is — no follow-up pages. Useful for endpoints that already return everything in a single response, or as the base case when you're relying on pages (see Partitioning below) to fan out across pages instead.

`page_number`¶

Increments a page-number query parameter on each request.

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "bearer")
    .option("token", api_token)
    .option("pagination_type", "page_number")
    .option("page_param", "page")
    .option("page_size_param", "page_size")
    .option("page_size", "200")
    .load()
)

Option	Default	Description
`page_param`	`page`	Query param carrying the page number.
`start_page`	`1`	First page number requested.
`page_size_param`	—	Query param carrying the page size, if the API accepts one.
`page_size`	—	Value sent for `page_size_param`.

Stop condition: pagination stops when a page comes back empty, checked via records_path the same way records are actually extracted — so a custom envelope's emptiness is read correctly, not just the top-level body or common wrapper keys. It also stops on a non-JSON response body (e.g. a 204 on the true last page) rather than continuing to increment the page number forever.

`offset_limit`¶

Increments an offset query parameter by a fixed page size on each request.

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "api_key")
    .option("api_key_value", api_key)
    .option("pagination_type", "offset_limit")
    .option("offset_param", "skip")
    .option("limit_param", "take")
    .option("limit", "500")
    .load()
)

Option	Default	Description
`offset_param`	`offset`	Query param carrying the offset.
`limit_param`	`limit`	Query param carrying the page size.
`limit`	`100`	Page size requested (and used to detect the last page).
`start_offset`	`0`	First offset requested.

Stop condition: pagination stops when a page returns fewer records than limit, again counted via records_path.

`cursor`¶

Follows a next-cursor value returned inside the response body — the common pattern for APIs that don't support arbitrary offsets (e.g. keyset pagination).

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "oauth2_client_credentials")
    .option("oauth_token_url", "https://auth.example.com/oauth/token")
    .option("oauth_client_id", client_id)
    .option("oauth_client_secret", client_secret)
    .option("pagination_type", "cursor")
    .option("next_cursor_path", "meta.next_cursor")
    .load()
)

Option	Default	Description
`cursor_param`	`cursor`	Query param the next-cursor value is sent back as.
`next_cursor_path`	—	Required. Dot path to the next-cursor value in the parsed response body (JSON/XML only).

Stop condition: pagination stops only when next_cursor_path resolves to null or "". A falsy-but-real value like 0 or false is a legitimate cursor and does not stop pagination — only a genuinely absent value does.

Not available with response_format=csv (there's no envelope to read a next-cursor from) — see Response formats.

Batch cursor pagination vs. streaming

This is page-by-page pagination within one batch read — not the same thing as the incremental, cursor-driven streaming source. See Streaming for the readStream variant, which tracks a high-water-mark field across micro-batches instead.

`link_header`¶

Follows the standard Link: <url>; rel="next" response header, exactly as returned by the API — no query-param bookkeeping needed on your end.

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "bearer")
    .option("token", api_token)
    .option("pagination_type", "link_header")
    .load()
)

Stop condition: pagination stops when the Link header with rel="next" is absent from the response.

Safety valve: `max_pages_per_partition`¶

Option	Default	Description
`max_pages_per_partition`	`10000`	Hard cap on pages fetched by a single partition, regardless of `pagination_type` — a safety valve against runaway pagination loops (e.g. a misbehaving API that never signals "no more").

Partitioning — how the pull fans out¶

Partitioning decides how many independent units of work Spark creates, each of which runs the configured pagination_type on its own.

Option	Description
`partitions`	JSON array of objects; one partition per object, each merged into that partition's query params. A `_url` key inside an object overrides the base URL entirely (e.g. per-resource-id endpoints). Each partition still runs the configured `pagination_type` to walk all of its own pages. Every entry must be a JSON object — a non-object entry raises a `ConfigError` naming the offending index.
`pages`	Integer — auto-generates one partition per page number (`1..pages`, using `page_param`), for maximum parallelism against APIs where pages can be fetched independently. Mutually exclusive with in-partition pagination (each partition fetches exactly its one page).
(neither set)	Single partition; `pagination_type` walks all pages sequentially.

`partitions`: explicit, per-partition query params (or URLs)¶

Each object in the partitions array becomes one Spark partition. Its keys are merged into that partition's query parameters — handy for splitting a pull by region, tenant, category, or any other dimension the API filters on.

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "bearer")
    .option("token", api_token)
    .option("pagination_type", "cursor")
    .option("next_cursor_path", "meta.next_cursor")
    .option(
        "partitions",
        '[{"region": "us"}, {"region": "eu"}, {"region": "apac"}]',
    )
    .load()
)

Here, each of the three partitions (us, eu, apac) independently walks its own cursor chain to completion — pagination and partitioning composing cleanly: partitioning decides what each unit of work covers, pagination decides how each unit walks through its own pages.

_url override: an object can include a _url key to override the base url entirely for that partition, instead of (or in addition to) merging query params — useful for APIs where each partition corresponds to a different resource path rather than a query-string filter, e.g. [{"_url": "https://api.example.com/v1/regions/us/items"}, {"_url": "https://api.example.com/v1/regions/eu/items"}].

Every entry in the array must be a JSON object; a non-object entry raises a ConfigError naming the offending index.

`pages`: auto fan-out across page numbers¶

Instead of one partition paginating sequentially through many pages, pages auto-generates one partition per page number, from 1 to the given count, each requesting its single page via page_param. This is the way to get maximum parallelism against APIs that support arbitrary page numbers and don't need per-partition, in-order pagination at all.

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "bearer")
    .option("token", api_token)
    .option("pages", "20")
    .load()
)

pages is mutually exclusive with in-partition pagination: each generated partition fetches exactly its one page, so pagination_type is not walking multiple pages within any single partition here — the fan-out itself is the pagination.

Neither set¶

If neither partitions nor pages is set, the read runs as a single partition, and pagination_type walks all pages sequentially within it. This is the simplest configuration and a reasonable default until an API's volume or shape calls for explicit fan-out.

Putting it together¶

A useful way to think about the four combinations:

	`pagination_type = none`	`pagination_type` set (page_number/offset_limit/cursor/link_header)
no `partitions`/`pages`	One request, total.	One partition, sequentially walking every page.
`partitions` set	One request per partition entry.	Each partition entry walks all of its own pages independently.
`pages` set	(not applicable — `pages` generates the requests)	One partition per page number, no further pagination within each.

Pagination and partitioning¶

Pagination (pagination_type)¶

none¶

page_number¶

offset_limit¶

cursor¶

link_header¶

Safety valve: max_pages_per_partition¶