Request & auth¶

These are the options that shape every HTTP request the reader makes: the URL and headers, timeouts and retries, TLS and proxy settings, and how the request gets authenticated. They apply the same way regardless of response format, pagination style, or whether you're doing a batch or streaming read.

Request basics¶

Option	Default	Description
`url`	required	Base URL of the GET endpoint.
`headers`	`{}`	JSON object of static request headers.
`query_params`	`{}`	JSON object of static query-string params sent on every request.
`timeout`	`30`	Per-request timeout in seconds.
`max_retries`	`3`	Retries on transient failures (connection errors + `retry_status_codes`).
`backoff_factor`	`0.5`	Exponential backoff factor between retries.
`retry_status_codes`	`[429,500,502,503,504]`	JSON array of status codes to retry (entries are coerced to integers; a non-numeric entry raises a `ConfigError`). `Retry-After` is honored when present, regardless of status code.
`verify_ssl`	`true`	Set `false` to skip TLS verification (not recommended).
`ca_bundle_path`	—	Path to a custom CA bundle; overrides `verify_ssl` when set.
`proxies`	`{}`	JSON object, e.g. `{"https": "http://proxy:8080"}`. Only one proxy is used per client (prefers `https`, then `http`, then any key given) — httpx applies a single proxy to all requests rather than mounting per scheme.
`requests_per_second`	—	Client-side throttle applied per partition.

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("headers", '{"Accept": "application/json"}')
    .option("query_params", '{"region": "us-east"}')
    .option("timeout", "60")
    .option("max_retries", "5")
    .option("requests_per_second", "10")
    .load()
)

Auth (`auth_type`)¶

auth_type selects how the request is authenticated. Every variant below is an independent, copy-pasteable snippet in examples/auth_examples.py — see Examples for the full annotated tour.

Secrets: resolve secrets (e.g. dbutils.secrets.get(...)) on the driver before passing them into .option(...). Options are plain strings shipped to executors as part of the reader, and dbutils isn't available on executors — see Getting started for the full rationale.

`none` (default)¶

No authentication at all.

`auth_type`	Extra options
`none`	—

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/public-items")
    .load()
)

`basic`¶

HTTP Basic auth.

Option	Description
`username`	Basic auth username.
`password`	Basic auth password.

username = dbutils.secrets.get("api-scope", "username")
password = dbutils.secrets.get("api-scope", "password")

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "basic")
    .option("username", username)
    .option("password", password)
    .load()
)

`bearer`¶

A static bearer token, sent as Authorization: Bearer <token>.

Option	Description
`token`	The bearer token.

token = dbutils.secrets.get("api-scope", "api-token")

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "bearer")
    .option("token", token)
    .load()
)

`api_key`¶

An API key sent either as a header or as a query parameter.

Option	Default	Description
`api_key_value`	—	The API key.
`api_key_name`	`X-API-Key`	Header or query param name, depending on `api_key_location`.
`api_key_location`	`header`	`header` or `query`.

As a header (default):

api_key = dbutils.secrets.get("api-scope", "api-key")

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "api_key")
    .option("api_key_value", api_key)
    .option("api_key_name", "X-Api-Key")  # header name; api_key_location defaults to "header"
    .load()
)

As a query parameter:

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "api_key")
    .option("api_key_value", api_key)
    .option("api_key_name", "api_key")  # query param name
    .option("api_key_location", "query")
    .load()
)

`header`¶

The escape hatch for custom or non-standard auth schemes — a JSON object merged into request headers verbatim. Use it for a vendor-specific signature header computed ahead of time, or when multiple headers are required together.

Option	Description
`auth_headers`	JSON object merged into request headers verbatim.

signing_key = dbutils.secrets.get("api-scope", "signing-key")

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "header")
    .option("auth_headers", f'{{"X-Signing-Key": "{signing_key}", "X-Client-Id": "acme-corp"}}')
    .load()
)

`oauth2_client_credentials`¶

RFC 6749 client_credentials grant. Tokens are fetched lazily per executor process and cached until ~30 seconds before expiry, then refreshed automatically — no manual refresh logic needed.

Option	Default	Description
`oauth_token_url`	—	Token endpoint URL.
`oauth_client_id`	—	Client ID.
`oauth_client_secret`	—	Client secret.
`oauth_scope`	—	Optional scope; omit if the API doesn't use scopes.
`oauth_audience`	—	Optional audience parameter.
`oauth_extra_params`	—	JSON object of any additional params to send in the token request.
`oauth_client_auth_mode`	`basic`	`basic` or `body` — whether `client_id`/`client_secret` are sent as HTTP Basic auth on the token endpoint or in the request body.
`oauth_token_field`	`access_token`	Field name in the token response holding the access token.
`oauth_expires_in_field`	`expires_in`	Field name in the token response holding the token's lifetime in seconds.

client_id = dbutils.secrets.get("api-scope", "client-id")
client_secret = dbutils.secrets.get("api-scope", "client-secret")

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "oauth2_client_credentials")
    .option("oauth_token_url", "https://auth.example.com/oauth/token")
    .option("oauth_client_id", client_id)
    .option("oauth_client_secret", client_secret)
    .option("oauth_scope", "items.read")  # optional, omit if the API doesn't use scopes
    .load()
)

Some auth servers expect client_id/client_secret in the token request body instead of HTTP Basic auth on the token endpoint — set oauth_client_auth_mode to body for those (default is basic):

.option("oauth_client_auth_mode", "body")

Next steps¶

Response formats — how the response body gets parsed and where records are extracted from once the request comes back.
Pagination & partitioning — how a single partition walks through multiple pages, and how the overall pull fans out across Spark.
Error handling — catching in-band errors that a 200 OK response can still be hiding.

Request & auth¶

Request basics¶

Auth (auth_type)¶

none (default)¶

basic¶

bearer¶

api_key¶

header¶

oauth2_client_credentials¶