Skip to content

Getting started

This walks through registering the data source, defining a schema, reading a paginated REST API into a DataFrame, and writing the result to Delta — followed by a conceptual map of the options you'll configure for any given API.

Quick start

Register RestApiDataSource once per Spark session — typically at the top of a job or notebook:

from pyspark.sql.types import IntegerType, StringType, StructField, StructType

from spark_rest_api_reader import RestApiDataSource

spark.dataSource.register(RestApiDataSource)

Define a schema

Schemas follow the structs pattern: nested objects are declared as real StructType/ArrayType columns, not flattened into JSON strings. Build one with pyspark.sql.types:

items_schema = StructType(
    [
        StructField("id", IntegerType()),
        StructField("name", StringType()),
        StructField("status", StringType()),
    ]
)

For nested fields, arrays, inferred schemas, or type coercion details, see Schema.

Read

Pass the schema to .schema(...), point url at the endpoint, pick an auth_type, and pick a pagination_type:

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "bearer")
    .option("token", dbutils.secrets.get("api-scope", "api-token"))
    .option("pagination_type", "page_number")
    .load()
)

Write to Delta

The result is an ordinary DataFrame — write it wherever a batch job would write anything else:

df.write.format("delta").mode("append").saveAsTable("main.bronze.items_raw")

That's the whole batch shape: register once, describe the API through options, read, write. See Examples for a fuller annotated version of this same script, including alternative pagination styles and partitioning for parallelism.

Secrets handling

Resolve secrets — dbutils.secrets.get(...) or any other secret lookup — on the driver, before passing the value into .option(...):

api_token = dbutils.secrets.get("api-scope", "api-token")

df = (
    spark.read.format("rest_api")
    .option("url", "https://api.example.com/v1/items")
    .option("auth_type", "bearer")
    .option("token", api_token)
    .load()
)

Options are plain strings that get shipped to executors as part of the reader. Do not put a dbutils call anywhere reachable from executor code — dbutils isn't available there. Resolve the secret to a string on the driver first, then hand that string to .option(...).

How it's organized

The option surface breaks down along a handful of independent axes. Once you know which axis an option belongs to, picking the right one for a new API is mostly a matter of reading off what that API needs.

  • Auth (auth_type) — how the request is authenticated: none, basic, bearer, api_key, header (a raw escape hatch), or oauth2_client_credentials. See Request & auth.
  • Pagination (pagination_type) — how a single partition walks through multiple pages of results until the API says "no more": none, page_number, offset_limit, cursor, or link_header. See Pagination & partitioning.
  • Partitioning (partitions / pages) — how the overall pull is split into independent units of work that Spark can fan out across executors, orthogonal to pagination. See Pagination & partitioning.
  • Response format (response_format) — how the response body is parsed: json (default), csv, or xml. Everything downstream (records extraction, schemas, pagination stop-conditions) works the same on all three. See Response formats.
  • Records extraction (records_path, flatten) — how to find the list of records inside the parsed response body, and whether to flatten nested objects into dotted column names. See Response formats.
  • Schema — supply a StructType via .schema(...) (or the schema option), or set infer_schema=true to sample and infer one, Auto Loader-style. See Schema.
  • Streaming — an incremental variant driven by a cursor field in the response (e.g. updated_at), read via spark.readStream instead of spark.read. See Streaming.

Partitioning and pagination are independent: you can partition by region (partitions) and have each region's partition separately paginate through everything on its own (pagination_type), or auto-generate one partition per page number (pages) for maximum parallelism when the API doesn't need per-partition pagination at all.

There's also a cross-cutting piece not tied to any one axis above: in-band error handling (error_path, response_validator), for APIs that report failure inside an otherwise-200 OK response body. See Error handling.

Next steps

  • Request & auth for the full request-configuration and authentication reference.
  • Pagination & partitioning to walk a real API's pages and fan the pull out across executors.
  • Streaming for incremental, cursor-driven reads.
  • Examples for annotated, copy-pasteable scripts covering batch, auth, and streaming.