Schema & schema drift¶

The schema decides what a record looks like once it's out of the parsed response body and into a DataFrame column. You can supply it explicitly, let the reader infer one from a sample, or mix the two — an inferred schema adjusted with schema_hints. Whichever path you take, schema resolution happens once, up front; it does not change mid-batch or mid-stream. This page covers defining a schema, inference, type coercion, and what happens when the API sends fields the schema didn't expect.

The structs pattern¶

Schemas follow the structs pattern: nested objects are declared and returned as real StructType/ArrayType columns. Build them as StructType objects, not DDL strings, and never model nested data as JSON-string columns that downstream code has to re-parse.

from pyspark.sql.types import ArrayType, IntegerType, StringType, StructField, StructType

schema = StructType(
    [
        StructField("id", IntegerType()),
        StructField("address", StructType([StructField("city", StringType())])),
        StructField("tags", ArrayType(StringType())),
    ]
)

df = spark.read.format("rest_api").schema(schema).option("url", "https://api.example.com/v1/items").load()

Nested response objects map onto nested StructType/ArrayType schema fields automatically — a JSON address object becomes a StructType column, a JSON array becomes an ArrayType column, and so on for however deeply the response nests. This holds for all three response formats (json, csv, xml), though csv rows are flat by nature and xml element trees map onto the same nested shape as JSON.

Ways to supply a schema¶

Option	Description
`.schema(StructType(...))`	Preferred — pass the `StructType` directly on the `DataFrameReader`, as in the snippet above.
`schema`	Same information as an option: pass `your_struct_type.json()` (Spark's JSON schema representation — also what `df.schema.json()` produces on any DataFrame).
`infer_schema`	`true` to sample the first page and infer a schema from the sampled values — see Inference below.

Use .schema(...) when you're building the StructType in the same script that configures the reader. Use the schema option when the schema needs to travel as a string — for example, stored alongside other job configuration, or reconstructed from a previous run's df.schema.json().

Why `schema` uses `fromJson`, not `fromDDL`¶

The schema option is parsed with StructType.fromJson. A DDL string like "id int, name string" (auto-detected by the missing leading {) is still accepted for back-compat, but fromJson is the primary path, and the reason is where schema resolution actually runs: DataSource.schema() executes in an isolated Python worker process with no driver context, and fromJson is pure Python with no dependency on an active SparkContext. fromDDL, by contrast, needs one — it wouldn't work in that worker process. Passing your_struct_type.json() sidesteps the issue entirely.

df = (
    spark.read.format("rest_api")
    .option("url", "https://api.example.com/v1/items")
    .option("schema", schema.json())
    .load()
)

Inference¶

Set infer_schema=true to have the reader sample the first page and infer a schema from it, Auto Loader-style. This works across all three response formats.

Option	Default	Description
`infer_schema`	`false`	Sample the first page and infer the schema from the sampled records.
`infer_sample_size`	`100`	Max records sampled for inference.
`infer_column_types`	`true`	Infer real leaf types. Set `false` for Auto Loader's conservative default — every leaf a `string` — while still preserving the full nested struct/array shape (nested data is never collapsed to a JSON string, in either mode). For untyped formats (`csv`/`xml`, where every raw value is a string), `true` additionally sniffs string values into `long`/`double`/`boolean`/`date`/`timestamp`.
`schema_hints`	—	Auto Loader-style overrides applied on top of the inferred schema: comma-separated `"<column path> <type>"` entries, e.g. `"version int, user_info.dob date, tags array<int>"`. Dotted paths descend through structs and array-of-struct elements; a hinted top-level column that wasn't inferred is appended. Only valid together with `infer_schema=true` (with an explicit schema, fold the type into the schema itself).

Inference merges field sets across sampled records, widens mixed numeric samples to double (and mixed date/timestamp to timestamp), and falls back to string only on genuine type conflicts. Nested response objects map onto nested StructType/ArrayType fields automatically, the same as with an explicit schema, and ISO-8601 timestamp/date strings are parsed for TimestampType/DateType columns.

df = (
    spark.read.format("rest_api")
    .option("url", "https://api.example.com/v1/items")
    .option("infer_schema", "true")
    .option("infer_sample_size", "200")
    .option("schema_hints", "updated_at timestamp, user.dob date")
    .load()
)

Because every page of every request — including the schema-inference sample — is checked for in-band errors, an API that reports failure inside a 200 OK body won't silently produce an empty or malformed inferred schema; configure error_path or a response_validator alongside inference if the API does that.

Type coercion¶

Values are best-effort coerced to match the declared column type — a stringified number or boolean ("123", "true") coerces to Integer/Boolean, and a non-string value landing in a StringType column is stringified. This matters because REST APIs commonly stringify numbers to dodge JS float-precision issues, and a rigid type check would otherwise reject perfectly good data.

A value that genuinely can't be coerced (e.g. an object where a scalar is expected) raises SchemaTypeError naming the exact field path and both types involved, instead of silently reaching Spark's Arrow conversion and failing there with an opaque, unattributed error several layers removed from the actual problem.

Schema drift¶

The schema is fixed once resolved — it is not re-inferred mid-batch or mid-stream. By default, fields the API returns that aren't in the schema are silently dropped, and fields the schema expects but the API omits become null. Two options change that default:

Option	Default	Description
`rescued_data_column`	—	Name of a `StringType` column (must be declared in the schema) that captures any top-level record keys not otherwise mapped, as a JSON string. `null` when nothing drifted. With `infer_schema=true`, this column is added to the inferred schema automatically; with a DDL/JSON `schema`, you must declare it yourself or resolution fails with a clear error.
`fail_on_new_fields`	`false`	Raise immediately the first time a record has a top-level key outside the schema, instead of silently dropping it. Takes precedence over `rescued_data_column` if both are set.

schema = StructType(
    [
        StructField("id", IntegerType()),
        StructField("name", StringType()),
        StructField("_rescued_data", StringType()),
    ]
)

df = (
    spark.read.format("rest_api")
    .schema(schema)
    .option("url", "https://api.example.com/v1/items")
    .option("rescued_data_column", "_rescued_data")
    .load()
)

Or, to fail fast instead of quietly capturing drift:

    .option("fail_on_new_fields", "true")

The top-level-only caveat¶

Drift detection is top-level only — new keys appearing inside an array element aren't (currently) surfaced by either rescued_data_column or fail_on_new_fields. If an API starts adding a new field inside items of a tags array, neither option will notice.

One nuance changes this for a specific combination: with flatten=true (see Response formats), nested dict keys are promoted to top-level dotted names before drift detection runs, so nested-dict drift actually is caught in that combination — only drift nested inside an array stays invisible either way.

Next steps¶

Response formats — response_format and records extraction (records_path, flatten), including how flatten interacts with drift detection above.
Error handling — catching in-band errors before they masquerade as an empty inference sample or a truncated pagination run.
Pagination & partitioning — how pages are walked and the pull fanned out, independent of schema.