Response formats & records extraction

Once a request comes back, the reader parses the response body into a Python structure (response_format), then finds the list of records inside that structure (records_path, or one of a few fallbacks). Everything downstream — schemas, pagination stop-conditions, streaming cursors — works the same way regardless of which response format produced the parsed body.

response_format

Option	Default	Description
response_format	json	How the response body is parsed: json, csv, or xml.

json (default)

The response body is parsed as JSON. This is the format everything else in the docs assumes by default — dot paths, nested structs, and the records extraction rules below all apply directly to the parsed JSON structure.

df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .load()
)

csv

Option	Default	Description
csv_delimiter	,	CSV only — the field delimiter.

A header row is required. Each data row becomes one record, and every value comes back as a string — coerce the types via an explicit schema, or let infer_schema sniff them. Empty cells become null.

CSV is incompatible with:

• records_path — the body already is the record list, so there's nothing to extract a path into.
• pagination_type=cursor — cursor pagination reads the next-cursor value out of a response envelope, and a CSV body has no envelope to read one from.

df = (
    spark.read.format("rest_api")
    .option("url", "https://api.example.com/v1/report.csv")
    .option("response_format", "csv")
    .option("infer_schema", "true")
    .load()
)

For a tab-delimited or semicolon-delimited export, set csv_delimiter accordingly:

.option("response_format", "csv")
.option("csv_delimiter", ";")

xml

Option	Default	Description
xml_attribute_prefix	_	XML only — prefix for keys derived from XML attributes (matching Spark's XML source convention).

The element tree maps onto nested dicts/lists:

• Repeated elements become arrays. Sibling elements sharing a tag name collapse into a single list under that tag.
• Attributes become {prefix}-prefixed keys, using xml_attribute_prefix (default _). An attribute named id on an element becomes the key _id in that element's dict.
• An element with both attributes and text keeps its text under {prefix}VALUE (e.g. _VALUE) so the attributes and the text content don't collide on the same key.

Once the tree is mapped this way, dot paths (records_path, next_cursor_path, stream_cursor_field) and nested-struct schemas work exactly as they do for JSON. The one thing to keep in mind is that paths are relative to inside the root element — for a body shaped like:

<response>
  <items>
    <item><id>1</id><name>Widget</name></item>
    <item><id>2</id><name>Gadget</name></item>
  </items>
</response>

the path to the record list is items.item (not response.items.item — the implicit root is already stripped):

df = (
    spark.read.format("rest_api")
    .option("url", "https://api.example.com/v1/items.xml")
    .option("response_format", "xml")
    .option("records_path", "items.item")
    .option("infer_schema", "true")
    .load()
)

Records extraction

Option	Default	Description
records_path	—	Dot path to the record list inside the parsed body (JSON/XML), e.g. envelope.payload. If unset, tries the body itself (if it's a list) or common wrapper keys (results, data, items, records, value), else treats the whole body as one record.
flatten	false	Flattens nested objects into dotted column names (e.g. address.city). Lists are left as-is.

records_path applies to JSON and XML bodies (CSV has no envelope, so it's not applicable there — see above). When records_path is unset, the reader falls back in order:

1. If the parsed body is itself a list, use it directly.
2. Otherwise, look for one of the common wrapper keys — results, data, items, records, value — in that order, and use whichever is present first.
3. Otherwise, treat the whole body as a single record.

For a custom envelope that doesn't match any of those wrapper keys, set records_path explicitly:

# Response body: {"envelope": {"payload": [{"id": 1, "address": {"city": "NYC"}}, ...]}}
df = (
    spark.read.format("rest_api")
    .schema(items_schema)
    .option("url", "https://api.example.com/v1/items")
    .option("records_path", "envelope.payload")
    .load()
)

flatten

Without flatten, a nested object like address in the example above stays a nested struct column — StructField("address", StructType([StructField("city", StringType())])). With flatten=true, address.city becomes its own dotted top-level column instead:

from pyspark.sql.types import IntegerType, StringType, StructField, StructType

schema = StructType(
    [
        StructField("id", IntegerType()),
        StructField("address.city", StringType()),
    ]
)

df = (
    spark.read.format("rest_api")
    .schema(schema)
    .option("url", "https://api.example.com/v1/items")
    .option("records_path", "envelope.payload")
    .option("flatten", "true")
    .load()
)

Lists are left as-is by flatten — only nested objects get promoted to dotted names; a nested array stays an ArrayType column (or a nested array-of-structs, if that's what the API returns).

Flattening also has a side effect on schema drift detection: since it promotes nested dict keys to top-level dotted names before drift detection runs, nested-dict drift becomes visible to rescued_data_column/fail_on_new_fields in a way it wouldn't be otherwise — see the schema page for the full nuance.

Next steps

• Schema — declaring or inferring a StructType for the extracted records, including type coercion and schema drift.
• Pagination & partitioning — how records_path is also used to detect an empty page and stop pagination.
• Streaming — cursor fields are read via the same dot-path mechanism described here.