In-band errors¶

Some APIs report failure inside a successful HTTP response. ArcGIS is the canonical example: it answers 200 OK with {"error": {...}} in the body. raise_for_status() never sees a response like that — the HTTP layer is perfectly happy — so left untreated, an in-band error surfaces later as a confusing, disconnected symptom: "zero records to infer from" at schema inference (see Schema), or pagination silently ending early and truncating the data because the malformed body just doesn't look like the next page.

Two composable mechanisms catch these before they can cause that kind of damage: error_path, a dot-path check for the common case, and response_validator, an escape hatch for anything a dot path can't express. Every page of every request is checked, including the schema-inference sample — so a bad response fails loudly and immediately, right where the problem actually is.

`error_path`¶

Option	Default	Description
`error_path`	—	Dot path into the parsed response body (JSON/XML). Any response where the path resolves to a non-null value fails the read with a `ResponseError` carrying that value.

For ArcGIS's {"error": {...}} envelope, the path is simply error:

df = (
    spark.read.format("rest_api")
    .option("url", "https://services.arcgis.com/.../FeatureServer/0/query")
    .option("error_path", "error")
    .load()
)

Any page — including the very first page sampled for infer_schema — where error resolves to something other than null now fails the read with a ResponseError instead of silently producing an empty or truncated result.

Dot paths here work the same way they do for records_path (Response formats), next_cursor_path (Pagination), and stream_cursor_field (Streaming) — for response_format=xml, the path is relative to inside the root element, since the element tree is parsed into the same nested dict/list shape as JSON.

`response_validator`¶

For checks a dot path can't express — inspecting more than one field, applying conditional logic, matching against a set of known error codes — attach a response_validator function to a RestApiDataSource subclass and register the subclass under its own format name. The validator receives the parsed body of each page and returns an error message (a string) to fail the read, or None to accept the page as valid. Raising its own exception from inside the validator also fails the read. When both mechanisms are configured, error_path is checked first.

from spark_rest_api_reader import RestApiDataSource

class ArcGisDataSource(RestApiDataSource):
    @classmethod
    def name(cls):
        return "arcgis_rest"

    @staticmethod
    def response_validator(body):
        if isinstance(body, dict) and "error" in body:
            return f"ArcGIS error: {body['error']}"

spark.dataSource.register(ArcGisDataSource)

df = spark.read.format("arcgis_rest").option("url", ...).load()

Why a subclass attribute, not an option¶

response_validator has to be defined as a @staticmethod on a RestApiDataSource subclass rather than passed via .option(...) for two reasons:

Options are plain strings. There's no way to hand a function to .option(...) — every option value has to serialize to a string, and a validation function isn't one.
Data source code runs in isolated worker processes. A subclass defined right in a notebook is cloudpickled by value at spark.dataSource.register(...) time, so the attached response_validator function travels along with it to every worker. Module-level state — a registry dict, a global variable holding the validator — would not make that trip, since only the class itself is pickled and shipped.

In practice this means: define your subclass, give it a distinct name() (so it doesn't collide with the base rest_api format), attach response_validator as a @staticmethod, register the subclass once per session, and read using that subclass's registered name — exactly as shown above for arcgis_rest.

In-band errors¶

error_path¶

response_validator¶

Why a subclass attribute, not an option¶

See also¶

`error_path`¶

`response_validator`¶