spark-rest-api-reader¶
A generic Spark Python Data Source for pulling data out of any GET REST API — returning JSON, CSV, or XML — and into a DataFrame, batch or streaming, without writing a bespoke connector for every API.
Requires Spark 4.0+ / Databricks Runtime 15.4+ (the Python Data Source API). Not tied to Databricks otherwise — it works with any Spark 4.0+ cluster.
Why¶
Every project ends up writing the same HTTP-get + pagination + retry +
auth-header boilerplate to land a REST API into Bronze. This wraps that
boilerplate once, behind Spark options, so pulling a new API is a matter of
configuring auth_type/pagination_type, not writing new Python.
Built on httpx for the HTTP client and
tenacity for retry/backoff.
Install¶
pip install spark-rest-api-reader
On a Databricks cluster, install it as a cluster library (PyPI source:
spark-rest-api-reader) or per-notebook:
%pip install spark-rest-api-reader
For air-gapped clusters, build the wheel yourself and upload it as a cluster library:
pip install build
python -m build # produces dist/spark_rest_api_reader-*.whl
pyspark/pyarrow are intentionally not pinned as install dependencies —
the cluster already provides them, and pinning risks conflicting with the
runtime's own version. For local development/tests, install .[dev] (see
Development).
Where to go next¶
- Getting started — a quick-start walkthrough, secrets handling, and how the library's options are organized.
- Options reference:
- Request & auth — URL, headers, query params,
timeouts, retries, proxies, TLS, rate limiting, and every
auth_type. - Response formats —
response_format(json/csv/xml) and records extraction (records_path,flatten). - Schema — explicit
StructTypeschemas, inference, type coercion, and schema drift handling. - Pagination & partitioning — every
pagination_type, plus howpartitions/pagesfan work out across Spark. - Streaming — incremental cursor-driven reads and checkpoint/restart semantics.
- Error handling — catching in-band errors with
error_pathandresponse_validator.
- Request & auth — URL, headers, query params,
timeouts, retries, proxies, TLS, rate limiting, and every
- Examples — an annotated tour of the example scripts.
- Development — dev setup, running tests, and contributing.