Development¶

This page covers setting up a local development environment, running the test suite, and the lint/type-check commands used in CI. For the full contribution workflow (issues, PR expectations, bug reports) see CONTRIBUTING.md on GitHub, and for release history see CHANGELOG.md.

Requirements¶

Python 3.10+
A Java 17 runtime — Spark requires a JVM, and the test suite starts a real local SparkSession.

Setup¶

git clone https://github.com/liam-mcelhaney122/spark-rest-api-reader.git
cd spark-rest-api-reader
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

pyspark/pyarrow are intentionally not pinned as install dependencies for the published package — a cluster already provides them, and pinning risks conflicting with the runtime's own version. Installing the [dev] extra pulls in a compatible pyspark/pyarrow locally so the test suite has something to run against. See index.md for the end-user install instructions.

Running the tests¶

pytest

The test suite deliberately avoids mocks where it matters: it spins up a real local SparkSession plus a real local HTTP server, so that requests issued from PySpark's separate worker processes are exercised end-to-end — the same code path that runs on an actual cluster, including auth headers, pagination loops, and streaming cursors resolving correctly from inside an isolated worker process rather than just from driver-side test doubles.

The first run is slow because Spark itself has to start up; subsequent tests within the same session are fast.

When you add or change an option, add or update an end-to-end test against the local HTTP server following the existing patterns in tests/ — this is expected for anything user-facing (see CONTRIBUTING.md).

Lint and type-check¶

Both run in CI, so run them locally before pushing:

ruff check src tests
ruff format --check src tests
mypy

Building the wheel¶

For air-gapped clusters that can't reach PyPI, build the wheel locally and upload it as a cluster library:

pip install build
python -m build   # produces dist/spark_rest_api_reader-*.whl

Contributing¶

Open an issue first for anything non-trivial, so the approach can be discussed before you invest time in it.
Keep pull requests focused — one change per PR.
Update the README option reference and CHANGELOG.md when you add or change user-facing behavior.
Bug reports should include the Spark/pyspark version (and DBR version if on Databricks), the full reader configuration (with secrets redacted), and the complete error output.

See CONTRIBUTING.md for the complete guide.