Spark Declarative Pipelines (SDP) - Complete Reference

Spark Declarative Pipelines (SDP) - Complete Reference

A comprehensive guide for working with Spark 4.1+ Declarative Pipelines, from first principles to production deployment.

---

Table of Contents

1. What is SDP?
2. Prerequisites
3. Core Concepts
4. Schema Discovery
5. Your First Pipeline
6. The SDP API
7. Pipeline Configuration
8. CLI Reference
9. Patterns & Best Practices
10. Streaming Pipelines
11. Performance Tuning
12. Testing Strategies
13. Production Operations
14. Migration from Imperative
15. Troubleshooting
16. When NOT to Use SDP
17. Quick Reference

---

What is SDP?

The Problem SDP Solves

Traditional PySpark pipelines require you to:

• Manually manage execution order
• Explicitly write data to tables
• Handle dependencies yourself
• Coordinate between batch and streaming

# Traditional approach - lots of boilerplate
def bronze_orders(spark):
    df = spark.read.parquet("/data/orders.parquet")
    df.write.mode("overwrite").saveAsTable("bronze.orders")

def silver_orders(spark):
    df = spark.table("bronze.orders")  # Must run after bronze_orders!
    df = df.filter(col("id").isNotNull())
    df.write.mode("overwrite").saveAsTable("silver.orders")

# You manage execution order
bronze_orders(spark)
silver_orders(spark)

The SDP Solution

SDP lets you declare **what** you want, not **how** to do it:

# SDP approach - declare intent, framework handles execution
@dp.materialized_view(name="bronze.orders")
def bronze_orders():
    return spark.read.parquet("/data/orders.parquet")

@dp.materialized_view(name="silver.orders")
def silver_orders():
    return spark.table("iceberg.bronze.orders").filter(col("id").isNotNull())

# Framework figures out order and handles writes

Key Benefits

Benefit	Description
Automatic dependency resolution	Framework detects dependencies from spark.table() calls
No explicit writes	Return DataFrames; framework handles persistence
Unified batch/streaming	Same patterns for both; switch with decorator change
Built-in validation	dry-run catches errors before execution
Incremental by default	Streaming tables maintain state automatically

Mental Model

Think of SDP like SQL views with superpowers:

SQL View:     CREATE VIEW silver.orders AS SELECT * FROM bronze.orders WHERE ...
SDP:          @dp.materialized_view(name="silver.orders")
              def silver_orders(): return spark.table("...").filter(...)

The decorator says "I want a table called X". The function body says "here's the data". The