Streaming Performance & Cost Optimization

Long-running stateful streaming jobs gradually slowed down and consumed more memory as state grew, and compute spend rose with them. The team needed visibility into state behaviour and a disciplined way to tune performance and cost.

1. 01A custom StreamingQueryListener captures per-batch progress and state-operator metrics.
2. 02Metrics are written to Delta tables, giving a queryable history of batch duration, state rows, and memory usage.
3. 03Dashboards and alerts are built on those tables for proactive monitoring.
4. 04Custom utilities analyze workload patterns to drive cluster right-sizing and configuration standards.

Measure state before tuning it

RocksDB state-store metrics — rows, memory, commit latency — are logged every batch so tuning decisions are based on trends, not guesses.

Fix the root cause of driver memory growth

Pinned RocksDB memory and unbounded state were diagnosed and addressed at the source rather than masked by larger drivers.

Apply performance features deliberately

Photon, partitioning, Delta optimization, and caching are applied per workload and captured as reusable standards.

Treat cost as an engineering metric

Workload analysis and cluster right-sizing became part of the operating standard, not a one-off clean-up.

• Databricks
• Structured Streaming
• RocksDB
• StreamingQueryListener
• Delta Lake
• Photon
• Cluster policies

• Approximately 45% lower Databricks compute cost.
• State-store health visible in Delta tables for monitoring and alerting.
• Performance practices embedded into reusable platform standards.

• Most streaming performance problems are state problems — watch state size first.
• Bigger clusters hide leaks; they do not fix them.
• Cost optimization sticks only when it becomes part of the platform's standards.