Real-Time Airline Data Engineering

Airline operations depend on fresh, consistent operational data. Events arrive continuously, can be duplicated or late, and must reach low-latency serving stores without gaps — including when a streaming cluster restarts or is redeployed.

1. 01Azure Event Hubs receives operational events from source systems.
2. 02Databricks Structured Streaming consumes the stream and applies stateful processing — deduplication, ordering, and per-key state with timers.
3. 03State is held in the RocksDB state store and checkpointed so it survives restarts.
4. 04Curated results are written to Azure Cosmos DB for low-latency serving and to Delta Lake for history.
5. 05Cosmos DB Change Feed propagates changes onward for synchronization and archival.

Keep deduplication state across restarts

Deduplication lives in checkpointed streaming state rather than in memory, so a cluster restart resumes from the last committed offsets with its state intact instead of re-emitting or dropping events.

Use arbitrary stateful processing where SQL operators fall short

transformWithStateInPandas gives explicit control over per-key state, TTLs, and timers for business rules that built-in aggregations cannot express.

Parallelize I/O-bound ingestion

Moving Event Hubs ingestion from sequential calls to a ThreadPoolExecutor removes idle I/O wait and raises throughput without adding cluster capacity.

Pick the right Cosmos DB integration per pipeline type

The Cosmos DB Spark connector does not work inside Spark Declarative Pipelines, so those flows use an alternative integration pattern while classic jobs keep the connector.

• Azure Event Hubs
• Databricks Structured Streaming
• transformWithStateInPandas
• RocksDB state store
• Azure Cosmos DB
• Cosmos DB Change Feed
• Delta Lake
• PySpark

• Streaming pipelines resume after cluster restarts without losing deduplication state.
• Event Hubs ingestion throughput improved by parallelizing I/O-bound work.
• Patterns documented and shared publicly as technical articles.

• Treat restarts as a normal event in streaming design, not an edge case.
• Stateful logic needs explicit TTLs; state that never expires eventually becomes an incident.
• Validate connector support for each Databricks pipeline type before committing to an architecture.