Articles

Technical writing from production

Deep dives on Databricks Structured Streaming, RocksDB state, Azure Event Hubs, Cosmos DB, and the problems that only show up at scale. All articles are published on Medium.

Follow on Medium

10 articles

AzureApr 12, 20264 min read
Speed Up Your Azure Event Hub Ingestion with Python ThreadPoolExecutor (opens on Medium in a new tab)
A practical guide to moving from sequential to parallel processing for Event Hub ingestion.
Read on Medium
DatabricksMar 30, 20268 min read
Why Your Cosmos DB Connector Fails in Databricks Spark Declarative Pipelines (opens on Medium in a new tab)
Why cosmos.oltp fails in Spark Declarative Pipelines and architecture patterns that work.
Read on Medium
StreamingFeb 12, 20267 min read
Why Your Databricks Streaming Job is Leaking Driver Memory (And How to Fix It) (opens on Medium in a new tab)
Diagnosing RocksDB memory bloat and fixing stateful streaming driver memory leaks.
Read on Medium
StreamingFeb 12, 20266 min read
Databricks: Monitoring RocksDB State Store in Structured Streaming (opens on Medium in a new tab)
How to build streaming observability with custom listeners and Delta logging.
Read on Medium
AzureJan 15, 20266 min read
Real-Time Data Sync with Azure Cosmos DB Change Feed (opens on Medium in a new tab)
Build scalable real-time synchronization pipelines with Cosmos DB Change Feed and Databricks.
Read on Medium
StreamingNov 20, 202510 min read
Mastering Stateful Streaming: A Technical Deep Dive (opens on Medium in a new tab)
In-depth stateful streaming patterns with transformWithStateInPandas.
Read on Medium
ArchitectureNov 20, 20259 min read
From Chaos to Control: Our Journey to Production-Ready Stateful Streaming (opens on Medium in a new tab)
Lessons from failed approaches to production-grade stateful streaming design.
Read on Medium
AI / GenAISep 4, 20257 min read
My Experience: Building a Kids Memory Game with GitHub Copilot (opens on Medium in a new tab)
A rapid prototyping story using GitHub Copilot and Stability AI.
Read on Medium
Data QualityJul 26, 20258 min read
Preventing Data Loss in Real-time Streaming Pipelines When Databricks Clusters Restart (opens on Medium in a new tab)
How to preserve deduplication state and protect integrity across cluster restarts.
Read on Medium
AzureJul 26, 20256 min read
Creating a Linked Service in Azure Data Factory for Azure Cosmos DB Table API (opens on Medium in a new tab)
Step-by-step setup guidance for ADF linked services with Cosmos DB Table API.
Read on Medium