Databricks Jobs

I’m ready to move my real-time data processing into Databricks and need a specialist who can build a robust streaming pipeline end-to-end. The source will be Apache Kafka; events arrive continuously and must be processed, transformed, and stored inside the Databricks environment with low latency and solid fault tolerance. You’ll design the job structure, configure Structured Streaming, handle schema evolution, and implement checkpoints so that the pipeline restarts cleanly after failures. I also want concise notebook-based documentation that shows how the pipeline is triggered, how it scales, and where metrics can be monitored in the workspace. Deliverables • A working Databricks notebook (or repo) that ingests from our existing Kafka topics, performs necessary tra...