IBM
Data Engineer - Databricks specialist
Job Description
**Introduction**
Are you passionate about building scalable data solutions and working with cutting-edge technologies? We're looking for a Data Engineer with deep expertise in Databricks to help us design and optimize modern data processing frameworks.
**Your role and responsibilities**
* Design and build streaming & batch data pipelines using Apache Spark on Databricks
* Optimize performance with Delta Lake, Z-ordering, and liquid clustering
* Work hands-on with PySpark, SQL, and develop our modern Data Lakehouse architecture
* Leverage the latest Databricks features: Structured Streaming, Lakeflow, Unity Catalog, DBSQL
* Support AI/ML workflows and implement CI/CD pipelines with tools like GitHub Actions or Azure DevOps
**Required technical and professional expertise**
* 3+ years in data/software engineering with large-scale distributed systems.
* Strong Python skills (OOP, pytest/unittest) and good expe...
Are you passionate about building scalable data solutions and working with cutting-edge technologies? We're looking for a Data Engineer with deep expertise in Databricks to help us design and optimize modern data processing frameworks.
**Your role and responsibilities**
* Design and build streaming & batch data pipelines using Apache Spark on Databricks
* Optimize performance with Delta Lake, Z-ordering, and liquid clustering
* Work hands-on with PySpark, SQL, and develop our modern Data Lakehouse architecture
* Leverage the latest Databricks features: Structured Streaming, Lakeflow, Unity Catalog, DBSQL
* Support AI/ML workflows and implement CI/CD pipelines with tools like GitHub Actions or Azure DevOps
**Required technical and professional expertise**
* 3+ years in data/software engineering with large-scale distributed systems.
* Strong Python skills (OOP, pytest/unittest) and good expe...