Data Engineer

Job Title: Data Engineer - Databricks Performance & Optimization

Houston TX

Compensation: Base salary range: $130,000–$150,000 plus 5% bonus. Benefits package available (details to be discussed).

Benefits: This position is eligible for benefits

Qualifications: Build and operate Lakehouse pipelines on Databricks (Bronze/Silver/Gold) using Delta Lake, Delta Live Tables (DLT), and/or Jobs.

• Optimize ingestion patterns (Autoloader, CDC, streaming).
• Model data, implement quality checks, and performance optimization.
• Profile and tune Spark/SQL workloads: partitioning, clustering, constraints, liquid clustering.
  
  

Job Description:

• Engineer Delta tables for speed and cost: partitioning, Z-Ordering/clustering, constraints, file sizing; manage table health with Auto Optimize, OPTIMIZE, and VACUUM.
• Implement incremental processing (MERGE with Change Data Feed, APPLY CHANGES INTO) with idempotency and exactly-once delivery.
• Deliver reliable, well-documented datasets with clear SLAs.
• Design and implement dashboards and reports using Power BI and other visualization tools.
• Collaborate with business units to gather requirements and deliver technical solutions.
• Integrate data from multiple sources, including real-time field equipment and sensors.
• Educate and support stakeholders on data tools and best practices.
• Engage in continuous improvement and adoption of new data management technologies.