Senior Data Engineer

About the role

Jobs for Humanity, a multinational technology consulting firm, is looking for a Senior Data Engineer to design, build, and maintain robust data pipelines that power AI‑driven applications and real‑time services. You will work in a collaborative, agile environment with engineers, data scientists, and product teams.

Key responsibilities

• Design, build, and maintain ETL/ELT pipelines in Databricks to ingest, clean, and transform data from diverse sources.
• Construct gold‑layer tables in a Lakehouse architecture for machine‑learning model training and real‑time APIs.
• Monitor data quality, lineage, and reliability using Databricks best practices.
• Collaborate with AI/ML teams to model data for natural‑language prompts, semantic retrieval, vector search, and Unity Catalog metadata.
• Develop data interfaces and agent tools that enable role‑based access to structured data and AI agents.
• Partner with backend engineers to design serverless APIs (AWS Lambda, TypeScript) that expose gold tables to front‑end applications.
• Ensure APIs are performant, scalable, secure, and compliant with data‑security standards.
• Implement provisioning, deployment, security, and monitoring frameworks for scaling pipelines, AI endpoints, and multi‑tenant environments.

Required profile

• 3+ years of experience as a Data Engineer in an agile, distributed team.
• Proven expertise with Databricks, including job/workflow orchestration, change‑data‑capture, and medallion architecture.
• Strong proficiency in Spark or Scala for data wrangling and transformation.
• Hands‑on experience with CI/CD, test‑driven development, and MLOps/AIOps lifecycles.
• Ability to work closely with product managers, front‑end engineers, and data scientists.

Required skills

• Databricks
• ETL/ELT pipelines
• Lakehouse architecture
• Spark
• Scala
• CI/CD
• MLOps / AIOps
• AWS Lambda (Node.js/TypeScript)
• API Gateway, RESTful & GraphQL endpoints
• Unity Catalog & vector search
• React basics
• A/B testing & experiment logging