Job details
About the Role We are seeking experienced Databricks Data Engineers to join our Data & AI. You will play a pivotal role in designing, developing, and maintaining scalable data pipelines and transformation workflows on Azure Databricks, enabling advanced analytics and data engineering initiatives. Key Responsibilities - Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark. - Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment to support Data Science use cases. - Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets. - Implement data quality checks, validation rules, and monitoring mechanisms. - Integrate data from various sources including structured, semi-structured, and unstructured formats. - Ensure data security, governance, and compliance with banking regulations. - Optimize Spark jobs for performance and cost-efficiency in Azure. - Participate in code reviews, design discussions, and agile ceremonies. - Document data pipelines, workflows, and technical decisions. . Required Skills & Qualifications Technical Skills - 12+ years of experience in data engineering, with at least 3+ years hands-on experience in Azure Databricks. - Strong proficiency in Apache Spark, especially PySpark for distributed data processing. - Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake. - Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts. - Proficiency in Python and/or Scala for data processing. - Experience with CI/CD pipelines using tools like Azure DevOps, Git, and Terraform (preferred). - Familiarity with SQL, NoSQL, and data lake architectures. - Knowledge of data governance, security, and compliance in financial services. Soft Skills - Strong analytical and problem-solving skills. - Excellent communication and stakeholder management abilities. - Ability to work collaboratively in cross-functional teams. - Agile mindset and experience working in Scrum/Agile environments. Preferred Qualifications - Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field. - Databricks Certified Data Engineer Associate/Professional certification is a plus. - Experience working in banking or financial services domain is highly desirable. Location & Work Rights - This role is based in Sydney/Melbourne. - Australian Citizens are highly preferred; however, Australian Permanent Residents (PR holders) will also be considered. .