CLOUD DATA ENGINEER

Job PurposeAt Cabot data and analytics are fundamental to our success and embedded in every step of our business decision making processes. When combined with great execution and obsessive measurement of results, this approach creates a feedback loop that enables continuous improvement and innovation. This restlessness amounts to a sustainable competitive advantage for which data and analytics are a critical ingredient.The Cloud Data Engineer will play a critical role in developing and optimizing our cloud data and analytics platform. Working alongside the Data Governance Lead, this role focuses on building robust ELT pipelines using but not limited to, Databricks and Data Factory, integrating a wealth of data from and for our pan-European customers into a unified Databricks Lakehouse architecture and satellite data marts.Key needs for this role are a strong software engineering and data modelling mindset to ensure the overall data architecture is robust, efficient and scalable. Data Governance will be at the heart of all operations, design and execution. DevOps, CI/CD and an agile mindset are crucial for effective time, team and activity management.You will be very familiar with, and passionate about, Databricks, Data Factory and the capabilities they both provide, such as Unity Catalog, medallion architecture, Data Flows and Delta Live Tables and understand the underlying technologies and how to use them to maximum effect. You will work with large, complex data sets and will be required to have a complete knowledge of the data as well as understanding of how the business areas function, coupled with strong stakeholder engagement. Key Accountabilities & Responsibilities Designing, building and maintaining efficient environment management and ELT pipelines using Databricks, Data Factory and other suitable tooling.Integrating data from various pan-European sources into a unified Gold Lakehouse and suitable data marts.Continuously monitoring and testing the availability, performance and quality of data pipelines.Collaborating with peers by employing and following SDLC best practice while maintaining code repositories and activity using Agile, DevOps and CI/CD methodologies through Dev, Test and QA environments.Collaborating with the Data Governance Lead and Head of Data Engineering to ensure the data product(s) align with the proposed unified model, data governance framework and key business objectives.Working closely with stakeholders to understand ongoing requirements, build effective products and align data modelling principles.Adhering to agreed Release and Change Management Processes.Troubleshoot and investigate anomalies and bugs in code or data.Adhering to test and reconciliation standards to produce confidence in delivery.Championing Data Governance, compliance and security principles, treating PII with utmost respect and adhering to regulatory obligation.Produce appropriate and comprehensive documentation to support ease of access for technical and non-technical users.Engage in a culture of continual process improvement and best practice.Efficiently respond to changing business priorities through effective time management.Conduct all activities and duties in line with company policy and compliantly.To carry out any other ad-hoc duties as requested by management.Person specification:Knowledge & ExperienceKey skills required : Kimball, Python, PySpark, SparkSQL, T-SQL, Delta Live Tables, Structured Streaming.Solid experience of ADF meta-data driven Integration runtimes, pipelines, data flows.Experienced in design and implementation of end-to-end ELT pipelines using Databricks, Data Factory, Delta Live Tables, structured streaming.Experienced in automating data extraction, transformation/manipulation, summarisation, and delivery, and the roles they play within a Databricks medallion architecture and DAG.Understanding of principles such as SOLID and RESTful APIs and a software engineering mindset.Experience and understanding of Azure fundamentals, networking and security.Experience working with traditional relational Data warehouses, SQL Server or other database technologies.Databricks and Data Factory performance tuning methods such as clustering, partitioning, scaling, Delta/Parquet optimization, and an understanding of Databricks cluster and Spark principles for performance tuning.Developing, tuning and maintaining semantic models in Power BI service and Power BI Desktop.Purview experience is a preferable.Personal AttributesPassionate about quality, governance and attention to detail when building great products.Passionate about building and being part of, a high-performing team and brand.Collaboration, communication and influence: ability to build trusted relationships with internal business stakeholders and SMEs, to design, scope and plan deliveries that support business value-add initiatives.Highly numerate and proven problem-solving skills.What happens next?If this sounds like you and you’d like to join our rapidly expanding company that offers excellent career progression, then apply now!What you can expect as a Cloud Data Governance Lead:PermanentHybrid roleOffering a competitive salary and a fantastic bonus schemeWorking for Cabot:We’re passionate about the ethical treatment of our customers and employees. Our mission is to create pathways to economic freedom. Our vision is to make credit accessible by partnering with our consumers to restore their financial health.Diversity and inclusion are very important to us at Cabot, and we value a multitude of diverse talent within our business. We want everyone to be themselves at work and encourage a culture that includes everyone. Our policies ensure that every candidate and employee are treated fairly and with equal opportunities.