• Publicar trabajo
  • Subir CV
  • Acceder
Ver oferta completa

DATA ENGINEER (SCALA - SPARK)

Descripción de la oferta de empleo

At Sandav, we are expanding our Madrid team with a Data Engineer to join a stable and growing project within the international financial sector.Responsabilities:Data Modeling and Pipelines development with Spark and Scala to ingest and transform data from different sources (Kafka topics, APIs, HDFS, structured databases, files…) intoHDFS, IBM Cloud Storage (generally in parquet format) or SQL/NOSQL databases followign complex business rulesManage big data storage solutions in the platform (HDFS, IBM Cloud Storage, structured and non-structured databases)CI/CD Pipeline implementation.Infrastructure Migration: migrate the existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, Object Storage (IBM Cloud storage), Spark as a service on Scala (to build the data pipelines), and Airflow as a service (to orchestrate and schedule the data pipelines)Implementation of schemas, queries, and views in SQL/NOSQL databases like Oracle, Postgres or MongoDBDevelop and configure scheduling of data pipelines with a combination of shell scripting and AirFlow as a serviceValidation Testing: conduct unit and validation tests to ensure accuracy and integrity.Code improvement.Configure Dremio Data Virtualization to interface with Parquet or as a way to expose thedata in the different data products.Qualifications:English: at least B2.Spark on Scala as legacy data pipeline development languageSpark as a service on Scala as data pipeline development platformExperience in the design and development of streaming procesess using Spark Streaming,Spark Structure Streaming and Apache KafkaManagement of legacy big data storage solutions (HDFS)Management of big data storage solutions (IBM Cloud Object Storage and parquet format)Implementation of SQL/NO SQL database schemas ,queries and views (MongoDB, Oracle,Postgres)Shell scripting and Airflow as data pipelie scheduling solutionDremio as data virtualization toolDataiku as data preparation tool as bonus.We offer:Permanent contract with Sandav.Salary: €38,000 – €42,000.Hybrid work model: 1 month in-office at 50%, followed by 1 month of remote work.Location: Madrid Río.Schedule: Start between 8:00 and 9:00 AM. 8.5-hour workdays from Monday to Thursday; Fridays and summer, intensive morning workday.---Sandav is a unique technology consulting firm.We offer Team as a Service, recruitment, training, outsourcing and application maintenance services, as well as software development.Sandav is set out to become a paradigm shift within the industry and to do so we put our values into practice. Starting by taking care of our colleagues and building trusting and lasting relations with our clients.We are people, not numbers. We are different!www.sandavteam.com
Ver oferta completa

Detalles de la oferta

Empresa
  • Sandav
Localidad
  • En toda España
Dirección
  • Sin especificar - Sin especificar
Fecha de publicación
  • 17/09/2024
Fecha de expiración
  • 16/12/2024
Cómo detectar ofertas falsas
Data Engineer
Innoit

Are you a big data engineer looking for a new challenge? so... solid knowledge of data structures and experience with integration of data from multiple sources... ensures that non-functional/support aspects of data classification and data sharing agreements (dsas) and sign-offs are fulfilled... com/es-es/meetup-de-innoit-consulting-en-barcelona/?_locale=es-es......

Qa automation engineer
Innoit

Your profile: at least 3y of experience working as a qa automation engineer... are you a qa automation engineer looking for new challenge? we aspire to reach everyone and connect them to top projects... collaborate closely with developers, designers, and product owners... selenium, cypress)... com/es-es/meetup-de-innoit-consulting-en-barcelona/?_locale=es-es......

Remote Data Contributor – Image collection
TransPerfect DataForce

Our division focuses on gathering, enriching, and processing data for machine learning in different ai domains... we offer high-quality data for human-machine interaction to some of the most prestigious technology companies in the world... position: data contributor project location: spain(remote) engagement......

Data Collection Project Tahoe - Dutch or French Speaker
TransPerfect DataForce

Our division focuses on gathering, enriching and processing data for machine learning in different ai domains... we offer high-quality data for human-machine interaction to some of the most prestigious technology companies in the world... position: data contributor project location: barcelona, spain......

CALL 37-2023-1 Satellite Communications Engineer
Centre Tecnològic de Telecomunicacions de Catalunya

Who are we looking for ? the space and resilient communications and systems unit is looking for a satellite communications engineer... * cvs and any other information gathered during this process will be handled confidentially who are we? the center tecnològic de telecomunicaciones de catalunya (cttc)......

CALL 41-2023-1 - Satellite Communications Engineer
Centre Tecnològic de Telecomunicacions de Catalunya

Who are we looking for ? the space and resilient communications and systems unit is looking for a satellite communications engineer... * cvs and any other information gathered during this process will be handled confidentially who are we? the center tecnològic de telecomunicaciones de catalunya (cttc)......

Data Governance Analyst
LLYC

Parte de tus funciones serán: desarrollo e implementación de políticas de datos seguridad y cumplimiento datos maestros y metadatos gestión de calidad de datos evaluación y mitigación de riesgos formación y capacitación reportes innovación y mejora continua requisitos del puestoqué valoramosbuscamos......

Devops engineer
Innoit

Are you a devops / site reliability engineer seeking new interesting opportunity? so... keep reading it can be just what you're looking for! responsibilities: develop and maintain systems to support the company business... experience with databases (mysql, postgresql and elasticsearch)... com/es-es/meetup-de-innoit-consulting-en-barcelona/?_locale=es-es......

CALL 14-2024-1 Research Engineer for a Cloud 5G/6G Lab
Centre Tecnològic de Telecomunicacions de Catalunya

*cvs and any other information gathered during this process will be handled confidentiallywho are we?• the center tecnològic de telecomunicaciones de catalunya (cttc) is a non-profit public sector research institution dedicated to fundamental and applied research activities, focused mainly on technologies......

QA Engineer
Involve rh

Confidencial cuenta con una posición como qa engineer para garantizar la calidad del software mediante pruebas exhaustivas para identificar y corregir errores antes de su lanzamiento al mercado... colaborar con el equipo de desarrollo para mejorar los procesos de calidad del software......