Praca: Senior Big Data Specialist



Senior Big Data Specialist
[online recruitment]
Location: Wroclaw
Reference number: VR-56501
Project Description:

You’ll be working with various ETL (Extract Transform Load) pipelines based mainly on Apache Spark in AWS for batch and streaming processing, communicating daily with colleagues distributed locally and abroad. The project includes numerous activities like design, technical supervision, and development. You should be comfortable with discussing selected technical solutions with a customer, as well as assessing design risks, flaws, and eliminating issues. We expect you’ll be able to collaborate technical work among Big Data developers, QAs, and DevOps to ensure the appropriate quality of code, continuous integration, and continuous delivery per best practices of software development.

Project pipelines examples:

  • Spark pipeline to cleanse a dataset for further analysis by ML algorithms by other teams. Deployed on AWS EMR using CDK. Scheduled on a daily basis to process ~1 TB of data
  • Workflow to pull pre-processed data from Teradata database, hash sensitive data and store partitioned data on S3 in parquet format. Scheduled on a daily basis on an on-prem server as an Oozie workflow
  • Hourly Spark pipeline to normalize highly dynamic dataset in JSON format to satisfy SQL standards. Deployed on AWS EMR using CDK
  • Daily Spark pipeline to ingest data to Redshift database for BA analysis. Dataset is extremely dynamic (from 1k columns to 7k each day) which won’t fit a classic SQL database by default
Responsibilities
  • Gather requirements from users (Data Analytics), design and implement ETL pipelines from scratch
  • Support and migration of the existing workflows from on-premise to AWS
  • Understanding the Data Lake and Data Warehouse concepts
  • Analysis of the business requirements and task estimation
  • Refactoring of the existing application. Defects fixing
  • Design and implementation of ETL pipelines from scratch on distributed systems
  • Support PROD releases. Release documentation
  • Release automation using AWS CDK and CloudFormation Templates
Skills

5+ years experience in Java/Scala software development2+ years experience with AWS

Tech stack

  • Programming languages: Scala, Java
  • Hadoop: Spark, Kafka, YARN, HDFS, Oozie
  • AWS: EMR, S3, Redshift, Glue, Athena, CloudFormation, CDK, CloudWatch, Secret Manager
  • non-AWS: Couchbase, ElasticSearch
  • Data formats: Parquet, Avro, JSON
Our offer
Career and Professional Development
  • International projects for clients all over the world
  • Competitive salary
  • Individual development plan
  • Managerial Targeted Training programs
  • BRIDGE Mentoring Program
  • Luxoft Training Center
  • Language Classes
  • Self-learning online library
  • Global Relocation Program
  • Internal Mobility (a chance to gain experience in varied projects and technologies)
  • Professional communities for knowledge-sharing (Agile, Tech, Business)
Well-being
  • Group Life Insurance
  • Travel Insurance
  • Private Healthcare (dental care, unlimited consultations of specialist physicians)
  • Medical costs reimbursement for employees
  • Benefit Program (Cafeteria and Multisport Card)
Work-Life Balance
  • LuxGood Program (a wide range of health and well-being initiatives)
  • After-hours groups (sport, trips, board games, cultural activities)
  • Company and Team events
  • BeLux - discount offers program (banking, car leasing, other)
  • Convenient locations in modern offices
Career and Professional Development
  • International projects for clients all over the world
  • Competitive salary
  • Individual development plan
  • Managerial Targeted Training programs
  • BRIDGE Mentoring Program
  • Luxoft Training Center
  • Language Classes
  • Self-learning online library
  • Global Relocation Program
  • Internal Mobility (a chance to gain experience in varied projects and technologies)
  • Professional communities for knowledge-sharing (Agile, Tech, Business)
Well-being
  • Group Life Insurance
  • Travel Insurance
  • Private Healthcare (dental care, unlimited consultations of specialist physicians)
  • Medical costs reimbursement for employees
  • Benefit Program (Cafeteria and Multisport Card)
Work-Life Balance
  • LuxGood Program (a wide range of health and well-being initiatives)
  • After-hours groups (sport, trips, board games, cultural activities)
  • Company and Team events
  • BeLux - discount offers program (banking, car leasing, other)
  • Convenient locations in modern offices
Please send your CV with consent for processing your personal data for recruitment purposes using the below button
APPLY
Informujemy, że administratorem danych jest Luxoft z siedzibą w Krakowie, ul. Bora Komorowskiego(dalej jako "administrator"). Masz prawo do żądania dostępu do swoich danych osobowych, ich sprostowania, usunięcia lub ograniczenia przetwarzania, prawo do wniesienia sprzeciwu wobec przetwarzania, a także prawo do przenoszenia danych oraz wniesienia skargi do organu nadzorczego. Dane osobowe przetwarzane będą w celu realizacji procesu rekrutacji. Podanie danych w zakresie wynikającym z ustawy z dnia 26 czerwca 1974 r. Kodeks pracy jest obowiązkowe. W pozostałym zakresie podanie danych jest dobrowolne. Odmowa podania danych obowiązkowych może skutkować brakiem możliwości przeprowadzenia procesu rekrutacji. Administrator przetwarza dane obowiązkowe na podstawie ciążącego na nim obowiązku prawnego, zaś w zakresie danych dodatkowych podstawą przetwarzania jest zgoda. Dane osobowe będą przetwarzane do czasu zakończenia postępowania rekrutacyjnego i przez okres możliwości dochodzenia ewentualnych roszczeń, a w przypadku wyrażenia zgody na udział w przyszłych postępowaniach rekrutacyjnych - do czasu wycofania tej zgody. Zgoda na przetwarzanie danych osobowych może zostać wycofana w dowolnym momencie.
APLIKUJ TERAZ