Data Engineer / Apache Spark Performance Optimization Specialist (m/w/d) 100%
Join to apply for the Data Engineer / Apache Spark Performance Optimization Specialist (m/w/d) 100% role at Randstad Digital Switzerland
Data Engineer / Apache Spark Performance Optimization Specialist (m/w/d) 100%
5 days ago Be among the first 25 applicants
Join to apply for the Data Engineer / Apache Spark Performance Optimization Specialist (m/w/d) 100% role at Randstad Digital Switzerland
Our client is seeking an experienced Apache Spark Optimization Expert to join our data engineering team, working in the Finance InfoTech area of a leading reinsurance company.
Their Azure based data solution processes billions of records within challenging deadlines. Your primary focus will be on optimizing and tuning Spark workloads to enhance performance, stability, and efficiency. Their platform is based on Azure Databricks, with heavy usage of Delta Lake tables. We also have data integrations to various relational databases.
Key Responsibilities
* Analyse, troubleshoot, and optimise the performance of Spark workloads
* Ensure the ingestion from relational databases (usually via JDBC connections) provides optimal and reliable throughput
* Perform benchmarks, interpret and create executions from Spark execution plans and prepare examples (e.g. in notebooks) to unlock better performance and efficiency
* Analyze our Python based codebases for optimization potential
* help teams with application development, including complicated cashflow preparation logic
* Monitor and fine-tune Spark and Databricks clusters for cost-effectiveness and operational excellence
Top 3 essentials skills/experience-based requirements
* Significant hands-on experience with Apache Spark
* Experience with the Databricks platform, preferably in Azure
* Excellent communication skills in English (written and verbal)
Desired Skills And Qualifications
* Significant hands-on experience with Apache Spark, including detailed and demonstrable expertise in performance optimisation and tuning.
* Deep understanding of Spark internals, execution plans, different types of join, table layouts incl. liquid clustering, and Adaptive Query Execution (AQE).
* Extensive experience optimizing Spark MERGE statements, joins, aggregations, and transformations.
* Practical experience connecting Spark with relational databases using JDBC and optimizing their throughput
* Expert level skillset with Python and PySpark
* Experience with the Databricks platform, preferably in Azure
* Strong analytical skills to quickly diagnose performance bottlenecks and implement effective solutions
* Excellent communication skills in English (written and verbal)
Seniority level
* Seniority level
Mid-Senior level
Employment type
* Employment type
Part-time
Job function
* Job function
Information Technology
* Industries
IT Services and IT Consulting
Referrals increase your chances of interviewing at Randstad Digital Switzerland by 2x
Software Engineer, ML & Data Platforms (100%)
Professional Database Engineer - MS SQL (a) 80-100%
Software Engineer III, AI/ML, Google Search
Software Engineer II, Machine Learning, Google Search
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr