Senior Data Engineer (Spark & Python Specialist)

Location: Remote (South Africa)
Employment Type: Full-Time
Industry: Data Engineering | Cloud Platforms | Financial Services Technology

WatersEdge Solutions is partnering with a client to recruit a highly skilled Senior Data Engineer (Spark & Python Specialist). This is a strong opportunity for a technically advanced engineer who enjoys building scalable, high-performance data solutions in a modern cloud environment. The role is ideal for someone who thrives on optimisation, code-first engineering, and modernising legacy data logic into clean, portable, Python-centric solutions.

About the Role

As a Senior Data Engineer, you’ll serve as a key technical contributor within the engineering team, focused on building, maintaining, and optimising large-scale data processing engines. You’ll work extensively with Spark, PySpark, Delta Lake, and cloud-based lakehouse environments, helping shape a provider-agnostic platform with strong engineering standards, portability, and performance at its core.

Key Responsibilities

• Optimise Spark-based processing through best practices in memory management, shuffle tuning, and partitioning

• Build and maintain data pipelines using Python, PySpark, Delta Lake, and Parquet

• Refactor legacy SQL-based ETL logic into modular, testable, maintainable Python libraries

• Build and optimise medallion architecture layers across Bronze, Silver, and Gold

• Support code-first orchestration approaches using tools such as Airflow, Dagster, or Python-based wrappers

• Participate in code reviews and mentor junior engineers in PySpark best practices

• Contribute to automated testing frameworks using Pytest

• Work closely with data scientists, analysts, and business stakeholders to deliver fit-for-purpose solutions

• Lead initiatives that strengthen data engineering capability, tooling, and technical best practice

• Ensure strong security, compliance, and governance across engineering processes

What You’ll Bring

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field

• 6+ years of Spark / PySpark experience

• Strong production-grade Python capability

• Solid T-SQL skills with experience interpreting and migrating existing SQL logic

• Experience with Azure Synapse Analytics, Dedicated SQL Pools, and Data Factory

• Hands-on expertise with Delta Lake and Parquet in high-volume environments

• Experience with Docker and open-source, cloud-agnostic engineering standards

• Strong collaboration skills and a proven track record modernising large-scale data workloads

• Experience mentoring engineers and contributing to technical excellence across a team

• Strong understanding of security, compliance, and data governance principles

Nice to Have

• Exposure to Microsoft Fabric

• Experience with code-first orchestration tooling such as Airflow or Dagster

• Experience building reusable internal Python libraries and automated testing patterns

• Background in high-scale cloud data platform modernisation

What’s On Offer

• Fully remote role based in South Africa

• Flexible working arrangements

• Wellness support and home office reimbursement

• Continuous learning opportunities

• Competitive salary, ESOP, and recognition for performance

• Supportive and inclusive team culture focused on accountability and work-life balance

Company Culture

This is a team that values transparency, accountability, inclusion, and technical excellence. You’ll join an environment that supports strong engineering standards, continuous learning, and collaborative problem-solving while giving people the flexibility to do their best work in a sustainable way.

If you have not been contacted within 10 working days, please consider your application unsuccessful.