Logo of Huzzle

Home

Matches

Explore

Manage

Senior Python Spark Engineer

image

Mastercard

Today

  • Job
    Full-time
    Mid & Senior Level
  • Data
    Software Engineering
  • Dublin

AI generated summary

  • You need 4-6 years in software development, strong Python and Spark expertise, Big Data knowledge, OO design, agile experience, and skills in databases, cloud, security, and mentoring.
  • You will design, implement, and maintain a Python-based big data analytics platform, ensuring code quality, collaborating with teams, and participating in the full software lifecycle in an Agile environment.

Requirements

  • 4-6 years' experience working in software application development as a programmer
  • Strong and solid knowledge of software engineering principles and practices. University degree in Computer Science or equivalent is a MUST. Advanced degree preferred.
  • Strong experience developing Spark applications using Python.
  • Expertise with Big Data Hadoop platforms like Databricks, Cloudera, Teradata, etc. and solid fundamental understanding of the Hadoop architecture
  • Demonstrated professional expertise in understanding and implementing OO design patterns, developing container-based applications in a microservices architecture, using the state-of-the-art software engineering best practices
  • Experience working with relational and NoSQL databases (such as tuning and optimizing complex queries for highly scalable systems) and query languages (specifically Hive/SparkSQL and ANSI SQL)
  • Experience building large scale Spark applications & data pipelines, ideally with Batch processing running on Hadoop clusters
  • Creativity and passion for tackling challenging data problems and willingness to work in a start-up environment
  • Experience with Git or other similar version control software
  • Experience and commitment in developing high quality, robust, scalable, and maintainable software. Knowledge of testing frameworks, CI/CD, test automation, and software monitoring and alerting.
  • Experience working in a high performant collaborative agile team environment
  • Curious, and eager to learn, willing to expand its technical skills while helping other team members to grow
  • Strong communication and interpersonal skills
  • Experience with messaging queues such as Kafka, RabbitMQ or JMS and reactive architecture paradigm
  • Experience designing and developing highly available fault tolerant software.
  • Experience in architecture and development of data models and data dictionaries in big data systems
  • Familiar with at least one large cloud computing provider like Azure, AWS, or IBM cloud, in this order of preference.
  • Experience designing and implementing highly secure software and solutions. Knowledge with secure vulnerability tools such as Checkmarx, Snyk or Nexus IQ.
  • Proven hands-on coding experience with data analytics platforms and workflow engines, including SPSS, Knime, RapidMiner, SAS, Dataiku etc.
  • Experience in operating a UNIX shell and reading and developing simple shell scripts
  • 1-2 years' experience designing software systems, and mentoring junior software engineers

Responsibilities

  • - Participate in the architecture, design and implementation of customer oriented big data analytics platform using Python based on Spark.
  • - Ensure the long-term quality in term of architectural designs and code quality.
  • - Communicate frequently with product management, data pipelines, data science and other teams to coordinate work efforts.
  • - Full software lifecycle participation including design, development, testing, bug fixing, cloud deployment, and maintenance process automation.
  • - Participate in a highly collaborative Agile Scrum team-based software development process.

FAQs

What is the primary role of a Senior Python Spark Engineer at Trūata?

The primary role is to architect and build Trūata's Core Platform to support advanced GDPR compliant big data and data analytic applications using Python and Spark.

What qualifications are required for the Senior Python Spark Engineer position?

A university degree in Computer Science or equivalent is a must, with an advanced degree preferred. Candidates should also have 4-6 years of experience in software application development.

What are the key technologies and skills needed for this role?

Strong experience developing Spark applications using Python, expertise with Big Data Hadoop platforms, and familiarity with relational and NoSQL databases, including query languages like Hive/SparkSQL and ANSI SQL, are essential.

What kind of work environment can a candidate expect?

The candidate can expect a highly collaborative Agile Scrum team-based software development environment that emphasizes software engineering best practices.

Is prior experience with cloud computing providers necessary?

While not mandatory, familiarity with at least one major cloud computing provider like Azure, AWS, or IBM Cloud is considered a great asset.

What are the expectations regarding communication in this role?

Strong communication and interpersonal skills are required, as the role involves frequent communication with product management, data pipelines, and data science teams to coordinate work efforts.

Are there opportunities for professional growth in this position?

Yes, the role is designed for candidates who are intellectually curious and eager to learn, with the opportunity to expand technical skills while mentoring other team members.

How important is software quality and testing in this role?

High importance is placed on developing high quality, robust, scalable, and maintainable software, including experience with testing frameworks, CI/CD, and test automation.

Does this position involve working with messaging queues?

Yes, experience with messaging queues such as Kafka, RabbitMQ, or JMS is considered beneficial for this role.

What is the corporate security responsibility for the role?

All employees are expected to abide by Mastercard's security policies, ensure the confidentiality and integrity of accessed information, and report any suspected security breaches.

Mastercard

Connecting Everyone to Priceless Possibilities

Consulting
Industry
10,001+
Employees
1966
Founded Year

Mission & Purpose

Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments and businesses realize their greatest potential. Our decency quotient, or DQ, drives our culture and everything we do inside and outside of our company. With connections across more than 210 countries and territories, we are building a sustainable world that unlocks priceless possibilities for all.