Logo of Huzzle

Data Engineer

image

UST

4d ago

  • Job
    Full-time
    Senior Level
  • Data
    Software Engineering
  • Toronto

AI generated summary

  • You must be proficient in Apache Spark/PySpark, AWS Glue, Python, and SQL. Experience with ETL design, AWS services, Airflow, and data warehousing concepts is essential.
  • You will design and implement data pipelines, optimize ETL processes using AWS Glue and Spark, write Python code, and collaborate with teams while documenting workflows and best practices.

Requirements

  • Proficient in using Apache Spark and PySpark for big data processing and transformation.
  • Hands-on experience with AWS Glue for building ETL workflows in the cloud.
  • Strong programming skills in Python, particularly for data manipulation, automation, and integration with Spark and Glue.
  • Solid understanding of ETL principles and data pipeline design, including optimization techniques.
  • Experience working with AWS services, particularly those related to data processing (e.g., S3, Glue, Lambda, Redshift).
  • Must have proficiency in writing optimized SQL with performance tuning aspects.
  • Ability to translate complex business requirements into technical solutions.
  • Experience with Apache Airflow for orchestrating data workflows.
  • Knowledge of data warehousing concepts and cloud-native analytics tools.

Responsibilities

  • Design and implement scalable data processing pipelines using Apache Spark and PySpark for large-scale data transformations.
  • Develop, maintain, and optimize ETL processes to efficiently extract, transform, and load data across various data sources and destinations.
  • Utilize AWS Glue for serverless ETL jobs, including creating, running, and monitoring Glue jobs for data transformations and integrations.
  • Write efficient and reusable Python code to support data manipulation, analysis, and transformation in Spark and Glue environments.
  • Ensure that data workflows are optimized for performance, scalability, and cost-efficiency on AWS Cloud.
  • Work closely with data analysts, data scientists, and other engineering teams to build reliable data solutions that support business analytics and decision-making.
  • Document processes, workflows, and code, while adhering to best practices in data engineering, cloud architecture, and ETL design.

FAQs

What is the primary role of a Data Engineer at UST?

The primary role of a Data Engineer at UST involves designing, building, and optimizing data workflows on AWS Cloud using Apache Spark, PySpark, and Python for scalable data processing and ETL operations.

What skills are essential for this position?

Essential skills for this position include proficiency in Apache Spark and PySpark, hands-on experience with AWS Glue, strong programming skills in Python, and a solid understanding of ETL principles and data pipeline design.

What AWS services should candidates be familiar with?

Candidates should be familiar with AWS services such as Glue, S3, Lambda, and Redshift, especially in the context of data processing and ETL workflows.

Is there a requirement for knowledge of SQL?

Yes, candidates must have proficiency in writing optimized SQL with performance tuning aspects to effectively manipulate and analyze data.

How does UST prioritize its work culture?

UST prioritizes a people-first, human-centric culture that fosters diversity, embraces humility, humanity, and integrity, and commits to sustainable solutions.

What benefits are offered to full-time employees at UST?

Full-time employees receive a minimum of 10 days of paid vacation, 6 days of paid sick leave, paid holidays, and eligibility for healthcare coverage along with life insurance and accidental coverage.

Does UST support professional growth and innovation?

Yes, UST encourages innovation and professional growth by empowering its teams to act nimbly and create a lasting and sustainable impact for clients.

What is the compensation range for the Data Engineer role?

The compensation range for the Data Engineer role is between $98,000 and $120,000, dependent on various factors including location, skills, education, and experience.

Is there an opportunity for collaboration in this role?

Yes, the Data Engineer will work closely with data analysts, data scientists, and other engineering teams to build reliable data solutions that support business analytics and decision-making.

What kind of work environment does UST promote for its employees?

UST promotes a work environment free of discrimination and harassment, committed to equal opportunity employment practices and valuing diversity among employees.

IT Services and IT Consulting

Technology
Industry
10,001+
Employees

Mission & Purpose

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 26,000 employees in 25 countries, UST builds for boundless impact—touching billions of lives in the process.