Logo of Huzzle

Senior Data Engineer

  • Job
    Full-time
    Senior Level
  • Data
  • New York City, +3

AI generated summary

  • You must have 5+ years of data engineering experience with Scala, Python, AWS, SQL, Spark, Airflow, and strong problem-solving skills. Knowledge of MPP/cloud databases, GraphQL, AWS, Data Warehousing, and Agile methodologies is a plus. Bachelor's degree in related field or equivalent experience required.
  • You will maintain and expand data pipelines, implement new batch and streaming pipelines, collaborate with teams, ensure operational efficiency, and drive Core Data platform success.

Requirements

  • At least 5 years of data engineering experience developing large data pipelines
  • Strong algorithmic problem-solving expertise
  • Strong fundamental Scala and Python programming skills
  • Basic understanding of AWS or other cloud provider resources (S3)
  • Strong SQL skills and ability to create queries to analyze complex datasets
  • Hands-on production environment experience with distributed processing systems such as Spark
  • Hands-on production experience with data pipeline orchestration systems such as Airflow for creating and maintaining data pipelines
  • Some scripting language experience
  • Willingness and ability to learn and pick up new skillsets
  • Self-starting problem solver with an eye for detail and excellent analytical and communication skills
  • Preferred Qualifications:
  • Experience with at least one major Massively Parallel Processing (MPP) or cloud database technology (Snowflake, Redshift, Big Query)
  • Experience in developing APIs with GraphQL
  • Deep Understanding of AWS or other cloud providers as well as infrastructure as code
  • Familiarity with Data Modeling techniques and Data Warehousing standard methodologies and practices
  • Familiar with Scrum and Agile methodologies
  • Required Education:
  • Bachelor’s degree in Computer Science, Information Systems, Software, Electrical or Electronics Engineering, or comparable field of study, and/or equivalent work experience

Responsibilities

  • Contribute to maintaining, updating, and expanding existing Core Data platform data pipelines in Scala and Python / Spark while maintaining strict uptime SLAs
  • Extend functionality of current Core Data platform offerings, including metadata parsing, extending the Metastore API, and building new integrations with APIs both internal and external to the Data organization
  • Implement Ingestion of new batch and streaming data pipelines using Scala, Databricks, and Airflow
  • Implement the Lakehouse architecture, working with customers, partners, and stakeholders to shift towards a Lakehouse centric data platform
  • Implementation shared libraries in Scala and Python that abstract complex business logic to allow consistent functionality across all data pipelines across the Data organization
  • Tech stack includes Airflow, Spark, Databricks, Delta Lake, Snowflake, Scala, Python
  • Collaborate with product managers, architects, and other engineers to drive the success of the Core Data platform
  • Contribute to developing and documenting both internal and external standards and best practices for pipeline configurations, naming conventions, partitioning strategies, and more
  • Ensure high operational efficiency and quality of the Core Data platform datasets to ensure our solutions meet SLAs and project reliability and accuracy to all our stakeholders (Engineering, Data Science, Operations, and Analytics teams)
  • Be an active participant and advocate of agile/scrum ceremonies to collaborate and improve processes for our team
  • Engage with and understand our customers, forming relationships that allow us to understand and prioritize both innovative new offerings and incremental platform improvements
  • Maintain detailed documentation of your work and changes to support data quality and data governance requirements

FAQs

What are the key responsibilities of a Senior Data Engineer on the Core Data team?

The key responsibilities include establishing foundational data pipelines and datasets, enabling engineering and analytical teams to utilize data for key business decisions, ensuring observability, lineage, data quality, logging, and alerting across all engineering teams, and working towards creating a single pane of glass for data within the organization.

What skills and qualifications are required for a Senior Data Engineer on the Core Data team?

Candidates should have a strong background in data engineering, experience with data pipelines and datasets, proficiency in observability, lineage, data quality, logging, and alerting tools, and a passion for making a positive impact through data. Additional qualifications may include experience with large-scale streaming services and a strong understanding of data architecture principles.

How important is collaboration and teamwork in this role as a Senior Data Engineer?

Collaboration and teamwork are essential in this role, as the Senior Data Engineer will be working closely with engineering and analytical teams across the organization to establish foundational data pipelines and datasets. The ability to communicate effectively, collaborate with others, and contribute to a positive team dynamic is crucial for success in this position.

Entertainment & Media
Industry
10,001+
Employees

Mission & Purpose

From classic animated features and exhilarating theme park attractions to cutting edge sports coverage, and the hottest shows on television, The Walt Disney Company has been making magic since 1923, creating unforgettable stories that connect with audiences around the world. And we’re just getting started! The key to our success…. The Cast, Crew, Imagineers and Employees who honor Disney’s rich legacy by stretching the bounds of imagination to create the never-before-seen, bringing unparalleled entertainment experiences to people of all ages. Begin a career that delivers unparalleled creative content and experiences to audiences around the world and just imagine the stories you could be part of… What is #LifeAtDisney like? It’s a series of magical moments with cast members and employees developing and telling our stories in the most innovative ways. Whether it’s a day spent as a Disney VoluntEAR, or celebrating the release of a new interactive experience, retail product or movie, our days are filled with the knowledge that we are creating entertainment experiences the whole family can enjoy. Follow @DisneyCareers on Facebook, Twitter and Instagram for a peek behind-the-curtain, and discover how you could connect to a world of stories with Disney!