Logo of Huzzle

Home

Matches

Explore

Manage

Data Engineer II

image

Microsoft

10d ago

  • Job
    Full-time
    Senior Level
  • Data
    IT & Cybersecurity
  • Hyderabad, +1

AI generated summary

  • You need a degree in a related field, 8+ years in data lake development, cloud Big Data apps, ETL/ELT, OLAP systems, data modeling, and coding in Scala/Spark. Security clearance is required.
  • You will design and optimize data pipelines, implement process improvements, collaborate with stakeholders, mentor team members, and drive innovation in a scalable data architecture.

Requirements

  • Bachelor's Degree in Computer Science, Data Science, Mathematics or equivalent field OR Master's Degree in Computer Science, Data Science, Mathematics or equivalent
  • A successful history of architecting, building, and maintaining end-to-end, high-throughput data lake systems and their supporting services.
  • Minimum of 8+ years of hands-on development experience designing and building data lakes/datawarehouses
  • 7+ years of experience developing Big Data applications in the cloud using technologies such as Kusto, Databricks, Spark, Azure Data Factory, Azure ML, Azure Synapse or equivalent.
  • 8+ years of hands-on coding experience on either Scala, Spark SQL, pyspark, good to have optimized Query writing, Performance Tuning, Troubleshooting & development experience.
  • 8+ years of hands-on data pipelines and ETL/ELT experience
  • 5+ years of experience in building OLAP systems like Azure Analysis Services, Power BI etc.
  • Strong understanding of Data modeling including Dimensional and relational modeling.
  • Proficient working knowledge of cloud-computing environments in the data stack - Azure preferred.
  • Utilizing a variety of data stores, including data lakes/warehouses, relational database systems, in-memory caches, and searchable document databases.
  • Developing for continuous integration and automated deployments.
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Responsibilities

  • Understanding of modern data architecture patterns and concepts like Data mesh, Lambda Architecture, Delta lake etc. Understanding existing systems and lay out the path for a more modern and highly scalable data platform.
  • In depth understanding of Spark internals and passion for data engineering at scale. Identify, design, and implement process improvements to optimize our data delivery, and re-designing infrastructure and pipelines to achieve greater scalability.
  • Design, develop, and maintain data pipelines and back-end services for real-time decisioning, reporting, optimization, data collection, and related functions.
  • Ensure that all features embody engineering best practices, efficient, scalable architecture, elegant design, and high-quality implementation.
  • Collaborate with stakeholders, including Product, Executive and Domain teams to support them on data-related issues and data infrastructure needs.
  • Seek and incorporate continuous feedback to improve the efficiency of the business outcomes.
  • A team player who is self-starter, customer obsessed, highly inquisitive and biased towards action while engaging everyone in team.
  • Comfortable with cold start problems, thrive in ambiguity with an eye for detail.
  • Delivers results through innovation and persistence.
  • A person with an eye for details and ‘can-do’ attitude.
  • Provide mentorship and guidance to other team members.

FAQs

What is the primary focus of the Cloud Supply Chain (CSCP) organization at Microsoft?

The primary focus of the CSCP organization is to enable the infrastructure underlying Microsoft's expanding Cloud business, delivering Cloud capacity differentiated at scale through intelligent systems.

What is the role of a Data Engineer in the Decision Sciences Engineering team in Azure?

A Data Engineer in the Decision Sciences Engineering team is responsible for delivering data solutions by combining data science and engineering, designing data pipelines, and optimizing data delivery for real-time decision-making and analysis.

What qualifications are required for the Data Engineer II position?

A Bachelor's or Master's Degree in Computer Science, Data Science, Mathematics or equivalent field, along with 8+ years of hands-on development experience in data lakes/data warehouses and 7+ years of experience with Big Data applications in the cloud is required.

What technologies should a candidate be experienced with for this position?

Candidates should have experience with technologies such as Kusto, Databricks, Spark, Azure Data Factory, Azure ML, Azure Synapse, and coding experience in Scala, Spark SQL, or pyspark.

Is experience with artificial intelligence or machine learning important for this role?

Yes, having a flair for Artificial Intelligence/Machine Learning technology is one of the desired qualifications for the Data Engineer II position.

What is the work culture like in the Decision Sciences Engineering team?

The work culture emphasizes a growth mindset, inclusivity, and leadership, where learning from failures is celebrated, every voice is valued, and team members strive to create clarity and deliver success.

How does the team approach process improvements in data delivery?

The team identifies, designs, and implements process improvements focused on optimizing data delivery and redesigning infrastructure and pipelines to achieve greater scalability.

What does the term "cold start problems" refer to in this context?

"Cold start problems" refer to situations where there is little to no data available to formulate a prediction or recommendation, and candidates for this position should be comfortable working in such ambiguous situations.

Are mentorship and guidance expected from this role?

Yes, providing mentorship and guidance to other team members is part of the responsibilities for the Data Engineer II position.

What ongoing requirements does the position have regarding security screenings?

The position requires passing the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter to meet Microsoft, customer, and/or government security screening requirements.

image

Microsoft

Technology
Industry
10,001+
Employees
1975
Founded Year

Mission & Purpose

Every company has a mission. What's ours? To empower every person and every organization to achieve more. We believe technology can and should be a force for good and that meaningful innovation contributes to a brighter world in the future and today. Our culture doesn’t just encourage curiosity; it embraces it. Each day we make progress together by showing up as our authentic selves. We show up with a learn-it-all mentality. We show up cheering on others, knowing their success doesn't diminish our own. We show up every day open to learning our own biases, changing our behavior, and inviting in differences. When we show up, we achieve more together. Microsoft operates in 190 countries and is made up of more than 220,000 passionate employees worldwide.