Azure Data Engineer – Azure Cloud, Services, Databricks, ADT, Python/Pyspark, SQL, SSIS, Healthcare Azure Automation Tech Lead Requirements: Azure, ADT, .NET core, Python/Pyspark, SSIS, SQL Healthcare Domain Experience required Position Overview We are seeking a skilled Azure Databricks Tech Lead to join our dynamic team. As the Azure Databricks Tech Lead, you will play a pivotal role in architecting, implementing, and optimizing Azure Databricks solutions to support our data engineering and analytics initiatives. You will collaborate closely with cross-functional teams to understand business requirements, design scalable data solutions, and lead the execution of projects. Position will require knowledge and experience in the healthcare domain. Key Responsibilities
* Participate and provide technical leadership in all phases of BI projects
* Technical Leadership: Lead the design, development, and implementation of Azure and .NET core solutions ensuring alignment with business objectives and best practices.
* Architecture and Design: Architect robust and scalable data pipelines on Azure Databricks
* Data Engineering: Develop ETL processes, data ingestion pipelines, and data transformations to support data integration and analytics requirements.
* Design and implemented real-time data integration pipeline to process semistructured data, across multiple DBMS. RDBMS and NoSQL using Kafka and PySpark
* Implement and design web and data solutions that use the following Azure services: Azure function App, Azure queue Azure ,Web App, Azure storage, Azure Cosmos DB, Azure Data Factory.
* Excellent knowledge of Python, PySpark, SQL/SSIS, C# etc.
* Performance Optimization: Optimize Azure Databricks workloads for performance, reliability, and cost-efficiency, utilizing techniques such as partitioning, caching, and resource management.
* Collaboration: Collaborate with data scientists, analysts, and business stakeholders to understand requirements, translate them into technical solutions, and deliver actionable insights.
* Experience with data modeling is a plus with building data warehouse from scratch identifying Dimension and Fact entities using Star and Snowflake schema.
* Mentorship and Training: Provide guidance, mentorship, and technical expertise to team members, fostering a culture of continuous learning and development.
* Documentation and Best Practices: Document architecture, design decisions, and technical guidelines, ensuring adherence to best practices and standards
* Discovery and documentation of processes – as-is and to-be
* Creating and executing test plans to ensure the highest quality of products.
* Creating data quality rules and ensuring verification
* Creating and maintaining detailed documentation for all projects.
Qualifications
* Bachelor’s degree in computer science/information technology or related field.
* Minimum 5 years of experience in Azure and .NET core, MS SQL and Azure SQL and over 7 year of experience overall
* Excellent knowledge of different software testing practices and techniques specifically as it relates data pipelines, data integration and building up of a data foundational architecture.
* Excellent verbal and written communication skills.
* Strong problem-solving and critical thinking skills.
* Highly organized and detail-oriented.
Statistical Analysis System (SAS)
Exp – 6+ Years
Compensation – 1.7 LPM
