Date:
Feb 17, 2025
Location:
Any Marlabs Office Location, IN
Company:
Marlabs Innovations Pvt Ltd
Description:
| • Provide technical leadership for Data Engineers and guide the implementation of Databricks-based data processing pipelines. • Define best practices for ETL/ELT development, data transformation, and data quality using Databricks and Delta Lake. • Lead the design and implementation of data ingestion pipelines from various structured and unstructured sources into Databricks and ADLS. • Establish data ingestion frameworks for batch ingestion using Azure Data Factory and other relevant tools. • Optimize data storage, partitioning, and indexing strategies in Delta Lake to ensure efficient query performance. • Collaborate with Data Architects and Data Engineers to ensure efficient data modeling, governance, and security in Databricks. • Troubleshoot and optimize Spark jobs, query performance, and resource utilization. • Ensure role-based access control (RBAC) and security best practices for data access and sharing. • Conduct knowledge-sharing sessions and mentor the Data Engineering team. |