Date:
Feb 17, 2025
Location:
Any Marlabs Office Location, IN
Company:
Marlabs Innovations Pvt Ltd
Description:
| • Develop scalable ETL/ELT pipelines using Databricks, Spark, and Delta Lake. • Implement data ingestion frameworks for batch and streaming data using Azure Data Factory, and Spark • Optimize data storage, partitioning, and indexing strategies in Delta Lake. • Write efficient Spark transformations and ensure performance tuning of data processing workloads. • Ensure data quality, validation, and lineage tracking is available for pipelines. • Work with Data Architects to design and implement data models for structured and semi-structured data. • Develop monitoring and logging for ETL jobs to ensure reliability and troubleshooting capabilities. • Ensure role-based access control (RBAC) and security best practices for data access and sharing. |