Note: The job is a remote job and is open to candidates in USA. MediaRadar, Inc. is a company that provides marketing, sales, and analytics leaders with critical intelligence through its AI-enabled platform. The Data Engineering Lead will be responsible for designing scalable systems, integrating Machine Learning models into ETL pipelines, and leading a team towards agile practices while maintaining high performance and cost-efficiency.
Responsibilities
- Architect and implement complex, end-to-end data pipelines using Azure Databricks and PySpark
- Optimize Apache Spark jobs, tune Databricks units, and define cluster policies to minimize compute costs
- Develop a proactive monitoring and alerts framework to ensure 99.9% reliability and mitigate system issues before they impact end-users
- Architect and design high-performance schemas in PostgreSQL, managing indexing, partitioning, and optimizing complex analytical queries
- Lead a lean team toward cross-trained agility, moving away from 'siloed specialists'
- Anticipate future data needs and design High-Velocity Architecture that is highly scalable and manageable to handle sudden volume increases
- Collaborate with ML teams to integrate automated model orchestration into robust ETL pipelines
- Collaborate with the offshore team lead to facilitate seamless knowledge transfer and operational continuity across time zones
Skills
- Core: Python, PostgreSQL + pgvector
- Big Data: Azure Databricks, PySpark, Delta Lake
- DevOps: Docker, Git, Azure DevOps, CI/CD
- 10+ years of experience in Data or Software Engineering with deep codebase involvement
- 3+ years as a Technical Lead managing agile teams
- Proven ability to lead lean, high-impact teams while maintaining high individual output
- Experience with cross-training advocacy and scaling data processing through automation
- Workflow Orchestration: Experience with Apache Airflow
- Containerization: Familiarity with Azure Kubernetes Service (AKS)
Company Overview
MediaRadar, now including the data and capabilities of Vivvix, powers the mission-critical marketing and sales decisions that drive competitive advantage. It was founded in 2006, and is headquartered in New York, New York, USA, with a workforce of 501-1000 employees. Its website is http://www.mediaradar.com.Company H1B Sponsorship
MediaRadar, Inc. has a track record of offering H1B sponsorships, with 1 in 2026, 1 in 2025, 2 in 2024, 2 in 2021. Please note that this does not guarantee sponsorship for this specific role.