Note: The job is a remote job and is open to candidates in USA. Cardlytics is an industry-leading purchase intelligence and incentives platform founded in 2008. They are seeking a Principal Software Engineer, Tech Lead for their Databricks Platform to manage and evolve their Databricks environment on AWS while leading a small team and collaborating with various stakeholders.
Responsibilities
- Platform Administration: Own day-to-day administration of the Databricks environment on AWS, including workspace configuration, cluster policies, user and group management, access controls, and cost governance
- Architecture & Design: Design and evolve the Cardlytics Databricks architecture; establish reference architectures and standards for batch, streaming, and ML workloads
- Governance & Security: Drive adoption of Unity Catalog for centralized RBAC, column-level security, data lineage, and metadata management; implement and maintain security best practices including encryption and audit logging
- Performance & Reliability: Monitor platform health, define SLAs, and lead performance tuning for Spark jobs, Delta tables, and SQL warehouses; own incident response and root-cause analysis
- User Enablement: Serve as the primary technical partner for Analysts, ML Engineers, and Business users; develop self-service guides, onboarding materials, and best-practice documentation to reduce friction across all user groups
- Infrastructure as Code: Own the CI/CD strategy for Databricks deployments using Terraform, GitHub Actions, and Databricks Asset Bundles; enforce DevOps practices across the team
- Leadership: Lead and mentor a small team of data/platform engineers; provide technical direction, drive engineering excellence, and manage the Databricks vendor relationship
Skills
- 7+ years of experience in data engineering or data platform roles, with at least 2 years in a technical lead or architecture capacity
- Deep, hands-on expertise with the Databricks platform: Delta Lake, Spark (PySpark), Databricks SQL, Unity Catalog, Workflows, and cluster management on AWS
- Proven experience designing and delivering Lakehouse architectures and large-scale ELT/ETL pipelines in production environments
- Proficiency in Infrastructure as Code tools, preferably Terraform
- Demonstrated experience supporting diverse stakeholder groups including Analysts, ML Engineers, and non-technical Business users
- Strong understanding of data governance, data modeling, and cloud security best practices
- Excellent communication and collaboration skills, with the ability to translate technical concepts for non-technical audiences
- Familiarity with or willingness to work within a macOS and Google Workspace environment
Benefits
- Flexible paid time off plus company holidays
- Medical, dental, and vision insurance begins on your first day
- 401(k) retirement plan with company match, plan also includes a student loan debt repayment option
- Employee Stock Purchase Plan
- Educational assistance for continuing education
- Lifestyle Spending Account for physical, emotional, and financial wellness (like gym memberships, home down payments, art classes, park passes, and more!)
- Complimentary Calm app subscriptions to support employee mental health and wellbeing
Company Overview
Company H1B Sponsorship