Lead Spark Data Engineer

About Confidencial (Apenas para Cadastrados)

Confidencial (Apenas para Cadastrados) is a leading AI strategy, talent, and education services provider. Founded by Sameer Maskey Ph.D., Adjunct Associate Professor at Columbia University, Confidencial (Apenas para Cadastrados) has a core mission of democratizing AI. With a presence in 4 countries (Nepal, the United States, Canada, and the Dominican Republic) and more than 450 full-time employees, Confidencial (Apenas para Cadastrados) brings global AI expertise to transform companies worldwide. Founded in 2013, Confidencial (Apenas para Cadastrados) is a global provider of enterprise AI products and services, on a mission to democratize AI. Leveraging proprietary AI Studio and AI Engines, the company helps drive the clients’ AI Enterprise Transformation, regardless of where they are in their Digital AI journeys. With offices in North America, Asia, and Latin America, Confidencial (Apenas para Cadastrados) provides a suite of enterprise AI offerings and specialty services that allow organizations of any size to implement and scale AI. Confidencial (Apenas para Cadastrados) serves companies in industries such as retail, manufacturing, and government.

Confidencial (Apenas para Cadastrados) continues to actively pursue the mission of democratizing AI for the masses by providing high-quality AI education in underserved communities and helping organizations achieve their full potential with AI.

Job Description:

We are looking for an experienced Lead Data Engineer to join our team to build the "Brain" of an IoT platform, a library that allows definition and metrics, validates it against a Virtual Schema, and generates optimized execution plans for both Spark (Batch) and Flink (Stream).

Qualification / Skill Set Requirement:

• 5+ years of hands-on data engineering experience with deep expertise in the Azure ecosystem.
• Expert-level Java, Python and SQL.
• Deep understanding of Apache Spark Internals (Catalyst Optimizer, Logical Plans).
• Experience with ANTLR v4 or writing custom DSLs/Parsers.
• Experience with Databricks and Delta Lake optimization.
• Experience constructing Abstract Syntax Trees (ASTs).
• Strong understanding of SDLC and Agile methodologies with hands-on experience in Azure DevOps, GitHub, CI/CD, and artifact management.
• Skilled in data modeling, data design, and data warehousing solutions on Azure Databricks.
• Knowledge of data quality, governance, and security best practices within Azure (AD, NSG, encryption, compliance).
• Certifications preferred: Azure Fundamentals, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional and Azure Solutions Architect Expert (nice to have).

Responsibilities

• Architect, design, and implement scalable and efficient data solutions on Spark and Flink.
• Implement the grammar for the IoT Query Language.
• Build the Query Validator to enforce semantic constraints before a query is executed.
• Develop a Spark Adapter: A translation layer that converts definition on metrics into Spark code.
• Implement relationships logic (traversing a Graph/Ontology) within the core to avoid database bottlenecks.
• Ensure 100% logic parity between Spark (Batch) and Flink (Stream) implementations.
• Manage and optimize Azure and Databricks resources, for performance, reliability, and cost-efficiency.
• Transform, clean, and prepare data using SQL, Python and Java.
• Monitor and fine-tune workloads and pipelines for optimal performance and reliability.
• Maintain clear documentation of solutions, configurations, and workflows.
• Actively participate in Agile team activities and continuous improvement initiatives.
• Promote and enforce data engineering best practices, including data governance, security, and data quality.

Confidencial (Apenas para Cadastrados) is an Equal Opportunities Employer, committed to diversity and inclusion. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or any other characteristic protected by applicable federal, state, or local laws.

aiDMSyUTrM