Role Summary

We are seeking a Senior Data Engineer to design, build, and operate the data infrastructure that powers our AI and analytics initiatives. This is not a traditional data engineering role — you will build the foundational data layer for LLM applications, RAG systems, and AI-powered products alongside classic data pipelines and analytics infrastructure. You will own the full data lifecycle: from ingestion and transformation to quality, governance, and serving — with a particular focus on the emerging data patterns required by modern AI systems.

You will be responsible for building and maintaining vector databases and RAG infrastructure, designing high-performance ETL/ELT pipelines, and ensuring data quality at every stage. Your work directly enables AI engineers, data scientists, and business analysts to build and deploy AI-powered solutions with confidence in the underlying data.

Key Responsibilities

Data Pipelines & ETL/ELT

• Design and build scalable, fault-tolerant data pipelines for batch and real-time/streaming workloads;
• Implement modern ELT patterns using dbt, Spark, or Dataflow for transformation within cloud data warehouses;
• Build data ingestion pipelines from diverse sources: APIs, databases, SaaS platforms, file systems, event streams, and document repositories;
• Implement incremental processing, CDC (Change Data Capture), and event-driven pipeline architectures for near-real-time data availability;
• Design pipeline orchestration using Apache Airflow, Prefect, Dagster, or cloud-native workflow services;
• Build and maintain data contracts between producers and consumers to ensure schema stability and backwards compatibility.

Vector Databases & RAG Infrastructure

• Design, deploy, and optimize vector database infrastructure for AI applications: Pinecone, Weaviate, ChromaDB, pgvector, Qdrant, or Milvus;
• Build document ingestion and processing pipelines for RAG: document parsing (PDF, DOCX, HTML, images), chunking strategies (semantic, recursive, sentence-window), and metadata enrichment;
• Implement and optimize embedding generation pipelines using models from OpenAI, Cohere, Voyage AI, or open-source alternatives (BAAI/bge, Nomic);
• Design hybrid search architectures combining dense vector search with sparse retrieval (BM25) and metadata filtering for optimal RAG performance;
• Build and maintain knowledge base management systems: versioned document corpora, incremental indexing, and stale content detection;
• Implement RAG evaluation infrastructure: retrieval accuracy metrics (MRR, NDCG, Hit Rate), context relevance scoring, and end-to-end RAG benchmarks.

Data Quality & Governance

• Design and implement comprehensive data quality frameworks: validation rules, anomaly detection, freshness monitoring, and schema enforcement;
• Build data quality pipelines using Great Expectations, Soda, dbt tests, or Monte Carlo for automated data validation at every pipeline stage;
• Implement data lineage tracking and impact analysis across the data platform;
• Design and enforce data governance policies: access control, data classification, PII detection and masking, and retention policies;
• Build data catalogs and discovery tools that enable self-service data access for AI engineers and analysts;
• Monitor and alert on data quality SLAs: completeness, accuracy, timeliness, and consistency.

Data Platform & Infrastructure

• Design and maintain the core data platform architecture on cloud-native services (AWS, Azure, GCP) — optimizing for cost, performance, and reliability;
• Build and operate data lake/data lakehouse architectures using Delta Lake, Apache Iceberg, or Apache Hudi on cloud object storage;
• Implement data warehouse solutions using Snowflake, Databricks, BigQuery, or Redshift — with proper partitioning, clustering, and materialization strategies;
• Design data serving layers for diverse consumers: low-latency APIs (feature stores), analytical dashboards, AI model training, and RAG retrieval;
• Implement data platform observability: pipeline monitoring, cost tracking, performance dashboards, and capacity planning;
• Build self-service data infrastructure patterns that enable other teams to create and manage their own data pipelines with guardrails.

AI/ML Data Infrastructure

• Build and maintain feature stores for ML model training and serving: offline (batch) and online (real-time) feature computation and storage;
• Design data pipelines for ML workflows: training data preparation, validation sets, evaluation datasets, and model monitoring data;
• Implement data versioning and reproducibility for ML experiments using DVC, LakeFS, or Delta Lake time travel;
• Build feedback loop infrastructure: capturing AI model predictions, user interactions, and ground truth labels for continuous model improvement;
• Design and implement data infrastructure for AI model monitoring: input drift detection, output quality monitoring, and population stability metrics.