Data Engineer

Role summary: The Data Engineer builds the data foundation that AI agents, scorecards, and dashboards run on: ingestion pipelines, connectors, metadata and search indexes, vector stores, data marts, and quality instrumentation. Experience: 3 to 6 years in data engineering. Senior Data Engineer: 2+ years, including ownership of data foundation architecture and connector frameworks. Key responsibilities • Design and build ingestion pipelines for structured data, unstructured content (documents, PDFs), and metadata. • Build reusable connectors to enterprise systems, catalogs, content repositories, and third-party or licensed sources via APIs. • Model and build raw-to-mart data layers that serve analytics and AI use cases. • Implement metadata extraction, enrichment, and search indexing, including semantic and vector indexes for RAG. • Set up and manage vector databases and knowledge repositories used by LLM agents. • Implement data quality rules, profiling, scoring outputs, and exception handling. • Register lineage and maintain source registries, version tracking, and refresh controls. • Design data stores for signals, findings, audit trails, and user feedback loops. • Apply security and governance controls: RBAC, PII/sensitivity flagging, and approved data handling. • Support SIT/UAT data validation, defect fixes, and production release activities. Required skills • Strong Python and SQL; solid data modeling (dimensional and normalized). • Hands-on experience with a modern data platform such as Databricks, Snowflake, or Azure/AWS data services. • Pipeline orchestration and transformation (Spark, Airflow, ADF, dbt, or similar). • API-based integration (REST), JSON handling, and incremental/CDC ingestion patterns. • Data quality frameworks and testing practices for pipelines. • Version control (Git) and CI/CD for data workloads. Preferred skills • RAG data preparation: chunking, embeddings, vector databases (Azure AI Search, pgvector, Pinecone, or similar). • Unstructured content processing: text extraction, OCR, document parsing. • Metadata management, data catalogs, ontologies, or knowledge graphs (for example Neptune or other graph databases). • Experience supporting LLM or agentic applications with grounded, traceable data. • Life sciences data exposure (commercial, medical, regulatory, or launch data) and regulated-data handling. • Cloud certification (Azure Data Engineer, Databricks, AWS, or Snowflake). Department: Engineering Experience: 3-6