1. Working with Agentic AI: Data Engineer

How Human Skills and Agentic Skills Dynamically Combine Across Data Engineering Responsibilities

1.1. Data Engineering as Professional Work

A data engineer designs, builds, integrates, operates, and improves the technical systems through which data is collected, transported, transformed, stored, organized, governed, and made available for analytical, operational, reporting, machine-learning, and other computational uses. The role involves understanding data requirements, examining source systems and data structures, designing data architectures and models, developing batch and streaming pipelines, implementing data transformations, integrating heterogeneous data sources, managing databases and data platforms, establishing data-quality controls, optimizing data-processing performance, monitoring production data workflows, troubleshooting failures, maintaining metadata and documentation, and supporting secure and reliable access to data. data engineers also collaborate with data analysts, data scientists, machine-learning engineers, software engineers, platform engineers, business stakeholders, governance specialists, security teams, and other participants who produce, manage, or consume data. Their responsibilities therefore extend beyond moving data between systems: they continuously connect source data, transformation logic, storage technologies, schemas, pipelines, infrastructure, quality requirements, downstream consumers, security constraints, operational conditions, and changing organizational needs while applying engineering knowledge and judgment to ensure that data remains accurate, available, scalable, traceable, maintainable, and appropriate for its intended uses.

1.2. Working with Agentic AI

Agentic AI can participate across data engineering activities by applying the comprehension, reasoning, comparison, generation, coding, evaluation, classification, and tool-use capabilities of increasingly advanced large language models and other foundation models together with data context, engineering instructions, schemas, metadata, source-system information, technical documentation, data-platform tools, code repositories, monitoring systems, and reusable or dynamically constructed agentic skills. Within a data lifecycle, an AI agent can analyze data requirements, inspect schemas and metadata, discover relationships among data sources, generate preliminary data models, develop or modify SQL and data-processing code, construct transformation logic, assist with pipeline development, generate data-quality checks, compare schemas, identify anomalous data conditions, analyze pipeline logs and failures, inspect dependencies and lineage, prepare tests and documentation, support query and workload optimization, and consolidate information distributed across data platforms and engineering systems. These activities can also be organized into goal-directed workflows in which the agent decomposes a data engineering task into subtasks, retrieves relevant technical context, selects appropriate tools, generates or executes authorized code, evaluates intermediate outputs, detects failures or inconsistencies, and modifies subsequent actions as schemas, source systems, data volumes, quality conditions, pipeline results, or downstream requirements change. Their practical value does not depend on transferring an entire data engineering responsibility to AI. Instead, agentic skills can participate selectively across individual tasks and subtasks, while the data engineer provides architectural judgment, data-context interpretation, production awareness, technical validation, risk evaluation, and engineering decision-making where reliability, scalability, security, maintainability, or downstream consequences require them.

1.3. Human Skills and Agentic Skills

Human skills in data engineering refer to the professional capabilities that a data engineer applies to understand and manage relationships across data sources, schemas, pipelines, transformations, storage systems, infrastructure, applications, analytical workloads, data consumers, governance requirements, and production environments. They include data architecture knowledge, software and database engineering, data modeling, systems thinking, debugging, performance reasoning, experiential judgment, requirements interpretation, security awareness, interdisciplinary communication, and the ability to recognize how changes in one part of a data system may affect downstream datasets, applications, reports, models, or operational processes. Human skills are particularly important when data requirements are ambiguous, source systems behave unexpectedly, architectural alternatives involve competing performance and maintainability objectives, failures span multiple technologies, data meaning depends on organizational context, or engineering decisions involve significant reliability, security, privacy, cost, or production consequences. data engineers also perform activities that may overlap substantially with AI capabilities, including coding, schema analysis, data transformation, debugging, query optimization, documentation, and technical problem solving. Human capability in data engineering should therefore not be defined simply by separating human coding or analytical work from activities that AI can also perform.

Agentic skills in data engineering are task-oriented capabilities through which an AI agent can participate in the design, implementation, operation, testing, and improvement of data systems. Depending on the available models, technical information, tools, and authorization, these skills may inspect schemas, analyze metadata, generate SQL or data-processing code, construct transformation logic, map fields between source and target systems, compare schema versions, identify data-quality problems, generate tests, analyze pipeline dependencies, trace data lineage, examine logs, investigate failed jobs, suggest query or processing optimizations, generate infrastructure or configuration artifacts, prepare documentation, and monitor operational indicators for conditions requiring attention. Agentic skills can draw simultaneously on model capabilities and system-specific information contained in databases, data catalogs, schemas, pipeline definitions, code repositories, orchestration systems, cloud platforms, monitoring environments, documentation, quality records, and other authorized sources. Their usefulness consequently changes with the particular data architecture, technology stack, workload, data sensitivity, production environment, information quality, and reliability required for the task.

The important distinction in data engineering is therefore not that coding, data modeling, pipeline development, debugging, optimization, or other technical activities inherently belong either to the data engineer or to AI. Both human and agentic capabilities may contribute to many of these activities, but their suitability can differ substantially according to the engineering situation. An agentic skill may rapidly inspect hundreds of schemas for compatibility problems while a Data Engineer determines whether the proposed mappings preserve the actual meaning of the data; AI may generate transformation code while the engineer evaluates whether its assumptions, performance characteristics, and failure behavior are appropriate for production; and AI may analyze pipeline logs and identify probable causes while the engineer determines whether a proposed correction could affect dependent datasets or downstream consumers. The practical question is therefore which available human or agentic skill, or combination of skills, should perform a particular data engineering task or subtask under the current data, architectural, operational, technical, and organizational conditions.

1.4. Dynamic Allocation of Tasks and Subtasks

A professional responsibility such as data pipeline development, data modeling, platform optimization, or production troubleshooting is usually composed of multiple tasks and subtasks rather than being a single indivisible activity. These subtasks may require different combinations of comprehension, information retrieval, coding, data analysis, technical reasoning, architectural judgment, debugging, generation, evaluation, decision-making, and execution. The participation of human skills and agentic skills can therefore vary substantially within the same responsibility.

The allocation of work should be understood as dynamic rather than fixed. A subtask may initially be performed by an agentic skill, transferred to the data engineer when architectural or production judgment is required, returned to AI for additional analysis after new information is obtained, and later validated by the data engineer before the result is incorporated into a production data system. Other subtasks may be performed independently by the professional or independently by an AI agent when their capabilities and authority are sufficient. Allocation can change according to task complexity, data sensitivity, production consequences, available information, model capability, tool availability, output reliability, workload characteristics, organizational policies, intermediate results, and changing data-system conditions.

A useful way to understand this structure is:

Occupation → Responsibility → Task → Subtask → Human and/or Agentic Skills → Evaluation → Possible Reallocation

The purpose of integrating agentic skills is therefore not to eliminate traditional professional skills. It is to expand the set of capabilities available for performing work and to enable the professional to determine, continuously and contextually, how human and agentic capabilities can be combined most effectively.

1.5. Data Engineering Responsibilities

1.5.1. Data Requirements and Source Analysis

Data requirements and source analysis involves understanding what data is needed, where it originates, how it is structured, what it represents, how frequently it changes, how it will be consumed, and what technical or organizational constraints affect its use. Human skills are particularly important when communicating with data producers and consumers, interpreting the meaning of fields and business concepts, recognizing ambiguous requirements, understanding source-system behavior, and determining whether available data can support the intended use. Agentic skills can inspect schemas, metadata, documentation, sample data, queries, pipeline definitions, and other authorized sources to identify candidate datasets, summarize structures, detect relationships, compare source characteristics, and identify missing or inconsistent information. Data discovery can therefore become iterative as human clarification generates additional context for agentic analysis and AI findings reveal questions requiring further investigation.

1.5.2. Data Architecture and Platform Design

Data architecture and platform design determines how data should move through and be organized across ingestion, processing, storage, serving, governance, and consumption layers. Human skills contribute architectural experience, systems thinking, technology knowledge, workload understanding, and judgment about scalability, reliability, maintainability, security, latency, and cost trade-offs. Agentic skills can retrieve technical information, compare architectural patterns, analyze requirements against platform capabilities, generate preliminary architecture alternatives, identify dependencies, and evaluate options against specified criteria. Rather than generating a single architecture, agentic reasoning can help explore alternatives while the data engineer determines which designs are appropriate for actual workloads and operational constraints.

1.5.3. Data Pipeline Development

Data pipeline development involves building workflows that reliably move data from source systems through ingestion, transformation, processing, storage, and delivery stages. Pipelines may operate in batches, near real time, or through continuous streaming depending on data and application requirements. Human skills are important for selecting appropriate pipeline patterns, understanding source and target behavior, designing failure handling, managing dependencies, and ensuring that pipeline behavior remains appropriate under production conditions. Agentic skills can generate preliminary pipeline code and configurations, construct transformation steps, identify dependencies, create orchestration definitions, generate tests, inspect existing implementations, and suggest modifications as requirements change.

1.5.4. Data Integration and Ingestion

Data integration and ingestion brings information from databases, applications, APIs, files, event streams, external platforms, and other sources into data systems where it can be processed and used consistently. Human skills are important for understanding source-system semantics, choosing integration approaches, resolving incompatible representations, evaluating extraction impacts, and determining how changes in upstream systems should be handled. Agentic skills can analyze source and target schemas, generate connectors or ingestion logic, map fields, compare formats, identify incompatibilities, generate conversion code, and inspect ingestion results for anomalies.

1.5.5. Data Transformation and Processing

Data transformation and processing converts raw or intermediate data into structures and representations appropriate for analytical, operational, reporting, or machine-learning uses. Human skills contribute understanding of data meaning, transformation requirements, business rules, processing dependencies, and the consequences of changing transformation logic. Agentic skills can generate SQL and processing code, translate transformation specifications into executable logic, refactor existing transformations, identify repetitive patterns, compare outputs, generate tests, and analyze intermediate processing results. Human validation remains important because syntactically correct transformations may still alter meaning, introduce unintended exclusions, or produce downstream inconsistencies.

1.5.6. Data Modeling and Schema Design

Data modeling and schema design defines how entities, attributes, relationships, events, metrics, and other data concepts should be represented for storage and downstream use. Human skills are important for understanding domain meaning, selecting appropriate levels of normalization or denormalization, anticipating access patterns, balancing analytical and operational requirements, and determining whether a model will remain understandable and maintainable. Agentic skills can analyze source schemas, infer candidate relationships, generate preliminary logical or physical models, compare modeling alternatives, identify inconsistencies, and generate schema definitions or migration artifacts.

1.5.7. Database, Data Warehouse, and Data Lake Engineering

Data Engineers develop and maintain storage environments such as relational databases, analytical warehouses, lakehouses, data lakes, distributed stores, and other platforms according to workload requirements. Human skills contribute technology selection, storage design, partitioning strategy, access-pattern understanding, reliability planning, and judgment about operational trade-offs. Agentic skills can generate database definitions, analyze schemas and workloads, inspect query patterns, suggest partitioning or indexing approaches, generate migration scripts, compare configurations, and assist with platform administration.

1.5.8. Data Quality Engineering

Data quality engineering establishes mechanisms for identifying and preventing incomplete, inaccurate, inconsistent, duplicated, stale, malformed, or otherwise unreliable data. Human skills are important for determining what constitutes acceptable data within a particular domain, defining meaningful quality expectations, interpreting anomalies, and deciding whether a detected issue requires correction, exclusion, escalation, or acceptance. Agentic skills can profile datasets, generate quality rules, identify anomalies, compare distributions, detect schema changes, analyze recurring failures, generate validation code, and organize quality results across large data collections.

1.5.9. Data Testing and Validation

Data testing and validation determines whether pipelines, transformations, models, schemas, and datasets behave according to defined expectations. Human skills are important for determining what should be tested, defining meaningful expected outcomes, interpreting discrepancies, and deciding whether results are acceptable for downstream use. Agentic skills can generate unit, integration, regression, reconciliation, and data-quality tests; derive test cases from transformation logic or requirements; compare outputs; identify missing coverage; and summarize failures.

1.5.10. Data Orchestration and Workflow Management

Data orchestration coordinates the execution, scheduling, dependency management, retries, sequencing, and monitoring of interconnected data-processing activities. Human skills contribute workflow design, understanding of critical dependencies, service-level expectations, failure behavior, and production priorities. Agentic skills can analyze dependency graphs, generate orchestration configurations, detect scheduling conflicts, identify failed dependencies, recommend retry or recovery actions, and update workflow definitions when authorized changes occur.

1.5.11. Data Performance and Scalability Optimization

Performance and scalability optimization ensures that data systems continue to operate effectively as data volumes, processing complexity, concurrency, and downstream demand change. Human skills are important for understanding workload characteristics, identifying meaningful bottlenecks, evaluating architectural trade-offs, and determining whether performance improvements justify additional cost or complexity. Agentic skills can analyze query plans, execution statistics, pipeline metrics, resource utilization, partitioning patterns, and code to identify candidate bottlenecks and generate possible optimizations.

1.5.12. Data Security, Privacy, and Access Control

Data Engineering must account for how data is protected, accessed, transmitted, stored, and processed according to security, privacy, governance, and organizational requirements. Human skills are important for interpreting data sensitivity, understanding intended use, evaluating consequences of inappropriate access, and coordinating with security, privacy, and governance specialists. Agentic skills can inspect configurations, identify potentially inconsistent permissions, classify documented data characteristics, compare implementations with defined policies, analyze access patterns, and generate preliminary configuration or compliance checks where authorized.

1.5.13. Metadata, Lineage, and Data Governance Support

Metadata and lineage describe what data exists, what it means, where it originates, how it changes, and how it moves through data systems. Human skills are important for validating semantic meaning, determining which relationships matter, resolving unclear ownership, and connecting technical metadata with organizational context. Agentic skills can extract metadata, infer lineage from code and pipeline definitions, connect datasets with transformations, identify missing documentation, compare catalog information with implemented systems, and maintain structured relationships as data environments evolve.

1.5.14. Monitoring, Reliability, and Production Operations

Production Data Engineering requires continuous observation of pipeline execution, data freshness, processing latency, resource consumption, failure rates, quality indicators, and other signals that reveal whether data systems are operating correctly. Human skills are important for defining meaningful operational indicators, evaluating incident severity, coordinating responses, and determining whether failures require immediate intervention or broader architectural changes. Agentic skills can monitor authorized telemetry, correlate events, summarize failures, identify unusual patterns, retrieve relevant runbooks, generate candidate diagnoses, and organize incident information.

1.5.15. Data Pipeline Troubleshooting and Incident Resolution

Troubleshooting addresses failed jobs, missing or delayed data, incorrect transformations, schema mismatches, performance degradation, infrastructure problems, and unexpected downstream results. Human skills are important for framing incidents, interpreting incomplete evidence, understanding cross-system interactions, developing hypotheses, and evaluating whether proposed corrections could create additional problems. Agentic skills can analyze logs, code, schemas, dependency information, recent changes, monitoring data, and previous incidents to identify patterns, correlate failures, generate candidate causes, and suggest possible corrective actions.

1.5.16. Data Migration and Platform Modernization

Data migration and platform modernization moves datasets, transformations, workloads, and data services between technologies or architectures while preserving required meaning, quality, availability, security, and downstream behavior. Human skills contribute migration strategy, architectural judgment, risk assessment, sequencing, compatibility evaluation, and understanding of downstream dependencies. Agentic skills can inventory data assets, compare source and target schemas, generate mapping logic, translate queries or transformations, identify compatibility problems, generate migration tests, and analyze reconciliation results.

1.5.17. Technical Documentation and Data Knowledge Management

Data Engineers maintain documentation describing data architectures, pipelines, schemas, transformations, dependencies, operational procedures, data contracts, quality expectations, and other technical information needed to understand and operate data systems. Human skills remain important for determining what information is significant, explaining architectural intent, validating technical meaning, and ensuring that documentation accurately represents production behavior. Agentic skills can generate preliminary documentation from code and metadata, summarize pipeline behavior, identify undocumented components, normalize terminology, compare documentation with implementations, and update authorized artifacts as systems change.

1.5.18. Cross-Functional Data Collaboration

Data Engineers collaborate with analysts, data scientists, machine-learning engineers, software engineers, platform teams, security specialists, governance functions, product teams, and business stakeholders to ensure that data products and infrastructure support downstream needs. Human skills are central to clarifying requirements, negotiating interfaces and responsibilities, resolving conflicting priorities, explaining technical constraints, and coordinating consequential decisions. Agentic skills can summarize discussions, consolidate requirements, organize action items, retrieve relevant technical context, identify unresolved questions, and maintain structured knowledge across teams and systems.

1.5.19. Data Engineering Process and Platform Improvement

Data Engineering process and platform improvement examines how pipeline development, testing, deployment, monitoring, data quality, documentation, orchestration, and platform operations can be made more reliable, efficient, repeatable, and maintainable. Human skills contribute engineering experience, organizational understanding, prioritization, and judgment about whether an improvement reduces long-term complexity or merely introduces additional tooling. Agentic skills can analyze recurring incidents, code patterns, workflow information, quality failures, performance metrics, development histories, and operational evidence to identify recurring problems and generate candidate improvements.

1.5.20. Data, Platform, and Technology Knowledge

Data Engineers need to maintain current knowledge of database systems, distributed computing, data-processing frameworks, cloud data platforms, storage technologies, orchestration systems, programming languages, data modeling, streaming technologies, governance approaches, security practices, and emerging methods that may affect data-system design and operation. Human skills contribute professional understanding, experiential knowledge, interpretation of relevance, and the ability to connect new technologies with specific workloads and organizational needs. Agentic skills can retrieve and summarize technical documentation, compare technologies, organize platform updates, explain unfamiliar concepts, generate experimental implementations, and filter information according to current engineering priorities. As AI becomes an increasingly important consumer and producer of organizational data, Data Engineers may also need to understand how data architecture, data quality, retrieval systems, metadata, governance, and pipeline design affect AI and agentic applications.

1.6. Human–Agentic Involvement Across Data Engineering

The relative involvement of human and agentic skills differs not only between responsibilities but also among the tasks and subtasks contained within the same responsibility. The following matrix therefore represents common tendencies rather than fixed assignments.

1.7. Why the Division of Work Changes

The boundary between human and agentic participation should not be treated as permanent. It can change from one data platform, dataset, pipeline, workload, engineering responsibility, task, subtask, or moment of execution to another.

These factors can operate simultaneously. A task that is largely agentic under one set of conditions may become human-led under another, while improvements in models, agentic skills, data platforms, development tools, information systems, and organizational processes may gradually change the practical boundary over time.

1.8. Developing Data Engineering Capability with Agentic AI

The increasing availability of agentic skills does not eliminate the need for traditional Data Engineering capabilities. Data architecture, software engineering, database knowledge, data modeling, pipeline design, debugging, performance reasoning, production awareness, security understanding, and engineering judgment remain important because they help the Data Engineer understand what should be built, determine whether generated implementations are technically appropriate, and connect individual data-processing activities with the reliability and usefulness of the broader data environment.

At the same time, professional capability increasingly includes the ability to work effectively with agentic systems. A Data Engineer may need to determine which engineering tasks can be delegated, provide the necessary schemas, metadata, requirements, architectural context, and operational constraints, select appropriate agentic skills, evaluate generated code and intermediate results, identify unsupported assumptions or incorrect transformations, request refinement, combine outputs from multiple agentic activities, recognize when direct engineering intervention is necessary, and incorporate validated outputs into controlled development and production workflows. Competence therefore increasingly includes not only writing code or constructing data pipelines directly but also orchestrating how Data Engineering work should be performed across available human and agentic capabilities.

This creates a broader model of professional capability:

Professional Capability = Human Skills + Agentic Skills + Human–Agentic Coordination

The proportions of these components will differ across responsibilities and will continue to change as models, data technologies, platforms, organizations, and engineering practices evolve.

1.9. The Evolving Boundary Between Human and Agentic Skills

The relationship between a Data Engineer and Agentic AI is best understood as an adaptive division of work rather than a fixed separation between human tasks and AI tasks. Some subtasks may be performed largely through agentic skills, especially where large amounts of schemas, metadata, pipeline definitions, SQL, transformation code, logs, quality results, lineage information, technical documentation, or operational evidence must be retrieved, generated, compared, tested, classified, analyzed, checked, or repeatedly updated. Other activities may require substantial human participation because they involve architectural trade-offs, ambiguous data meaning, production risk, cross-system dependencies, security or privacy consequences, stakeholder coordination, incomplete evidence, or consequential engineering decisions. Between these areas lies a much larger collection of tasks in which human and agentic skills can interact through delegation, code generation, execution, information exchange, analysis, review, testing, correction, evaluation, refinement, escalation, and repeated reassignment.

The resulting work arrangement is therefore better represented as an adaptive human–agentic workflow than as a simple sequence in which either the professional or AI performs an entire responsibility independently. At one stage of pipeline development, an AI agent may perform much of the schema inspection, SQL generation, transformation implementation, testing, documentation, or quality analysis while the Data Engineer supervises and validates the results. At another stage, the Data Engineer may lead architectural decisions, production incident response, migration planning, security-sensitive changes, or performance trade-offs while AI performs supporting analysis, code generation, information retrieval, or evidence organization. A subtask can move from human execution to agentic execution, return to human review, and later be delegated again as schemas change, pipelines execute, tests fail, data-quality problems emerge, source systems evolve, workloads increase, or downstream requirements change.

The long-term value of Agentic AI in Data Engineering therefore lies not only in automating individual coding or data-processing activities. It lies in expanding the set of usable capabilities available to the professional and enabling those capabilities to be dynamically selected, combined, evaluated, and reallocated according to the evolving requirements of data systems and their operational environments. Data Engineers who understand both the enduring value of professional human skills and the rapidly developing possibilities of agentic skills can use this changing division of work to improve development speed, information coverage, data quality, reliability, consistency, maintainability, adaptability, and engineering effectiveness without reducing the occupation to either traditional human execution or fully automated AI performance.