代理式資料工程
AWS Open-Sources ADOP: A Multi-Agent Data Pipeline Generator That Runs Only Deterministic Artifacts in Production
ADOP uses about 15 agents to generate ETL, quality rules, a semantic layer, DAGs, and infrastructure configuration from natural-language specifications. It deliberately confines models to the development stage; production environments deploy only code that has passed human review and CI/CD validation.
AWS has released a reference implementation of the Agentic Data Operations Platform (ADOP), which aims to delegate the work of onboarding new data sources across Bronze, Silver, and Gold layers to specialized agents. After users describe S3, Kafka, Kinesis, or JDBC sources, update frequency, quality thresholds, and governance requirements, an orchestrator agent breaks the work into tasks covering metadata, ontology, data quality, transformation, orchestration, and DevOps. It then generates PySpark, SQL, Airflow DAGs, Step Functions, Terraform, CloudFormation, tests, and OWL/R2RML semantic artifacts. The repository is available under the MIT-0 license.
A key design principle is that “agents stay in development; artifacts move into production.” Models handle reasoning and file generation, but CI/CD deploys version-controlled, testable, deterministic code along with IAM and Cedar policies. Production data pipelines do not need to call a model on every run. This is easier to audit than allowing agents to operate directly on production data, and it prevents inference costs and model changes from directly altering existing workloads. A dynamic workflow can launch about 15 agents in parallel, with AWS estimating that specification-driven builds take 10 to 20 minutes. Sequential mode takes about 30 minutes, while parallel mode may increase token costs by three to five times.
From a security perspective, sub-agents generate files by default and are not granted access to MCP, CLI, or AWS tools. The main workflow determines tool access based on server health, intent routing, and enforced invariants. The project also recommends providing models only with schemas, column statistics, and small isolated samples, while resolving credentials at deployment time through AWS Secrets Manager or an external vault. Every tool selection, result, and cost can be exported to Amazon CloudWatch or OpenTelemetry.
However, this remains a sample architecture rather than a managed service or a product validated by a third party. Claims of reducing work from weeks to hours are primarily directional estimates from AWS and the project authors. Some README wording about “built-in GDPR, HIPAA, and PCI DSS compliance” is also no substitute for legal or security review. Generated masking rules, retention periods, and access policies may appear reasonable while still being semantically incorrect. Engineering teams should first use non-critical data to measure first-pass artifact acceptance rates, regression-test coverage, and the amount of manual correction required before deciding whether an agentic approach truly reduces total cost.