AI 改变的是数据接口，而非查询引擎

AI 代理如今可以生成分析代码、读取对象存储中的文件，并将运营数据转化为结构化报告。这是否意味着数据湖、湖仓和 SQL 引擎都已过时？我们在自动分析一场大型灾难恢复演练时遇到了这个问题。我们的首个可运行设计将每三十秒采集的服务指标以 Parquet 格式存储在兼容 S3 的对象存储中，使用 DuckDB 进行确定性聚合，并让 AI 代理对故障切换结果进行分类并解释相关证据。这个原型不依赖共享湖仓也能运行，但这并不是因为查询引擎消失了，而是因为引擎变成了代理工作流中的一个嵌入式组件。本文将介绍这一架构，阐述确定性计算与概率推理之间一种实用的分工方式，并提出四项指标，帮助判断何时嵌入式引擎已足够，何时应采用托管湖仓。AI agents can now generate analytical code, read files from object storage, and turn operational data into structured reports. Does that make data lakes, lakehouses, and SQL engines obsolete? We encountered this question while automating the analysis of a large disaster-recovery exercise. Our first working design stored thirty-second service metrics as Parquet in S3-compatible object storage, used DuckDB for deterministic aggregation, and asked an AI agent to classify failover outcomes and explain the evidence. The prototype worked without a shared lakehouse, but not because the query engine disappeared. Instead, the engine became an embedded component inside the agent workflow. This article explains the architecture, introduces a practical separation between deterministic computation and probabilistic reasoning, and presents four signals for deciding when an embedded engine is sufficient and when a managed lakehouse is justified.

原型成功运行后出现的问题The Question That Appeared After the Prototype Worked

这个问题并非来自白板演练，而是在一个真实工作流成功生成首份报告后出现的。当时我们正在分析一次灾难恢复（DR）孤岛演练。演练期间，我们有意隔离了一个区域，流量则应切换到仍正常运行的区域。对于每项服务，我们都需要回答三个问题：The question did not come from a whiteboard exercise. It appeared after a real workflow successfully produced its first report. We were analyzing a disaster-recovery (DR) islanding exercise. During the exercise, one region was intentionally isolated, and traffic was expected to fail over to surviving regions. For every service, we needed to answer three questions:

• 存留下来的区域是否吸收了从隔离区域流出的流量？
  Did the surviving region absorb the traffic that left the isolated region?
• 吸收这些流量后，服务是否仍然可用？
  After absorbing the traffic, did the service remain available?
• 延迟是否仍处于可接受范围内，还是额外负载导致了持续性性能下降？
  Did latency remain within an acceptable range, or did the additional load create sustained degradation?

我们以 30 秒的分辨率导出了服务指标，将其写入与 S3 兼容的对象存储中的 Parquet 文件，并让 AI 代理生成结构化分析报告。整个工作流端到端运行后，一个颇具吸引力的想法浮现出来：We exported service metrics at thirty-second resolution, wrote them as Parquet files to S3-compatible object storage, and let an AI agent generate a structured analysis report. Once the workflow ran end to end, an attractive idea emerged:

如果智能体可以从对象存储中读取文件并生成答案，为什么还需要表、目录、湖仓或 SQL？

If the agent can read files from object storage and produce the answer, why do we need tables, a catalog, a lakehouse, or SQL at all?

这个问题听起来合情合理，因为面向用户的体验已经发生了变化。工程师不再需要打开仪表板或编写查询，而是让智能体分析演练。智能体会识别相关数据，根据需要生成 DuckDB SQL 或 Python，通过确定性工具执行分析，并解释结果。例如，要验证故障切换期间流量是否从区域 A 转移到了区域 B，智能体可以针对一组预定义的表和时间范围生成只读 DuckDB 查询。执行前，系统会根据可访问数据集和操作的允许列表验证该查询，然后在沙箱化的分析运行时中执行。运行时会返回结构化查询结果，再由确定性应用代码将其规范化为事实数据包，其中包含按区域统计的流量、时间戳和百分比变化等字段。一个简化的实现如下所示：The question sounds reasonable because the user-facing experience has changed. Instead of opening a dashboard or writing a query, an engineer asks the agent to analyze the exercise. The agent identifies the relevant data, generates DuckDB SQL or Python as needed, executes the analysis through deterministic tools, and explains what happened. For example, to verify whether traffic shifted from Region A to Region B during failover, the agent may generate a read-only DuckDB query over a predefined set of tables and time ranges. Before execution, the query is validated against an allowlist of accessible datasets and operations, then executed in a sandboxed analysis runtime. The runtime returns structured query results, which deterministic application code normalizes into a fact packet containing fields such as traffic volume by region, timestamps, and percentage changes. A simplified implementation looks like this query:

generated_sql = """
SELECT
    region,
    SUM(request_count) AS requests
FROM traffic_metrics
WHERE ts BETWEEN :start_time AND :end_time
GROUP BY region
"""

validate_sql(
    generated_sql,
    allowed_tables={"traffic_metrics"},
    allowed_operations={"SELECT"},
)

rows = sandbox.execute(
    generated_sql,
    params={
        "start_time": exercise_start,
        "end_time": exercise_end,
    },
)

fact_packet = {
    "metric": "request_count",
    "window": [exercise_start, exercise_end],
    "by_region": {
        row["region"]: row["requests"]
        for row in rows
    },
}

在这个简化示例中，智能体只生成分析查询。在执行之前，确定性的验证层会拒绝访问不在允许列表中的表，或执行非只读操作。随后，查询会在沙箱运行时中运行，应用程序代码再将返回的行转换为结构化事实包。模型接收的是这个事实包（例如区域 A 和区域 B 的请求数），而不是自行编造或直接估算这些数值。In this simplified example, the agent generates only the analytical query. A deterministic validation layer rejects access to non-allowlisted tables or non-read-only operations before execution. The query then runs in a sandboxed runtime and the returned rows are converted by application code into a structured fact packet. The model receives this packet (for example, request counts for Regions A and B) rather than inventing or directly estimating those values.

为什么 DR 分析不只是扫描指标Why DR Analysis Is More Than Scanning Metrics

DR 分析的难点不在于获取时间序列，而在于判断这段时间序列意味着什么。假设区域 A 已隔离，而区域 B 仍在运行。一次正常的故障切换应当呈现出一段可辨识的序列：The difficult part of DR analysis is not retrieving a time series. It is deciding what the time series means. Suppose Region A is isolated and Region B remains active. A healthy failover should produce a recognizable sequence:

• 区域 A 的 QPS 下降。
  QPS decreases in Region A.
• 区域 B 的 QPS 相应增加。
  QPS increases in Region B by a corresponding amount.
• B 区域的成功率仍高于服务目标。
  The success rate in Region B remains above the service objective.
• 尾延迟可能会短暂上升，但不应持续处于高位。
  Tail latency may rise briefly, but it should not remain elevated.

每项指标都需要结合上下文解读。QPS 上升并不意味着故障切换一定成功；其增幅可能不足以弥补隔离区域流失的流量。如果成功率指标包含符合预期的业务拒绝请求，那么成功率降低也不一定意味着灾难恢复失败。如果系统能迅速恢复稳定状态，那么切换边界处的延迟激增可能并无大碍。因此，我们采用了两个基线和一个稳态窗口：Each statement requires context. A QPS increase is not automatically a successful failover. It may be too small to account for the traffic lost in the isolated region. A lower success rate is not automatically a DR failure if the metric includes expected business rejections. A latency spike at the cutover boundary may be harmless if the system quickly returns to a steady state. We therefore used two baselines and a steady-state window:

• 历史基线。前一天的同一时间窗口，可捕捉正常的时段行为。
  Historical baseline. the same time window on the previous day, which captures normal time-of-day behavior.
• 即时基线，即演练前一小时记录的最新服务状态。
  Immediate baseline, the hour before the exercise, which captures the most recent service state.
• 稳态窗口，即切换后的一段时间，排除短暂的过渡期，重点关注持续行为。
  Steady-state window, a post-cutover interval that excludes the short transition period and focuses on sustained behavior.

分析收集了四类核心信号：The analysis collected four core signal families:

• 每秒查询数（QPS）
  qps
• err_qps
  err_qps
• success_rate
  success_rate
• latency_pNN, including tail latency such as p99

对于每项服务，确定性代码都会计算一些事实，例如流量吸收比例、SLO 违规情况、延迟变化以及持续性能下降的时长。随后，智能体会结合服务本身、演练配置和现有证据来解读这些事实。这一区分至关重要。难点往往在于判断，但只有当判断所依据的事实是确定的，这种判断才值得信赖。For each service, deterministic code calculated facts such as traffic absorption ratio, SLO violations, latency change, and the duration of sustained degradation. The agent then interpreted those facts in the context of the service, the exercise configuration, and the available evidence. This distinction is fundamental. The hard part is often the judgment, but the judgment is only trustworthy when the facts underneath it are deterministic.

架构：对象存储、嵌入式引擎和智能体Architecture: Object Storage, an Embedded Engine, and an Agent

该工作流包含三个逻辑层：数据准备、确定性计算和智能体驱动的解读，并将可复现性作为贯穿各层的要求。数据准备层采用可复用的 DuckDB 构建逻辑，该逻辑只生成一次，随后重复使用，将原始指标转换为对象存储中的自描述资产。对于每个分析请求，智能体都会选择相关输入，生成受约束的 SQL 或 Python 代码，并协调 DuckDB/Python 执行确定性计算。随后，智能体会解读生成的结构化事实包，产出最终报告。整个工作流都会保留输入数据、配置、可复用的构建逻辑、确定性事实和智能体执行元数据，以便日后重建、重放和评估分析过程。The workflow used three logical layers: data preparation, deterministic computation, and agent-driven interpretation, with reproducibility treated as a cross-cutting requirement. The data-preparation layer used reusable DuckDB build logic, which was generated once and then reused, to transform raw metrics into self-describing assets in object storage. For each analysis request, the agent selected the relevant inputs, generated constrained SQL or Python, and orchestrated deterministic execution in DuckDB/Python. The resulting structured fact packets were then interpreted by the agent to produce the final report. Throughout the workflow, the input data, configuration, reusable build logic, deterministic facts, and agent execution metadata were retained so that the analysis could later be reconstructed, replayed, and evaluated.

图 1。AI 代理启动并统筹分析，确定性工具计算事实，代理则解读由此生成的事实数据包。来源：作者自行绘制，并借助 ChatGPT 完成。Figure 1. The AI agent initiates and orchestrates the analysis, deterministic tools compute the facts, and the agent interprets the resulting fact packet. Source: Authors’ own illustration, created with assistance from ChatGPT.

存储与自我描述Storage and Self-Description

物理数据存储在兼容 S3 的对象存储中。每个练习都有一个自描述目录：The physical data lived in S3-compatible object storage. Each exercise had a self-describing directory:

s3://<bucket>/<root-prefix>/
  index.json
  <exercise-id>/
    manifest.json
    exercise_config.json
    metrics-*.parquet
    service_dimension.parquet

这些文件用途各不相同：The files served different purposes:

• index.json 列出了可用的练习。
  index.json listed available exercises.
• manifest.json 记录了架构、行数、文件大小和校验和。
  manifest.json recorded the schema, row counts, file sizes, and checksums.
• exercise_config.json 记录了时间窗口、区域、阈值和演练标识。
  exercise_config.json captured time windows, regions, thresholds, and exercise identity.
• metrics-*.parquet 中包含三十秒指标事实。
  metrics-*.parquet contained the thirty-second metric facts.
• service_dimension.parquet 包含服务元数据，例如所有权和服务层级。
  service_dimension.parquet contained service metadata such as ownership and service tier.

清单之所以重要，是因为一个文件夹中的文件还不能算是可信的数据资产。使用者需要知道其中包含哪些内容、使用了什么架构、写入是否已完成，以及之后能否对这些文件进行验证。The manifest was important because a folder of files is not yet a trustworthy data asset. Consumers need to know what is present, which schema was used, whether the write completed, and whether the files can be validated later.

使用 DuckDB 进行确定性处理Deterministic Processing with DuckDB

DuckDB 在存储边界的两侧都发挥了作用。在数据摄取阶段，它读取原始指标记录，对其进行排序，写入压缩的 Parquet 文件，并为清单生成验证统计信息。在分析阶段，它直接读取 Parquet 文件，筛选相关时间窗口，关联服务元数据，并计算智能体使用的事实数据包。这种架构得益于 DuckDB 的以下几项特性：DuckDB was used on both sides of the storage boundary. During ingestion, it read the raw metric records, sorted them, wrote compressed Parquet files, and generated validation statistics for the manifest. During analysis, it read the Parquet files directly, filtered the relevant windows, joined service metadata, and computed the fact packet used by the agent. This architecture benefited from several properties of DuckDB:

• 它在进程内运行，因此可以随代理工作流一起部署，而无需单独的数据库服务。
  It runs in-process, so it can travel with the agent workflow instead of requiring a separate database service.
• 它可以直接读取 Parquet，并将过滤条件和列选择下推到 Parquet 扫描中。
  It reads Parquet directly and can push filters and column selection into Parquet scans.
• 它为聚合和联接提供 SQL 语义。
  It provides SQL semantics for aggregation and joins.
• 它可以将较大的操作转移到本地存储，而不必让整个工作集始终驻留在内存中。
  It can spill larger operations to local storage instead of requiring the entire working set to remain in memory.

重点并不是 DuckDB 在所有情况下都优于 Spark、Trino 或托管式湖仓。重点在于，这项工作负载目前还不需要共享的分布式引擎。The point was not that DuckDB is universally better than Spark, Trino, or a managed lakehouse. The point was that the workload did not yet need a shared distributed engine.

借助 AI 智能体进行概率解读Probabilistic Interpretation with the AI Agent

智能体并没有接收数十万行原始指标数据，然后尝试自行统计错误数或计算平均延迟。相反，它为每项服务接收了一个精简且确定性的证据包。简化后的数据包如下例所示：The agent did not receive hundreds of thousands of raw metric rows and attempted to count errors or average latency itself. Instead, it received a compact, deterministic evidence packet for each service. A simplified packet looked like this example:

{
  "service_id": "service-a",
  "exercise_window": "post-cutover-steady-state",
  "expected_failover_qps": 1200,
  "observed_qps_increase": 910,
  "traffic_absorption_ratio": 0.758,
  "success_rate_baseline": 0.9992,
  "success_rate_during_exercise": 0.9941,
  "service_slo": 0.9990,
  "p99_latency_baseline_ms": 84,
  "p99_latency_during_exercise_ms": 191,
  "sustained_degradation_minutes": 18,
  "data_quality_flags": []
}

智能体使用该数据包对结果进行分类、说明支持性证据，并提出下一步调查措施。相应的输出可能是：The agent used the packet to classify the outcome, explain the supporting evidence, and propose the next investigation step. A corresponding output might be:

{
  "classification": "partial_failover",
  "confidence": "high",
  "evidence": [
    "The surviving region absorbed approximately 76% of the expected traffic.",
    "Availability remained below the service SLO during the steady-state window.",
    "The p99 latency increase was sustained for 18 minutes."
  ],
  "recommended_action": "Review routing capacity and downstream dependency limits."
}

具体数字来自引擎。模型的职责是确定如何解读和传达这些数字。The exact numbers came from the engine. The model's responsibility was to determine how those numbers should be interpreted and communicated.

简单的基于阈值的规则并不总是足够。例如，切换后不久，某项服务的 p99 延迟可能会暂时超过阈值，但此时流量已经正确切换，且可用性仍符合服务目标。若规则将任何阈值超标都视为故障，就会把该服务判定为性能下降。代理则可以将切换窗口与稳态窗口进行比较，识别出延迟在几分钟内恢复至基线，并将该事件判定为伴有短暂性能下降的成功故障转移，而非持续性的灾难恢复故障。A simple threshold-based rule was not always sufficient. For example, a service might temporarily exceed its p99 latency threshold immediately after cutover while traffic had already shifted correctly and availability remained within the service objective. A rule that treated any threshold violation as a failure would classify the service as degraded. The agent could instead compare the transition window with the steady-state window, recognize that latency returned to baseline within a few minutes, and classify the event as a successful failover with transient degradation rather than a sustained DR failure.

更重要的是，有些案例包含相互矛盾的证据，无法通过单一阈值或时间规则加以判定。某项服务可能承接了预期流量并恢复了自身的延迟，但由于下游依赖项接近饱和，仍然面临风险。反过来，Tier-2 异步服务可能出现暂时的积压增加，却没有违反其可用性目标；而 Tier-0 同步依赖项出现相同情况，则需要升级处理。在这些案例中，确定性层仍会生成底层事实（即流量转移、延迟、成功率、积压、依赖项饱和度、服务层级和 SLO 状态），但智能体会结合服务上下文来判断哪些证据最重要，以及应如何对事件进行分类。More importantly, some cases contained conflicting evidence that could not be resolved by a single threshold or temporal rule. A service might absorb the expected traffic and recover its own latency, yet still remain at risk because a downstream dependency was operating near saturation. Conversely, a Tier-2 asynchronous service could show a temporary backlog increase without violating its availability objective, while the same pattern in a Tier-0 synchronous dependency would warrant escalation. In these cases, the deterministic layer still produced the underlying facts (i.e.,traffic shift, latency, success rate, backlog, dependency saturation, service tier, and SLO status), but the agent combined those facts with service context to determine which evidence was most important and how the event should be classified.

因此，智能体创造价值并非通过取代确定性计算，而是通过处理依赖上下文且有时相互冲突的证据；否则，就需要不断扩大的服务专属规则集来处理这些情况。The agent therefore added value not by replacing deterministic computation, but by resolving context-dependent and sometimes conflicting evidence that would otherwise require an increasingly large set of service-specific rules.

确定性之分The Determinism Split

这个项目中最值得借鉴的经验，是我所说的“确定性拆分”：将精确计算与上下文推理分开。The most reusable lesson from the project is what I call the determinism split: separate exact computation from contextual reasoning.

Give to the engine	Give to the AI agent
Filtering, aggregation, joins, sorting, and window calculations	Selecting relevant evidence and explaining why it matters
Exact and reproducible numerical results	Classifying failover behavior and suggesting investigation paths
Operations whose cost grows with data volume	Work that depends on context, trade-offs, and communication
Data quality checks and validation flags	Abstaining when the available evidence is insufficient

表 1. 精确计算与情境推理之间的实用区分。Table 1. A practical separation between exact computation and contextual reasoning.

这种划分避免了一种常见的反模式：将 LLM 当作查询引擎使用。语言模型不应读取数十万行数据，再计算错误数量、百分位数或平均值。这种做法速度慢、消耗大量 token，而且难以复现。更重要的是，它把确定性任务交给了概率系统。更安全的模式是：This division avoids a common anti-pattern, using an LLM as a query engine. A language model should not read hundreds of thousands of rows and calculate an error count, percentile, or average. That approach is slow, expensive in tokens, and difficult to reproduce. More importantly, it assigns deterministic work to a probabilistic system. The safer pattern is:

• 使用确定性引擎生成一小组权威事实。
  Use a deterministic engine to produce a small set of authoritative facts.
• 允许智能体生成用于推导这些事实的受限 SQL 或 Python 代码。
  Allow the agent to generate constrained SQL or Python needed to derive those facts.
• 通过确定性工具执行该代码，并将生成的事实数据包连同必要的上下文一起提供给模型。
  Execute that code through deterministic tools and give the resulting fact packet to the model with the necessary context.
• 要求模型使用事实数据包中的字段解释结论，同时保留输入、生成的代码、执行元数据和输出，以便重放和审计。
  Require the model to explain conclusions using fields from the fact packet, while retaining inputs, generated code, execution metadata, and outputs for replay and audit.

智能体可以决定调查哪个问题，动态生成受约束的 DuckDB SQL 或 Python 代码，并协调其执行。生成的代码可以通过概率方式提出，但最终的数值事实必须来自确定性执行，而不能由模型本身生成。The agent can decide which question to investigate, dynamically generate constrained DuckDB SQL or Python, and orchestrate its execution. The generated code may be proposed probabilistically, but the resulting numerical facts must come from deterministic execution rather than from the model itself.

工作负载揭示的架构信息What the Workload Told Us About the Architecture

该练习数据集包含：The exercise dataset contained:

• 30 秒指标粒度
  Thirty-second metric resolution
• 1,127,748 条事实数据行
  1,127,748 fact rows
• 1,523 项服务
  1,523 services
• 约 3.8 兆字节的压缩 Parquet 事实数据
  Approximately 3.8 megabytes of compressed Parquet fact data

使用并发工作线程对所有服务进行上游数据采集，大约耗时一小时。生成的分析数据集足够精简，嵌入式引擎可以在靠近代理的位置处理它，而无需引入共享数据平台。这一观察很重要，因为架构决策应该依据实际工作负载，而不是“大数据”这个标签。在设计评审中，超过一百万行听起来可能规模不小，但仅凭行数不足以成为部署分布式湖仓技术栈的理由。列数、压缩率、查询形态、并发量、访问模式和数据移动都很重要。The upstream collection run across all services took roughly an hour with concurrent workers. The resulting analytical dataset was compact enough that an embedded engine could process it close to the agent without introducing a shared data platform. That observation matters because architecture decisions should follow the actual workload rather than the label big data. More than a million rows can sound large in a design review, but row count alone is not a sufficient reason to deploy a distributed lakehouse stack. Column count, compression, query shape, concurrency, access patterns, and data movement all matter.

一个具有代表性的分析演练A Representative Analysis Walkthrough

以下是一个基于练习中所用工作流的典型示例；服务标识符和数值均已匿名化。The following is a representative example based on the workflow used in the exercise; service identifiers and values have been anonymized.

对于某项服务，确定性执行层生成了一份事实数据包，显示存活区域承接了约 76% 的预期故障转移流量；在稳态时间窗口内，可用性低于该服务的 SLO，且 p99 延迟持续偏高 18 分钟。与此同时，该服务仍在持续处理流量，并未发生完全中断。For one service, the deterministic execution layer produced a fact packet showing that the surviving region absorbed approximately seventy-six percent of the expected failover traffic, availability fell below the service SLO during the steady-state window, and p99 latency remained elevated for eighteen minutes. At the same time, the service continued to serve traffic and did not experience a complete outage.

该代理将这些事实解读为部分故障切换，而非简单的成功或失败。它在解释中提到了流量吸收比例、持续违反 SLO 的情况以及延迟恶化，并建议审查路由容量和下游依赖项的限制。The agent interpreted these facts as a partial failover rather than a binary success or failure. Its explanation cited the traffic absorption ratio, the sustained SLO violation, and the latency degradation, and recommended reviewing routing capacity and downstream dependency limits.

随后，研究人员依据底层的确定性事实和演练时间线对结论进行了验证。审查人员可以将智能体解释中的每一项陈述追溯到事实数据包中的字段，并使用保留的输入数据和配置重新运行相同的 DuckDB/Python 分析。结论确认后，后续行动是调查路由容量或下游瓶颈是否导致存续区域无法承接全部预期负载。The conclusion was then validated against the underlying deterministic facts and the exercise timeline. Reviewers could trace each statement in the agent’s explanation back to fields in the fact packet and rerun the same DuckDB/Python analysis against the retained input data and configuration. Once the conclusion was confirmed, the resulting action was to investigate whether routing capacity or a downstream bottleneck had prevented the surviving region from absorbing the full expected load.

这一步验证至关重要：智能体提出了解释和下一步行动，但数值证据仍然可复现且能够独立核验。This validation step was important: the agent proposed the interpretation and next action, but the numerical evidence remained reproducible and independently verifiable.

使工作流可复现Making the Workflow Reproducible

只有在我们将可复现性视为首要要求后，原型才真正成为工程系统。对于每次练习，我们都保留了一个回放包，其中包含：The prototype only became an engineering system after we treated reproducibility as a first-class requirement. For every exercise, we retained a replay bundle containing:

• Parquet 输入文件
  The Parquet inputs
• 清单和校验和
  The manifest and checksums
• 练习配置
  The exercise configuration
• 生成的 SQL 或 Python 代码
  The generated SQL or Python code
• 执行元数据
  The execution metadata
• 确定性派生事实
  The deterministic derived facts
• 模型输出
  The model output
• 智能体版本和提示词
  The agent version and prompt
• 模型标识符
  The model identifier

这个回放包使我们能够离线重新运行分析、调查有争议的结论，或基于相同证据比较两个版本的智能体的行为。我们还明确编码了数据语义。一个微妙的例子是缺失值的含义。在某些情况下，缺失的 QPS 或错误计数数据可能表示没有活动。而缺失的成功率或延迟数据，通常意味着没有可供测量的流量，而不是成功率为零或延迟为零毫秒。This replay bundle made it possible to rerun an analysis offline, investigate a disputed conclusion, or compare the behavior of two agent versions against the same evidence. We also encoded data semantics explicitly. One subtle example was the meaning of a missing value. Missing QPS or error-count data could represent zero activity in some contexts. Missing success-rate or latency data often meant no traffic was available to measure, not zero percent success or zero milliseconds of latency.

将每个缺失值都填为零会导致误报失败和误报恢复。必须通过架构和数据质量标记保留这种区别，以便引擎和代理以一致的方式解读数据集。Filling every missing value with zero would have created false failures and false recoveries. The schema and data-quality flags needed to preserve that distinction so the engine and the agent interpreted the dataset consistently.

当轻量级架构不再够用时When the Lightweight Architecture Stops Being Enough

Agent 加 DuckDB 并非适用于所有分析系统的最终架构。只有当工作负载符合适当的特征时，它才能发挥作用。有四种信号表明，这种轻量级架构开始面临压力。这并不意味着必须立即采用完整的 lakehouse 技术栈。每种信号都指向一种可以逐步引入的不同能力，包括用于共享发现与治理的目录、用于安全并发写入和演进的开放表格式，或是在计算必须更靠近大型共享数据集时使用的分布式查询引擎。Agent plus DuckDB is not the final architecture for every analytical system. It works when the workload has the right shape. Four signals indicate where the lightweight architecture begins to encounter pressure. They do not require adopting a complete lakehouse stack at once. Each signal points to a different capability that can be introduced incrementally, including a catalog for shared discovery and governance, an open table format for safe concurrent writes and evolution, or a distributed query engine when computation must move closer to large shared datasets.

独立消费者出现了Independent Consumers Appear

如果仪表板、定时作业、笔记本或其他团队需要直接访问，数据就需要有稳定的契约。这些使用方不应被迫调用原始代理，也不应需要通过逆向分析来摸清存储桶的布局。共享目录可以让他们按名称查找表、了解其架构、设置权限，并通过标准工具查询数据。If dashboards, scheduled jobs, notebooks, or other teams need direct access, the data needs a stable contract. Those consumers should not have to call the original agent or reverse-engineer a bucket layout. A shared catalog allows them to discover a table by name, understand its schema, apply permissions, and query it through standard tools.

数据变得过于庞大，无法移至代理环境中The Data Becomes Too Large to Move to the Agent Environment

当数据达到 TB 级规模时，将文件拉取到本地或开发环境进行分析会变得低效。查询应在数据存储的位置执行（即近数据处理），并采用分区裁剪、谓词下推和分布式处理。只有查询结果应返回给代理。At terabyte scale, pulling files toward a local or development environment for analysis becomes inefficient. The query should execute where the data is stored (i.e., near-data processing), with partition pruning, predicate pushdown, and distributed processing. Only the result should move back to the agent.

工作流必须与大型共享数据集关联起来The Workflow Must Join with Large Shared Datasets

当两侧数据都足够小时，DuckDB 可以高效地关联本地文件。若另一侧数据是组织范围内的权威数据集，已在共享目录中注册，且规模过大或更新过于频繁，无法安全导出，那么这种方式就会遇到瓶颈。此时，在同一环境中注册新数据，并通过共享引擎执行关联，会更加可靠。DuckDB can join local files efficiently when both sides are small enough. The boundary appears when the other side is an organization-wide authoritative dataset that is already registered in a shared catalog and is too large or too frequently updated to export safely. At that point, it is more reliable to register the new data in the same environment and execute the join through the shared engine.

需要并发写入或表级演进Concurrent Writes or Table-Level Evolution Are Required

单写入者约定对于早期工作流来说简单而有效。但当多个数据生产者更新数据集、架构独立演进，或读者需要一致性快照时，这种方式就难以为继。Apache Iceberg 这样的开放表格式在 Parquet 文件之上增加了表元数据、快照、架构演进和事务性提交语义。随后，托管式湖仓可以结合三种能力：A single-writer convention is simple and effective for an early workflow. It breaks down when multiple producers update the dataset, schemas evolve independently, or readers require consistent snapshots. An open table format such as Apache Iceberg adds table metadata, snapshots, schema evolution, and transactional commit semantics on top of the Parquet files. A managed lakehouse can then combine three capabilities:

• 用于发现、所有权、权限、架构和分区的目录
  A catalog for discovery, ownership, permissions, schema, and partitions
• 一种用于快照、演进和安全写入的开放式表格式
  An open table format for snapshots, evolution, and safe writes
• 用于分布式扫描和连接的共享查询引擎
  A shared query engine for distributed scans and joins

这些能力很有价值，但也会带来集成、治理、计算和运维成本。应根据实际的数据共享、规模和交易需求来采用，而不是因为默认认为每个 AI 项目从一开始就需要湖仓架构。These capabilities are valuable, but they introduce integration, governance, compute, and operational cost. They should be adopted in response to real sharing, scale, and transaction requirements, not because every AI project is assumed to need a lakehouse on day one.

实用决策表A Practical Decision Table

Workload signal	Embedded engine close to the agent	Managed lakehouse
Primary consumer is one agent workflow	Strong fit	Often unnecessary
Moderate, compressed analytical dataset	Strong fit	May be over-engineered
Ad hoc or exercise-based analysis	Strong fit	Useful only if sharing is required
Many independent consumers	Weak fit	Strong fit
Terabyte-scale scans	Weak fit	Strong fit
Joins with large shared authoritative tables	Limited	Strong fit
Concurrent writers and snapshot requirements	Weak fit	Strong fit
Organization-wide catalog and governance	Limited	Strong fit

表 2. 根据访问模式、规模和写入语义选择平台。Table 2. Choose the platform based on access patterns, scale, and write semantics.

这个决定并非是在 AI 和 SQL 之间做选择，也不是在文件和表格之间做选择。真正需要做出的决定是：The decision is not AI or SQL nor is it files or tables. The real decision is:

嵌入式、由智能体驱动的分析引擎是否足够，还是说该工作负载需要一个托管式共享数据平台？

Is an embedded, agent-driven analytical engine sufficient, or does the workload require a managed, shared data platform?

AI 智能体让数据分析更容易上手。它们可以将运营问题转化为分析步骤，筛选相关证据，对行为进行分类，并撰写工程师可据此采取行动的报告。但它们并不能免除精确执行的必要性。AI agents make data analysis more accessible. They can translate an operational question into analytical steps, select relevant evidence, classify behavior, and write a report that an engineer can act on. They do not eliminate the need for precise execution.

在我们的 DR 分析工作流中，Parquet 和 DuckDB 承担了机械性工作：存储、筛选、聚合、验证和联接。智能体则负责上下文相关的工作：判断如何综合解读流量吸收、可用性和延迟，并传达分析结果。我们有意保持架构轻量，因为实际工作负载尚不需要共享湖仓。In our DR analysis workflow, Parquet and DuckDB performed the mechanical work: storage, filtering, aggregation, validation, and joins. The agent performed the contextual work by deciding how to interpret traffic absorption, availability, and latency together, and communicating the result. The architecture remained intentionally lightweight because the real workload did not yet require a shared lakehouse.

运行这套工作流后，我们对分析界面应该设在哪里有了新的认识。工程师不再需要在每次调查时都直接与查询引擎交互，但查询引擎仍然是生成可信事实的关键。智能体则成为负责规划分析、生成所需 SQL 或 Python 代码并解读证据的那一层。What changed after operating the workflow was our view of where the analytical interface should live. Engineers no longer needed to interact directly with the query engine for every investigation, but the engine remained essential for producing trustworthy facts. The agent became the layer that planned the analysis, generated the required SQL or Python, and interpreted the evidence.

因此，主要的架构权衡并非 AI 与 SQL 之争，也不是文件与表格之争，而是针对当前工作负载，智能体背后需要配备多少分析基础设施。对于目标明确的工作流，小型嵌入式引擎或许就已足够；只有当数据共享、规模、写入并发或治理带来切实需求时，才应引入共享目录、开放表格式和分布式引擎。引擎依旧存在，改变的是交互方式。The main architectural trade-off is therefore not AI versus SQL, or files versus tables. It is how much analytical infrastructure needs to sit behind the agent for the workload at hand. A small embedded engine may be sufficient for a focused workflow; shared catalogs, open table formats, and distributed engines should be introduced only as sharing, scale, write concurrency, or governance create a concrete need. The engine remains; the interface changes.