Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions src/cli/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,20 @@

## [Unreleased]

## [0.3.0] - 2026-08-24

### Added
- `Specification` envelope 新增可选 `manifest` 输入契约(raw / map / config_tables / review_decisions)及确定性 YAML 校验。
- `catalog` 新增 `VolumeArtifactType` 与 `catalog set-status`,用于区分预审核产物、审核决策文件、最终交付产物并推进 volume 状态。
- 新增 `runtime/` 模块(`Runtime` trait + 注册表),支持 Python、R、Stata、Matlab、Bash 和 builtin 运行时。
- `implement` 增加 R、Stata、Matlab codegen;`process` 通过运行时注册表执行脚本。
- 全局 `--json` 命令错误输出与 `CliError` 错误码结构化;`spec validate` 成功结果支持结构化输出。

### Changed
- `blueprint list/show` 与 `pipeline list/show` 改为文件直读优先,cue 降为可选增强;`doctor` 中 cue 检查同步降为 warning。
- v0.2.2 计划文档收口:baidu/sftp 覆盖率补测后移至 v0.3.0,存储凭证常量表去重后移至未分配低优先事项
- 新增 `runtime/` 模块(`Runtime` trait + 注册表):codegen(implement 用)+ execute(process 用),`from_name`/`from_ext` 注册表驱动;Python codegen 逻辑从 `stage/implement.rs` 分离,`process` 执行分发改注册表(`.py`→python / `.sh`→bash)。旧 `stage::implement::{implement_step_prompt, implement_assemble_prompt, to_snake}` 保留为 deprecated 转发(随 v0.3 移除)
- `storage/` 统一概念命名:`StorageProvider` → `Storage`、`DropboxProvider` → `DropboxStorage`(等 6 平台)。旧名保留为 deprecated re-export(随 v0.3 移除)
- `doctor` 的运行时检查改由运行时注册表驱动
- `transfer` provider 改为枚举分发,`process` pipeline 改为结构化 Blueprint state 解析
- `storage/` 统一概念命名为 `Storage` 与具体 `*Storage` 类型,旧名保留为 deprecated re-export 供迁移使用

## [0.2.2] - 2026-08-02

Expand Down
14 changes: 7 additions & 7 deletions src/cli/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
| [TODO.md](TODO.md) | — | 按模块拆解的执行任务 |
| [CHANGELOG.md](CHANGELOG.md) | — | 版本变更记录(发布事实源) |
| [docs/index.md](docs/index.md) | main/lib/error/registry/util | 命令结构总览 + 文档映射表 + 横切基础(错误模型 + 注册表/工具机制) |
| [docs/transfer.md](docs/transfer.md) | transfer/storage | 传输服务与 StorageProvider |
| [docs/transfer.md](docs/transfer.md) | transfer/storage | 传输服务与 Storage |
| [docs/catalog.md](docs/catalog.md) | catalog | 数据格式(registry/jobs/delivery-links 字段级) |
| [docs/process.md](docs/process.md) | process | StepExecutor 编排 |
| [docs/llm.md](docs/llm.md) | clarify/design/implement/review | LLM 命令与 Handler 注入 |
Expand Down Expand Up @@ -118,7 +118,7 @@ qtcloud-devops plan audit --scope cli
**2. Code**:在 feature 分支上完成实现与文档改动

```bash
git switch -c codex/cli-v0.2.X-release
git switch -c codex/cli-v0.3.0-release
qtcloud-devops code audit src/cli
```

Expand All @@ -135,26 +135,26 @@ cargo test --locked
cargo clippy --locked -- -A warnings
```

**4. 更新发布记录**:`Cargo.toml` 版本号 + `CHANGELOG.md` 发布条目 + `ROADMAP.md`/`TODO.md` 勾选
**4. 更新发布记录**:`Cargo.toml` 版本号 + `CHANGELOG.md` 发布条目 + `ROADMAP.md`/`TODO.md` 收口

**5. 提交与审查**:release-prep 提交(`chore(cli): prepare v0.2.X release`),
**5. 提交与审查**:release-prep 提交(`chore(cli): prepare v0.3.0 release`),
feature 分支推远端 → Pull Request → review → 合并 `main`。release tag 必须指向 `main` 可达提交。
若变更已在 `main` 且 CI 通过(`push: [main]` 触发),可直接 `main` 发布。

**6. 发布预检**(合并后,从干净的 `main` checkout)

```bash
qtcloud-devops release status
qtcloud-devops release audit -v cli/v0.2.X --scope cli
qtcloud-devops release publish -v cli/v0.2.X --registry crates --dry-run
qtcloud-devops release audit -v cli/v0.3.0 --scope cli
qtcloud-devops release publish -v cli/v0.3.0 --registry crates --dry-run
```

dry-run 不得创建 tag、GitHub Release 或 crates.io 版本。

**7. 发布**(maintainer 确认后)

```bash
qtcloud-devops release publish -v cli/v0.2.X --registry crates -y
qtcloud-devops release publish -v cli/v0.3.0 --registry crates -y
```

命令创建并推送 `cli/v0.2.X` tag,随后 `release-cli.yml` GitHub Actions 完成:
Expand Down
2 changes: 1 addition & 1 deletion src/cli/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion src/cli/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "qtcloud-data-cli"
version = "0.2.2"
version = "0.3.0"
edition = "2024"
authors = ["QuantTide Inc. <opensource@quanttide.com>"]
description = "量潮数据云 CLI"
Expand Down
7 changes: 6 additions & 1 deletion src/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

当前 crates.io 发布版本:`qtcloud-data-cli` v0.2.1(GitHub Release `cli/v0.2.1`,含 Linux / Windows 二进制)。
v0.2.1 重点:统一错误处理模型(CliError)、store 模块收敛路径与 JSON 读写、cue 输出结构化解析、全链路 e2e,测试覆盖率提升至 83.7%。
当前开发版本为 v0.3.0,新增多语言 Runtime 注册表和命令错误结构化输出;正式发布状态以 `qtcloud-devops release status` 为准。

## 命令一览

Expand All @@ -12,7 +13,7 @@ v0.2.1 重点:统一错误处理模型(CliError)、store 模块收敛路
| `clarify` | 从客户聊天记录或上下文生成 DRD 数据需求文档 |
| `design` | 从 DRD 生成 Contract / Blueprint Specification(YAML + MD + HTML) |
| `spec` | 固化 Specification YAML envelope(wrap / validate) |
| `implement` | 从 Blueprint YAML 生成 Python 代码实现 |
| `implement` | 从 Blueprint YAML 生成 Python / R / Stata / Matlab 代码实现 |
| `review` | 审计 DRD 或 Specification 的完整性和一致性 |
| `version` | 查看和比较规格版本 |
| `transfer` | 数据传输(send / receive),支持 6 个平台 |
Expand Down Expand Up @@ -76,6 +77,8 @@ qtcloud-data design blueprint .quanttide/data/drd/context.md

# 从规格书生成 Python 实现
qtcloud-data implement .quanttide/data/spec/context-blueprint.yaml --lang python
# 也支持 R / Stata / Matlab
qtcloud-data implement .quanttide/data/spec/context-blueprint.yaml --lang r

# 将旧 Blueprint YAML 包装成稳定 Specification envelope
qtcloud-data spec wrap .quanttide/data/spec/context-blueprint.yaml
Expand All @@ -100,6 +103,8 @@ qtcloud-data process ABC "https://..." --blueprint csv-standardization

成功交付时,`process` 还会把最终产物登记到 `CATALOG_DIR/registry.json`,provider 为 `process`,source 为 `process:<job-id>`,status 为 `delivered`。

使用全局 `--json` 时,命令错误会输出包含稳定 `code` 和用户可读 `message` 的 JSON;`spec validate` 也支持结构化成功结果。

## DataOps 目录

默认目录根是 `.quanttide/data`,也可以用 `DATA_ROOT` 调整。单个目录仍可用对应环境变量覆盖。
Expand Down
15 changes: 5 additions & 10 deletions src/cli/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,14 @@

> **待排期(不阻塞 v0.2.2)**:CLI 发起 Provider run 的执行入口(`PROVIDER_URL` 配置 + run 请求参数校验),以及业务 e2e(raw + map.dta → review_master,依赖 Provider merge_review / export)——待 Provider ROADMAP [0.0.3] 的 merge_review/export 落地后启动。

## [0.3.0]
## [0.3.1]

> v0.3.0 已完成项已迁移至 [CHANGELOG.md](CHANGELOG.md)。

### Added

- [ ] `src/storage/` baidu/sftp 补测(需真实服务或本地模拟:CI 起 sshd 或本地模拟,0% → ≥50%)
- [ ] `Cargo.toml` 构建 Linux、Windows、macOS 二进制包
- [ ] `Cargo.toml` 自动上传各平台 Release 制品
- [ ] `src/lib.rs` 新增 runtimes 模块:RuntimeAdapter trait 与注册表(python / r / stata / matlab / bash / builtin)
- [ ] `src/process.rs` run_pipeline 改为注册表查表,替代扩展名 if-else 分发
- [ ] `src/implement.rs` implement 支持 --lang r / --lang stata
- [ ] `src/blueprint_core.rs` 新增 R / Stata codegen prompt 模板
- [ ] `src/doctor.rs` 检查表由 RuntimeAdapter 注册表驱动
- [ ] `src/main.rs` 全局 --json 结构化输出,供 Studio/CI 消费
- [ ] `Cargo.toml` / `.github/workflows/release-cli.yml` 构建并上传 macOS 二进制包
- [ ] 其余命令成功结果逐步迁移为结构化输出,供 Studio/CI 消费

### Changed

Expand Down
16 changes: 3 additions & 13 deletions src/cli/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,21 +26,11 @@
- [x] coverage:更新覆盖率基线(当前 83.7%,见 `CONTRIBUTING.md` 测试分层)
- [x] storage:凭证环境变量名集中为常量表迁移至 99. 后续(`DROPBOX_ACCESS_TOKEN` / `GOOGLE_DRIVE_ACCESS_TOKEN` 等魔法字符串去重,低优先,可选)

## [0.3.0]
## [0.3.1]

- [ ] coverage:`src/storage/` baidu/sftp 补测(需真实服务或本地模拟,0% → ≥50%,需 CI 起 sshd 或本地模拟)
- [ ] distribution:增加 macOS 二进制构建(`../../.github/workflows/release-cli.yml`)
- [ ] distribution:增加 macOS 二进制构建和 Release 上传(`../../.github/workflows/release-cli.yml`)
- [ ] distribution:增加发布后的 deploy、operate、monitor 记录(`../../.github/workflows/`、`docs/`)
- [ ] runtimes:`src/lib.rs` 注册 runtimes 模块(新增 src/runtimes/)
- [ ] runtimes:`src/process.rs` run_pipeline 改注册表查表,替代扩展名 if-else
- [ ] runtimes:`src/implement.rs` implement 支持 --lang r / --lang stata
- [ ] runtimes:`src/blueprint_core.rs` 新增 R / Stata codegen prompt 模板
- [ ] runtimes:`src/doctor.rs` 检查表由 RuntimeAdapter 注册表驱动
- [ ] structured-output:`src/main.rs` 全局 --json 结构化输出
- [ ] structured-output:`CliError` 携带结构化错误码(`--json` 前置,`src/error.rs`)
- [ ] structured-output:`run_command` 分发移入 lib(`Commands` 可单测,llm 参数注入,`src/main.rs` / `src/lib.rs`)
- [ ] structured-output:`src/transfer.rs` provider 枚举化,替代字符串匹配
- [ ] structured-output:`src/process.rs` pipeline 引用结构化(Blueprint states),替代逗号分隔字符串
- [ ] structured-output:其余命令成功结果逐步迁移为结构化输出(`src/`)

## [0.5.0]

Expand Down
14 changes: 10 additions & 4 deletions src/cli/docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
| 文档 | 对应模块 | 内容 |
|------|---------|------|
| [index.md](index.md)(本文件) | `main.rs`、`lib.rs`、`error.rs`、`registry.rs`、`util.rs` | 命令结构、文档映射、横切基础(错误模型 + 注册表/工具机制) |
| [transfer.md](transfer.md) | `transfer.rs`、`storage/` | 传输服务与 StorageProvider trait、添加新平台 |
| [transfer.md](transfer.md) | `transfer.rs`、`storage/` | 传输服务与 Storage trait、添加新平台 |
| [catalog.md](catalog.md) | `implementation/catalog.rs` | 数据格式(registry/jobs/delivery-links 字段级定义) |
| [process.md](process.md) | `stage/process.rs` | StepExecutor 编排(receive → pipeline → send) |
| [llm.md](llm.md) | `stage/clarify.rs`、`stage/design.rs`、`stage/implement.rs`、`review.rs` | LLM 命令与 Handler 注入模式 |
Expand All @@ -18,6 +18,7 @@
| [pipeline.md](pipeline.md) | `implementation/pipeline.rs` | 管道定义查看 |
| [doctor.md](doctor.md) | `doctor.rs` | 环境检查 |
| [version.md](version.md) | `spec/version.rs` | Specification 版本管理(spec version) |
| — | `runtime/` | Python / R / Stata / Matlab / Bash / builtin 运行时注册表 |

贡献与发布流程见 [CONTRIBUTING.md](../CONTRIBUTING.md)。

Expand All @@ -29,7 +30,7 @@ qtcloud-data
├── 生命周期命令(纵向流程,按数据流顺序)
│ ├── clarify 需求澄清:聊天记录 → DRD
│ ├── design 规格设计:DRD → Contract / Blueprint(contract / blueprint / formalize / preview)
│ ├── implement 代码实现:Blueprint → Python
│ ├── implement 代码实现:Blueprint → Python / R / Stata / Matlab
│ ├── process 流程编排:receive → pipeline → send(StepExecutor)
│ └── transfer 数据传输:send / receive(6 平台)
Expand All @@ -38,14 +39,18 @@ qtcloud-data
│ ├── review 质量审查:审计任意阶段产物(需求 / 设计 / 实现 / 交付)
│ ├── spec Specification 工具:wrap(包装 envelope)/ validate(结构校验)
│ ├── catalog 数据目录:volume 登记(list / show / add / rm)
│ └── version 规格版本管理:git 历史(list / show / diff)
│ └── spec version 规格版本管理:git 历史(list / show / diff)
└── 查看命令(定义查看)
├── blueprint 蓝图定义查看(list / show)
├── contract 契约定义查看(list / show)
└── pipeline 管道定义查看(list / show)
```

全局选项 `--json` 当前统一命令错误输出为 `{"error":{"code","message"}}`;
`spec validate`、`pipeline list/show`、`blueprint list/show` 和 `catalog` 已提供结构化成功输出,
其余成功结果按命令逐步迁移。

## 命令分类原则

命令按**用途定位**分三类,而非按名称平铺:
Expand Down Expand Up @@ -110,7 +115,8 @@ fn show(name: &str) -> Result<(), CliError> {
- 错误路径通过 `Result` 传播,**不直接 `std::process::exit(1)`**(仅 `main` 保留 bin 入口 exit)
- `Result<_, String>` 的公开函数已收敛为 `CliError`(`From<io::Error>/String/&str`)
- 错误路径因此可测试:`cmd_xxx(...).unwrap_err()`
- `CliError` 只携带用户可读消息(`Display` 即消息本体),不携带结构化错误码
- `CliError` 携带稳定错误码和用户可读消息;`Display` 仍只输出消息本体,
`to_json_value()` 提供 `--json` 可复用的 `{code, message}` 错误对象

### 注册表与工具机制(registry.rs + util.rs)

Expand Down
8 changes: 4 additions & 4 deletions src/cli/docs/transfer.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ pub fn send(file: &str, remote: Option<&str>, output: Option<&Path>, provider: &
```

- **委派逃生舱**:`QTDATA_CLI` 环境变量设置时,委派给外部 CLI(`transfer receive/send` 子命令)——测试与部署场景使用
- **进程内路径**:默认走进程内 provider(tokio runtime + StorageProvider
- **进程内路径**:默认走进程内 provider(tokio runtime + Storage
- 错误类型收敛为 `CliError`

## StorageProvider trait
## Storage trait

```rust
#[async_trait]
pub trait StorageProvider: Send + Sync {
pub trait Storage: Send + Sync {
fn name(&self) -> &'static str;
async fn send(&self, local_path: &str, remote_path: &str) -> Result<String, String>;
async fn receive(&self, url: &str, local_path: &str) -> Result<(), String>;
Expand Down Expand Up @@ -61,7 +61,7 @@ pub trait StorageProvider: Send + Sync {

## 添加新平台

1. 新建 `storage/<name>.rs`,实现 `StorageProvider` trait
1. 新建 `storage/<name>.rs`,实现 `Storage` trait
2. 在 `storage/mod.rs` 注册(`from_name` / `detect`)
3. 认证环境变量约定见上表
4. 需要 mock 的平台遵循 `*_with_base` 注入约定并补 wiremock 测试(`tests/storage_test.rs` 参考)
4 changes: 2 additions & 2 deletions src/cli/docs/version.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ qtcloud-data version diff <name> <v1> <v2> # 比较两个版本
- 优先 `spec/`(`<name>-blueprint.cue`),回退旧 `blueprint/`(`<name>.cue`)
- 依赖 git 可用

## 命令降级(v0.3 移除顶层)
## 命令降级

顶层 `qtcloud-data version` 已废弃,主入口改为 `qtcloud-data spec version`:

Expand All @@ -27,4 +27,4 @@ qtcloud-data spec version show <name> <version>
qtcloud-data spec version diff <name> <v1> <v2>
```

顶层 `version` 命令在 v0.3 移除前保留(帮助已标注废弃与替代入口)。
顶层 `version` 命令在兼容迁移期间保留(帮助已标注废弃与替代入口);新代码应使用 `spec version`
40 changes: 30 additions & 10 deletions src/cli/src/doctor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ use std::io::{self, Write};
use std::path::{Path, PathBuf};

use crate::error::CliError;
use crate::runtime;

#[derive(Args)]
pub struct DoctorArgs {
Expand Down Expand Up @@ -98,15 +99,24 @@ fn checks_with_dirs(dirs: &[DataDir]) -> Vec<Check> {
check_command("git", true, "版本记录和协作事实源需要 git"),
check_command("cargo", true, "CLI 开发和发布需要 cargo"),
check_command("rustc", true, "CLI 编译需要 rustc"),
check_command(
"python3",
false,
"process 执行 Python pipeline 时会用到 python3",
),
check_command("bash", false, "process 执行 shell pipeline 时会用到 bash"),
check_command("cue", false, "cue 可选增强:CUE 模块化目录查看时使用"),
];

for adapter in runtime::registered() {
if let Some(command) = adapter.doctor_command() {
let mut check = check_command(
command,
false,
format!(
"process 执行 {} pipeline 时会用到 {command}",
adapter.name()
),
);
check.name = format!("runtime:{}", adapter.name());
checks.push(check);
}
}

for dir in dirs {
checks.push(check_directory(&dir.path, &dir.name));
}
Expand Down Expand Up @@ -306,7 +316,8 @@ fn create_data_dirs(dirs: &[DataDir]) -> Vec<Check> {
}

// ── 检查函数 ──
fn check_command(command: &str, required: bool, purpose: &str) -> Check {
fn check_command(command: &str, required: bool, purpose: impl Into<String>) -> Check {
let purpose = purpose.into();
if command_exists(command) {
Check::pass(command, format!("{purpose}: found"))
} else if required {
Expand Down Expand Up @@ -629,10 +640,19 @@ mod tests {
let checks = checks_with_dirs(&dirs);
let names: Vec<&str> = checks.iter().map(|c| c.name.as_str()).collect();

// 6 个工具检查
for tool in ["git", "cargo", "rustc", "python3", "bash", "cue"] {
// 基础工具检查
for tool in ["git", "cargo", "rustc", "cue"] {
assert!(names.contains(&tool), "缺工具检查: {tool}");
}
// 运行时检查由 runtime 注册表驱动,builtin 不需要外部命令
for runtime in ["python", "r", "stata", "matlab", "bash"] {
assert!(
names
.iter()
.any(|name| name == &format!("runtime:{runtime}")),
"缺 runtime 检查: {runtime}"
);
}
// 2 个目录检查
assert!(names.contains(&"DRD"));
assert!(names.contains(&"SPEC"));
Expand All @@ -647,7 +667,7 @@ mod tests {
] {
assert!(names.contains(&env), "缺 env 检查: {env}");
}
assert_eq!(checks.len(), 6 + 2 + 6);
assert_eq!(checks.len(), 4 + 5 + 2 + 6);

std::fs::remove_dir_all(&root).ok();
}
Expand Down
Loading
Loading