ingest-hash-trigger SDD task 4:觸發鏈 Gitea webhook → collector sync 直打 named-webhook

- collector Go:trigger.go+sync 子命令(scan→R2 upload→POST collector-trigger.v1
  →2xx 才 MarkIngested 回寫;失敗不回寫=下輪自然重試;上傳失敗事件不送、同路徑
  renamed 連坐不回寫;防呆警告輪照送 warnings)。go test 19/19 綠(httptest mock)。
- workflow v3:rag-ingest-cards.local.yaml 改吃 collector payload(events[] 取代
  commits[];原稿從 R2 撈 CF REST API objects/{r2_key};新增 renamed 分支=只更新
  路徑映射 R5,跨界搬移轉 added/removed;removed 沿用 G9 雙判準;M3 recompute 尾鏈
  原樣保留,fetch_index 改 R2);rag-ingest.yaml v1 同步等價改。source_uri 遷移
  gitea://→kb://。schema_version!==1 整輪 no-op。
- push-demo-workflow.sh:補 __CF_ACCOUNT_ID__/__R2_BUCKET__/__CF_R2_TOKEN__ 佔位;
  GITEA_* 標 deprecated-for-ingest(rag-extract* 仍用)。
- Gitea 退場(本 repo 範圍):legacy JS collector 六檔刪除(git 記錄可查);
  install.sh 步驟 5 降級 deprecated;README 落「Gitea 退場切換 checklist」
  (推 v3→collector sync 實測→拆 Gitea webhook→觀察期+回滾法)。
- 驗證:yaml parse(raw+sed 模擬)/flow-config 對齊/JS 語法 node --check/
  demo cypher 唯讀 /cypher/search 編譯 missing=[]、出邊序如設計。
  未部署、未實跑 e2e(推 demo=leo 閘)。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-19 21:58:39 +08:00
parent fdb7a67484
commit 1c2fa6b294
10 changed files with 495 additions and 581 deletions
+30 -100
View File
@@ -1,18 +1,18 @@
# collector
> **本目錄有兩代 collector 並存(2026-07-19**
>
> - **新(現役開發線)= Go 版 hash 偵測 collector**`*.go`SDD `ingest-hash-trigger` task 2)——
> 變更偵測回歸檔案系統,manifestcontent hash**不經 git、不經 Gitea**。見下方「Go 版」段。
> - **舊(legacy,待 SDD task 4 拆除)= Node 版 Gitea push 鏈**`index.js`/`transform.js`/`git-sync.js`/`config.js`
> rag-wave1 T3/T4 產物)——watch → git commit/push → Gitea webhook。這正是 de-Gitea 要取代的鏈,
> 保留只為現行本機 demo 鏈還在用;新功能一律做在 Go 版。
> **legacy Node 版(Gitea push 鏈)已於 2026-07-19 刪除**SDD `ingest-hash-trigger` task 4
> 「Gitea webhook 接收端程式碼與路由已刪除(git 記錄可查)」)。`index.js`/`transform.js`/
> `git-sync.js`/`config.js` 那套 watch → git commit/push → Gitea webhook 鏈要考古請看 git 歷史
> (本檔同 commit 之前的版本有完整說明)。現役=Go 版 hash 偵測 collector
> **不經 git、不經 Gitea**scan → R2 上傳 → 直打 arcrun named-webhook。
> Markitdown 轉檔(docx/pptx/pdf→md)能力隨 legacy 一併退場,回歸產品化段 task 11 重做進 Go 版。
## Go 版:hash 偵測 collectorSDD ingest-hash-trigger
```
collector scan --root <知識資料夾> --manifest <manifest.json> [--max-removed-ratio 0.4] [--dry-run]
collector upload --root <知識資料夾> --manifest <manifest.json> [--max-removed-ratio 0.4] [--dry-run]
collector sync --root <知識資料夾> --manifest <manifest.json> [--max-removed-ratio 0.4] [--dry-run]
```
一次掃描:走訪資料夾(先只認 .md/.markdown/.txt/.docx/.pptx/.pdf)→ mtime+size fast-path
@@ -24,10 +24,10 @@ collector upload --root <知識資料夾> --manifest <manifest.json> [--max-remo
- **大量刪除防呆(R6**removed 數 > manifest 條目 × 40%`--max-removed-ratio` 可調)→
removed 全部不執行、manifest 條目保留、輸出 `mass_delete_guard` 警告。
- **重試語意**`ingested_hash` 只會在整條 ingest 鏈成功後回寫(回寫鉤子=`Manifest.MarkIngested`
task 4 觸發鏈呼叫);掃描與 R2 上傳都不寫它,所以「偵測過但未成功 ingest」的檔每輪都會
重發 added/modified——這是設計(design §2),不是 bugR2 端靠存在檢查 no-op不會重複上傳。
- 打 arcrun named-webhook 觸發 ingest、daemon 常駐(launchd)=之後的 task
SDD task 4、journeys/user-onboarding 環 6
`sync` 在觸發回 2xx 後呼叫);掃描與 R2 上傳都不寫它,所以「偵測過但未成功 ingest」的檔
每輪都會重發 added/modified——這是設計(design §2),不是 bugR2 端靠存在檢查 no-op
不會重複上傳。
- daemon 常駐(launchd)=產品化段 task 11
### `upload`R2 content-addressed 原稿上傳(SDD task 3design §4
@@ -62,97 +62,27 @@ hash 不符不上傳/env 缺漏報錯/MarkIngested 鉤子),httptest mock
live e2e2026-07-19):uncle6 帳號 `arcrun-rag-raw-demo` bucket 真上傳→重傳 no-op→
`wrangler r2 object get --remote` 下載 diff 一致、sha256 與 key 相符,全通。
---
# legacyNode 版收集端骨架(rag-wave1 T3+T4
### `sync`:一條龍(SDD task 4)= scan → upload → 觸發 ingest → 成功回寫
> design.md §4;跑在**客戶端機器**(NAS/VM,或導入者代管的 VPS),不是 arcrun workflowarcrun 零件禁檔案系統,見 CLAUDE.md 紅線)。
> ⚠️ **本骨架端到端(真實 NAS/VM + 真實 Gitea remote + systemd 常駐)需在本機/客戶環境驗,雲端 sandbox 沒有這些條件**——這裡只做得到:程式邏輯本身可跑、對本機臨時目錄的煙測(見下方「已驗證」)。
`sync``upload` 再加最後一步:把整輪的 collector-trigger.v1 payload POST 到
`ARCRUN_TRIGGER_URL`arcrun named-webhook 完整 URL,如
`{cypher}/webhooks/named/{ns}/rag_ingest/trigger`——端點是 arcrun 原生觸發機制,
design 鐵律段明言保留;被刪掉的是「Gitea push 事件」這個來源語意)。
## 做什麼
- **回寫語意**HTTP 2xx 才對本輪送出的 added/modified/renamed 檔呼叫 `Manifest.MarkIngested`
非 2xx/網路錯不回寫=下輪自然重試(exit 1)。
- **上傳失敗的事件不送**schema 約定 added/modified 的 `r2_key`=原稿已在 R2,上傳失敗還送
=叫消費端去 404 → 該事件本輪擋下(`dispatch.dropped_paths`),下輪重試補送;同路徑的
renamed 也不回寫(防「內容從未上 R2 卻被標 ingested」)。
- **防呆警告輪照送**mass_delete_guard 觸發時 removed 事件已被壓下,但 payload 連同
`warnings[]` 照送——消費端看得到警告、不執行下架;notify 呈現歸 collector 端輸出/daemon。
- **無變更輪不發送**`skipped_no_changes`)。
- 輸出 JSON`{"trigger": <實際送出的 payload>, "uploads": [...], "dispatch":
{status, http_status?, error?, marked_count, dropped_paths?}}`。
```
watch 知識資料夾 ──(新增/修改)──► transform.md passthroughdocx,pptx,pdf→Markitdown)──► 寫入 target repo 工作目錄
──(刪除) ──► target repo 移除對應檔
git add/commit/push
Gitea push webhook(既有機制,非本骨架自建)
arcrun ingest workflow(另案,如 km_wiki_ingest_drain 同款模式)
```
設定=upload 的三個環境變數+`ARCRUN_TRIGGER_URL`(皆絕不落 repo/code)。
**關鍵設計判斷**collector 的責任止於「commit + push」。design.md §4 說的「打 ingest webhook」= Gitea 自己的 push webhook 機制(同 `Leo/Arcrun` `registry/examples/km-wiki-ingest` 已驗證的模式:Gitea push webhook → arcrun workflow),**不是** collector 自己再打一支 HTTP webhook。collector 不需要知道 ingest 的內部細節(哪個 workflow、哪個 KBDB),它只管「客戶檔案的忠實鏡像」進 Gitea repo。
## 事件驅動、不輪詢
`chokidar`Nodewatch 檔案系統事件(inotify/FSEvents,非 polling),對齊「單一 repo、事件驅動」的鐵律。debounce 視窗(預設 3 秒)把同一批變更合併成一次 commit,避免每個檔案獨立 commit 洗歷史。
## 刪檔語意
檔案在來源資料夾被刪除 → collector 在 target repo 對應路徑也 `git rm` → commit → push。**deprecated 標記不是 collector 的責任**——ingest workflow 收到 Gitea push event 裡的 `removed` 檔案清單後,自己去 KBDB 把對應 entry 標 `status: deprecated`append-only,不物理刪,見 km-wiki-ingest description.md 冪等設計)。collector 只管檔案鏡像忠實,不碰 KBDB。
## 轉檔(T4
`transform.js``.md`/`.markdown` passthroughfrontmatter 補 `source_path` 溯源欄位);`.docx`/`.pptx`/`.pdf` 呼叫 `markitdown` CLI(子行程,需部署機器已 `pip install markitdown[docx,pptx,pdf]`)轉出 md,輸出路徑副檔名換成 `.md`,同時把**原檔**複製進 target repo 的 `assets/originals/<相對路徑>`design.md §4「原檔進 LFS」)。`git-sync.js``ensureGitAttributes()` 在啟動時冪等寫入 `.gitattributes``assets/originals/**/*.{pdf,docx,pptx} filter=lfs ...`)。
⚠️ **LFS 是否真的生效待本機驗**`.gitattributes` 宣告本身雲端驗過內容正確,但 LFS smudge/clean filter 要部署機器裝了 `git-lfs``git lfs install` 才真的把大檔案存進 LFS store(否則 git 仍會把二進位檔案當一般 blob 存進版控歷史——功能上檔案還是會進 repo,只是沒享受到 LFS 的空間/頻寬優化)。雲端 sandbox 沒有 `git-lfs` 二進位,這段驗不到。
其餘格式(xlsx、圖片等)不在 design.md §4 第一波承諾範圍,仍丟 `NotImplementedError` 並記警告日誌,誠實不假裝轉好。
## 檔案
| 檔案 | 職責 |
|---|---|
| `index.js` | 進入點:watch+debounce+事件分派+原檔複製 |
| `transform.js` | 檔案 → md 轉換:`.md` passthrough`docx,pptx,pdf` 走 Markitdown |
| `git-sync.js` | target repo 的 git add/commit/push 封裝+`.gitattributes` LFS 宣告 |
| `config.js` | 環境變數讀取(`WATCH_DIR`/`TARGET_REPO_DIR`/`DEBOUNCE_MS` |
## 設定(環境變數)
| 變數 | 說明 | 預設 |
|---|---|---|
| `WATCH_DIR` | 客戶知識資料夾(來源) | 必填 |
| `TARGET_REPO_DIR` | 已 clone 好、有 push 權限的 Gitea repo 工作目錄(去向) | 必填 |
| `TARGET_SUBDIR` | 在 target repo 內落地的子目錄 | `collected/` |
| `DEBOUNCE_MS` | 合併變更的等待視窗 | `3000` |
| `GIT_AUTHOR_NAME` / `GIT_AUTHOR_EMAIL` | commit 署名 | `collector` / `collector@localhost` |
## 部署(客戶端機器,設計稿——本輪未實際跑 systemd)
```ini
# /etc/systemd/system/arcrun-rag-collector.service(範本,未部署未測)
[Unit]
Description=arcrun-rag collector
After=network.target
[Service]
Environment=WATCH_DIR=/mnt/knowledge
Environment=TARGET_REPO_DIR=/opt/collector-repo
ExecStart=/usr/bin/node /opt/arcrun-rag/collector/index.js
Restart=always
User=collector
[Install]
WantedBy=multi-user.target
```
`TARGET_REPO_DIR` 需事先 `git clone`+設好有 push 權限的 remotecredential 用該機器的 git credential helper 或 SSH key,不是本骨架管的事)。
## 已驗證(雲端 sandbox 能做到的部分)
- `node --check` 語法檢查全過。
- 本機臨時目錄煙測(非真實客戶環境,兩輪):起一個 scratch git repo 當 target、一個 scratch 資料夾當 watch 來源:
1. **T3 事件機制**:新增/修改/刪除 `.md` 檔案 → collector 正確偵測、寫入/移除、debounce 後產生一次 commit`git log` 驗到 commit 內容與變更一致。
2. **T4 Markitdown**:用 `python-docx` 生一份真實 `.docx`(含標題+段落)丟進 watch 資料夾 → collector 呼叫 `markitdown` 轉出真實 md 內容(人工核對文字與原檔一致)、原檔複製進 `assets/originals/``.gitattributes` 正確寫入三種格式的 LFS 宣告、整批 commit+push 送達 bare remote`git log` 驗證)。
**不含**(待本機/客戶環境):真實 Gitea remote push(需真實 token+repo)、真實 git-lfs smudge/clean filtersandbox 無 `git-lfs` 二進位)、systemd 常駐、NAS/VM 環境、Gitea push webhook 是否真觸發 ingest workflow。
## 待本機/客戶環境驗(端到端)
1. 真實客戶知識資料夾 watch(含各類真實 docx/pptx/pdf 樣本,非合成測試檔)。
2. 真實 Gitea remote push(含憑證管理)。
3. 真實 git-lfs 安裝+`git lfs track`,確認大檔案真的走 LFS store 而非塞進一般 blob 歷史。
4. push 後確認 Gitea push webhook 真觸發 ingest workflow(另案)。
5. systemd 常駐穩定性(重開機自動起、崩潰自動重啟)。
測試:`go test ./...` 19/19——掃描 7+上傳 7sync/trigger 5(成功回寫+無變更輪不重發/
失敗不回寫+修復後重試成功/防呆警告輪照送 warnings 零事件/上傳失敗事件擋下+renamed
連坐不回寫/URL env 驗證),全部 httptest mock**未實跑雲端 e2e**——等切換日與 leo 一起驗)。
-20
View File
@@ -1,20 +0,0 @@
function requireEnv(name) {
const v = process.env[name];
if (!v) {
throw new Error(`collector: 缺少環境變數 ${name}(見 README.md 設定表)`);
}
return v;
}
function loadConfig() {
return {
watchDir: requireEnv('WATCH_DIR'),
targetRepoDir: requireEnv('TARGET_REPO_DIR'),
targetSubdir: process.env.TARGET_SUBDIR || 'collected/',
debounceMs: parseInt(process.env.DEBOUNCE_MS || '3000', 10),
gitAuthorName: process.env.GIT_AUTHOR_NAME || 'collector',
gitAuthorEmail: process.env.GIT_AUTHOR_EMAIL || 'collector@localhost',
};
}
module.exports = { loadConfig, requireEnv };
-61
View File
@@ -1,61 +0,0 @@
const fs = require('fs');
const path = require('path');
const { execFileSync } = require('child_process');
// design.md §4「原檔進 LFS」:assets/originals/ 底下的常見二進位格式走 git-lfs。
// ⚠️ 這裡只寫 .gitattributes 宣告(冪等、不重複寫)——LFS 是否真的生效取決於部署機器
// 有沒有裝 git-lfs 並 `git lfs install`;雲端 sandbox 沒有 git-lfs 二進位,只能驗到
// .gitattributes 內容正確,驗不到真實 LFS smudge/clean filter,見 README「待本機驗」。
const LFS_PATTERNS = ['assets/originals/**/*.pdf', 'assets/originals/**/*.docx', 'assets/originals/**/*.pptx'];
function ensureGitAttributes(repoDir) {
const gaPath = path.join(repoDir, '.gitattributes');
const existing = fs.existsSync(gaPath) ? fs.readFileSync(gaPath, 'utf8') : '';
const missing = LFS_PATTERNS.filter((p) => !existing.includes(p));
if (missing.length === 0) return false;
const lines = missing.map((p) => `${p} filter=lfs diff=lfs merge=lfs -text`);
const next = existing.length && !existing.endsWith('\n') ? `${existing}\n` : existing;
fs.writeFileSync(gaPath, next + lines.join('\n') + '\n', 'utf8');
return true;
}
/**
* target repo 的 git add/commit/push 封裝(design.md §4git commit/push 觸發既有
* Gitea push webhook → ingest workflow;本模組不打任何 ingest HTTP 端點)。
*/
class GitSync {
constructor(config) {
this.repoDir = config.targetRepoDir;
this.authorName = config.gitAuthorName;
this.authorEmail = config.gitAuthorEmail;
ensureGitAttributes(this.repoDir);
}
_git(args) {
return execFileSync('git', args, {
cwd: this.repoDir,
env: {
...process.env,
GIT_AUTHOR_NAME: this.authorName,
GIT_AUTHOR_EMAIL: this.authorEmail,
GIT_COMMITTER_NAME: this.authorName,
GIT_COMMITTER_EMAIL: this.authorEmail,
},
encoding: 'utf8',
});
}
/** 一批變更合併成一次 commit + push;無變更則不 commit(冪等,避免空 commit 洗歷史)。 */
commitAndPush(summary) {
this._git(['add', '-A']);
const status = this._git(['status', '--porcelain']);
if (!status.trim()) {
return { committed: false };
}
this._git(['commit', '-m', summary]);
this._git(['push']);
return { committed: true };
}
}
module.exports = { GitSync };
-90
View File
@@ -1,90 +0,0 @@
const fs = require('fs');
const path = require('path');
const chokidar = require('chokidar');
const { loadConfig } = require('./config');
const { transformFile, NotImplementedError } = require('./transform');
const { GitSync } = require('./git-sync');
function ensureDir(p) {
fs.mkdirSync(p, { recursive: true });
}
function outputPathFor(config, relOutputPath) {
return path.join(config.targetRepoDir, config.targetSubdir, relOutputPath);
}
function handleAddOrChange(config, srcPath) {
try {
const { relOutputPath, content, originalCopy } = transformFile(srcPath, config.watchDir);
const outPath = outputPathFor(config, relOutputPath);
ensureDir(path.dirname(outPath));
fs.writeFileSync(outPath, content, 'utf8');
console.log(`[collector] 寫入 ${relOutputPath}`);
if (originalCopy) {
const origOutPath = outputPathFor(config, originalCopy.relPath);
ensureDir(path.dirname(origOutPath));
fs.copyFileSync(srcPath, origOutPath);
console.log(`[collector] 原檔複製 ${originalCopy.relPath}(LFS 追蹤需部署機器已裝 git-lfs)`);
}
} catch (e) {
if (e instanceof NotImplementedError) {
console.warn(`[collector] 跳過(待 T4):${e.message}`);
} else {
console.error(`[collector] 轉檔失敗 ${srcPath}${e.message}`);
}
}
}
function handleUnlink(config, srcPath) {
const relSrc = path.relative(config.watchDir, srcPath);
const outPath = outputPathFor(config, relSrc);
if (fs.existsSync(outPath)) {
fs.unlinkSync(outPath);
console.log(`[collector] 移除 ${relSrc}(來源已刪除,deprecated 標記交給 ingest workflow`);
}
}
function startCollector(config, { git = new GitSync(config) } = {}) {
ensureDir(path.join(config.targetRepoDir, config.targetSubdir));
let pendingCount = 0;
let debounceTimer = null;
const flush = () => {
if (pendingCount === 0) return;
const n = pendingCount;
pendingCount = 0;
const result = git.commitAndPush(`collector: 同步 ${n} 項變更`);
if (result.committed) {
console.log(`[collector] 已 commit+push${n} 項變更)`);
}
};
const schedule = () => {
pendingCount += 1;
if (debounceTimer) clearTimeout(debounceTimer);
debounceTimer = setTimeout(flush, config.debounceMs);
};
const watcher = chokidar.watch(config.watchDir, {
ignoreInitial: false, // 首次啟動=首灌,掃過整個資料夾
persistent: true,
});
watcher
.on('add', (p) => { handleAddOrChange(config, p); schedule(); })
.on('change', (p) => { handleAddOrChange(config, p); schedule(); })
.on('unlink', (p) => { handleUnlink(config, p); schedule(); });
console.log(`[collector] watch 中:${config.watchDir}${config.targetRepoDir}/${config.targetSubdir}`);
return watcher;
}
if (require.main === module) {
const config = loadConfig();
startCollector(config);
}
module.exports = { startCollector };
+86 -24
View File
@@ -6,6 +6,8 @@
// [--max-removed-ratio 0.4] [--dry-run]
// collector upload --root <知識資料夾> --manifest <manifest.json 路徑> \
// [--max-removed-ratio 0.4] [--dry-run]
// collector sync --root <知識資料夾> --manifest <manifest.json 路徑> \
// [--max-removed-ratio 0.4] [--dry-run]
//
// scan:走訪 root、對照 manifest、把事件(符合 schemas/collector-trigger.v1.schema.json
// 以 JSON 輸出到 stdout,並更新 manifest--dry-run 不寫)。
@@ -15,7 +17,12 @@
// R2_BUCKET(絕不落 repo);--dry-run 只列出會上傳的 keyplanned),不碰網路不寫 manifest。
// 任一上傳失敗=exit 1manifest 照存:content_hash 反映現況、ingested_hash 不動=可重試)。
//
// daemon 常駐(launchd/打 named-webhook 觸發 ingest=之後的 taskSDD task 4、產品化段)。
// synctask 4)=scanupload+把整輪 payload POST 到 ARCRUN_TRIGGER_URLarcrun
// named-webhook 觸發 ingest cypher workflow)。HTTP 2xx 才對本輪送出的 added/modified/
// renamed 事件回寫 Manifest.MarkIngested;失敗不回寫(下輪自然重試)。上傳失敗的事件
// 不隨 payload 送出(r2_key 語意=原稿已在 R2)。防呆警告輪照送 warnings、不含下架事件。
//
// daemon 常駐(launchd)=之後的 task(產品化段 task 11)。
package main
import (
@@ -24,8 +31,14 @@ import (
"fmt"
"os"
"path/filepath"
"time"
)
type runMode struct {
withUpload bool
withTrigger bool
}
func main() {
if len(os.Args) < 2 {
usage()
@@ -33,9 +46,11 @@ func main() {
}
switch os.Args[1] {
case "scan":
os.Exit(run(os.Args[2:], false))
os.Exit(run(os.Args[2:], runMode{}))
case "upload":
os.Exit(run(os.Args[2:], true))
os.Exit(run(os.Args[2:], runMode{withUpload: true}))
case "sync":
os.Exit(run(os.Args[2:], runMode{withUpload: true, withTrigger: true}))
default:
usage()
os.Exit(2)
@@ -46,16 +61,18 @@ func usage() {
fmt.Fprintln(os.Stderr, `用法:
collector scan --root <dir> --manifest <file> [--max-removed-ratio 0.4] [--dry-run]
collector upload --root <dir> --manifest <file> [--max-removed-ratio 0.4] [--dry-run]
upload 需環境變數: CF_ACCOUNT_ID / CF_API_TOKEN / R2_BUCKET`)
collector sync --root <dir> --manifest <file> [--max-removed-ratio 0.4] [--dry-run]
upload 需環境變數: CF_ACCOUNT_ID / CF_API_TOKEN / R2_BUCKET
sync 另需: ARCRUN_TRIGGER_URLnamed-webhook 觸發完整 URL`)
}
// run 是 scan/upload 共用主體withUploadtrue 時掃描後把 added/modified 原稿上傳 R2
func run(args []string, withUpload bool) int {
// run 是 scan/upload/sync 共用主體。
func run(args []string, mode runMode) int {
fs := flag.NewFlagSet("collector", flag.ExitOnError)
root := fs.String("root", "", "知識資料夾根路徑(必填)")
manifestPath := fs.String("manifest", "", "manifest JSON 檔路徑(必填;不存在會建新)")
ratio := fs.Float64("max-removed-ratio", DefaultMaxRemovedRatio, "大量刪除防呆門檻(removed 數 > manifest 條目 × 本值 → 全部不下架、只發警告)")
dryRun := fs.Bool("dry-run", false, "只輸出事件(upload 模式另列 planned 上傳清單),不更新 manifest、不碰網路")
dryRun := fs.Bool("dry-run", false, "只輸出事件(upload/sync 模式另列 planned 清單),不更新 manifest、不碰網路")
if err := fs.Parse(args); err != nil {
return 2
}
@@ -64,15 +81,26 @@ func run(args []string, withUpload bool) int {
return 2
}
// upload 模式先驗設定(fail fast:缺 env 連掃都不掃,不留半套狀態)。
// 先驗設定(fail fast:缺 env 連掃都不掃,不留半套狀態)。
var client *R2Client
if withUpload && !*dryRun {
cfg, err := LoadR2ConfigFromEnv()
if err != nil {
fmt.Fprintln(os.Stderr, "collector:", err)
return 2
var triggerURL string
if !*dryRun {
if mode.withUpload {
cfg, err := LoadR2ConfigFromEnv()
if err != nil {
fmt.Fprintln(os.Stderr, "collector:", err)
return 2
}
client = NewR2Client(cfg)
}
if mode.withTrigger {
u, err := LoadTriggerURLFromEnv()
if err != nil {
fmt.Fprintln(os.Stderr, "collector:", err)
return 2
}
triggerURL = u
}
client = NewR2Client(cfg)
}
absRoot, err := filepath.Abs(*root)
@@ -96,8 +124,10 @@ func run(args []string, withUpload bool) int {
return fail(err)
}
exitCode := 0
var uploads []UploadResult
if withUpload {
if mode.withUpload {
if *dryRun {
uploads = []UploadResult{}
for _, ev := range payload.Events {
@@ -107,6 +137,38 @@ func run(args []string, withUpload bool) int {
}
} else {
uploads = UploadChanged(absRoot, payload.Events, client)
for _, r := range uploads {
if r.Status == "failed" {
exitCode = 1 // 有敗=非零退出;ingested_hash 未動=下輪自然重試
}
}
}
}
// syncPOST 觸發 → 2xx 才回寫 MarkIngested(在 m.Save 之前,回寫才進得了檔)。
outPayload := payload
var dispatch *TriggerResult
if mode.withTrigger {
if *dryRun {
dispatch = &TriggerResult{Status: "planned"}
} else {
sendable, dropped := BuildSendablePayload(payload, uploads)
outPayload = sendable
dispatch = &TriggerResult{DroppedPaths: dropped}
if len(sendable.Events) == 0 && len(sendable.Warnings) == 0 {
dispatch.Status = "skipped_no_changes" // 無變更輪不發送(schema 註明空發也合法,但沒必要)
} else {
status, terr := SendTrigger(triggerURL, sendable, nil)
dispatch.HTTPStatus = status
if terr != nil {
dispatch.Status = "failed"
dispatch.Error = terr.Error()
exitCode = 1
} else {
dispatch.Status = "sent"
dispatch.MarkedCount = MarkIngestedEvents(m, sendable.Events, dropped, time.Now().Unix())
}
}
}
}
@@ -116,18 +178,18 @@ func run(args []string, withUpload bool) int {
}
}
exitCode := 0
var out any = payload
if withUpload {
var out any = outPayload
if mode.withTrigger {
out = struct {
Trigger *TriggerPayload `json:"trigger"`
Uploads []UploadResult `json:"uploads"`
Dispatch *TriggerResult `json:"dispatch"`
}{outPayload, uploads, dispatch}
} else if mode.withUpload {
out = struct {
Trigger *TriggerPayload `json:"trigger"`
Uploads []UploadResult `json:"uploads"`
}{payload, uploads}
for _, r := range uploads {
if r.Status == "failed" {
exitCode = 1 // 有敗=非零退出;manifest 已存、ingested_hash 未動=下輪自然重試
}
}
}{outPayload, uploads}
}
data, err := json.MarshalIndent(out, "", " ")
if err != nil {
-201
View File
@@ -1,201 +0,0 @@
{
"name": "arcrun-rag-collector",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "arcrun-rag-collector",
"version": "0.1.0",
"dependencies": {
"chokidar": "^3.6.0"
}
},
"node_modules/anymatch": {
"version": "3.1.3",
"resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz",
"integrity": "sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==",
"license": "ISC",
"dependencies": {
"normalize-path": "^3.0.0",
"picomatch": "^2.0.4"
},
"engines": {
"node": ">= 8"
}
},
"node_modules/binary-extensions": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz",
"integrity": "sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==",
"license": "MIT",
"engines": {
"node": ">=8"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/braces": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz",
"integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==",
"license": "MIT",
"dependencies": {
"fill-range": "^7.1.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/chokidar": {
"version": "3.6.0",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.6.0.tgz",
"integrity": "sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw==",
"license": "MIT",
"dependencies": {
"anymatch": "~3.1.2",
"braces": "~3.0.2",
"glob-parent": "~5.1.2",
"is-binary-path": "~2.1.0",
"is-glob": "~4.0.1",
"normalize-path": "~3.0.0",
"readdirp": "~3.6.0"
},
"engines": {
"node": ">= 8.10.0"
},
"funding": {
"url": "https://paulmillr.com/funding/"
},
"optionalDependencies": {
"fsevents": "~2.3.2"
}
},
"node_modules/fill-range": {
"version": "7.1.1",
"resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz",
"integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==",
"license": "MIT",
"dependencies": {
"to-regex-range": "^5.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/fsevents": {
"version": "2.3.3",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
"integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
},
"node_modules/glob-parent": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.2.tgz",
"integrity": "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==",
"license": "ISC",
"dependencies": {
"is-glob": "^4.0.1"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/is-binary-path": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz",
"integrity": "sha512-ZMERYes6pDydyuGidse7OsHxtbI7WVeUEozgR/g7rd0xUimYNlvZRE/K2MgZTjWy725IfelLeVcEM97mmtRGXw==",
"license": "MIT",
"dependencies": {
"binary-extensions": "^2.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/is-extglob": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz",
"integrity": "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/is-glob": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz",
"integrity": "sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==",
"license": "MIT",
"dependencies": {
"is-extglob": "^2.1.1"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/is-number": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz",
"integrity": "sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==",
"license": "MIT",
"engines": {
"node": ">=0.12.0"
}
},
"node_modules/normalize-path": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz",
"integrity": "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/picomatch": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
"integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==",
"license": "MIT",
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/readdirp": {
"version": "3.6.0",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.6.0.tgz",
"integrity": "sha512-hOS089on8RduqdbhvQ5Z37A0ESjsqz6qnRcffsMU3495FuTdqSm+7bhJ29JvIOsBDEEnan5DPu9t3To9VRlMzA==",
"license": "MIT",
"dependencies": {
"picomatch": "^2.2.1"
},
"engines": {
"node": ">=8.10.0"
}
},
"node_modules/to-regex-range": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz",
"integrity": "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==",
"license": "MIT",
"dependencies": {
"is-number": "^7.0.0"
},
"engines": {
"node": ">=8.0"
}
}
}
}
-10
View File
@@ -1,10 +0,0 @@
{
"name": "arcrun-rag-collector",
"version": "0.1.0",
"description": "收集端骨架(rag-wave1 T3):watch 客戶知識資料夾 → 轉檔 → git commit/push",
"main": "index.js",
"private": true,
"dependencies": {
"chokidar": "^3.6.0"
}
}
-75
View File
@@ -1,75 +0,0 @@
const fs = require('fs');
const path = require('path');
const { execFileSync } = require('child_process');
class NotImplementedError extends Error {}
const MARKDOWN_EXT = new Set(['.md', '.markdown']);
// design.md §4 明列的第一波轉檔格式:docx/pptx/pdf。其餘格式仍走 NotImplementedError
// 不擴大承諾範圍(品質不承諾是 design 原話,但格式範圍先照 SDD 講定的三種)。
const MARKITDOWN_EXT = new Set(['.docx', '.pptx', '.pdf']);
/**
* 檔案 → md 轉換(T4Markitdown adapter)。
* srcPath: 來源檔案絕對路徑(客戶知識資料夾內)
* 回傳:{ relOutputPath, content, originalCopy? }
* - originalCopy(僅 Markitdown 轉檔路徑):{ relPath },供 index.js 把原檔複製進
* target repo 的 assets/originals/design.md §4:「原檔進 LFS」——LFS 追蹤本身
* 依賴部署機器裝 git-lfs 並 `git lfs track`,本模組只管把原檔放進約定路徑,
* 不假裝在雲端 sandbox 驗過真實 LFS push,見 README「待本機驗」)。
*/
function transformFile(srcPath, watchDir) {
const ext = path.extname(srcPath).toLowerCase();
const relSrc = path.relative(watchDir, srcPath);
if (MARKDOWN_EXT.has(ext)) {
const raw = fs.readFileSync(srcPath, 'utf8');
const content = stampSourcePath(raw, relSrc);
return { relOutputPath: relSrc, content };
}
if (MARKITDOWN_EXT.has(ext)) {
const md = convertWithMarkitdown(srcPath);
const content = stampSourcePath(md, relSrc);
const relOutputPath = relSrc.slice(0, -ext.length) + '.md';
return {
relOutputPath,
content,
originalCopy: { relPath: path.join('assets/originals', relSrc) },
};
}
// 其餘格式(xlsx/圖片/...)— 不在 design.md §4 第一波承諾範圍,誠實丟未實作。
throw new NotImplementedError(
`${relSrc}:格式 ${ext} 不在第一波 Markitdown 承諾範圍(docx/pptx/pdf),本骨架先跳過`,
);
}
/** 呼叫 markitdown CLI 轉檔(品質不承諾,design.md §4 原話)。 */
function convertWithMarkitdown(srcPath) {
try {
return execFileSync('markitdown', [srcPath], {
encoding: 'utf8',
maxBuffer: 20 * 1024 * 1024,
});
} catch (e) {
throw new Error(`markitdown 轉檔失敗(${srcPath}):${e.message}`);
}
}
/** md frontmatter 補 source_path(溯源用,design.md §4:「md frontmatter 記原檔路徑」)。 */
function stampSourcePath(raw, relSrc) {
const stamp = `source_path: "${relSrc}"`;
if (raw.startsWith('---\n')) {
const end = raw.indexOf('\n---', 4);
if (end !== -1) {
const fm = raw.slice(4, end);
if (fm.includes('source_path:')) return raw; // 已有就不重複塞
return `---\n${stamp}\n${fm}\n---${raw.slice(end + 4)}`;
}
}
return `---\n${stamp}\n---\n\n${raw}`;
}
module.exports = { transformFile, NotImplementedError };
+128
View File
@@ -0,0 +1,128 @@
// trigger.go — 把一輪掃描的 collector-trigger.v1 payload POST 到 arcrun named-webhook
// SDD ingest-hash-trigger task 4:觸發鏈從 Gitea push webhook 改為 collector 直打)。
//
// 目標端點=arcrun 原生 named-webhook 觸發機制(design 鐵律段明言保留):
//
// POST {cypher}/webhooks/named/{ns}/rag_ingest/trigger
//
// 完整 URL 走環境變數 ARCRUN_TRIGGER_URL(絕不落 repo)。語意:
// - HTTP 2xx =本輪觸發成功 → 對「實際送出」的 added/modified/renamed 事件回寫
// Manifest.MarkIngesteddesign §2 的回寫鉤子,至此才第一次被呼叫)。
// - 非 2xx/網路錯 =不回寫(ingested_hash 不動)→ 下輪掃描自然重發=重試,
// R2 端靠存在檢查 no-op、ingest 端靠 source_hash 冪等(design §5),無腦重試安全。
// - 上傳失敗的 added/modified 事件「不」隨 payload 送出(schema 約定 r2_key=原稿已在
// R2;沒上去就送=消費端 fetch 必 404)——下輪重試補送。renamed/removed 不依賴 R2,照送。
// - 防呆警告輪(mass_delete_guard):removed 事件已被 collector 壓下,但 payload 連同
// warnings 照送(消費端/執行紀錄看得到警告,不執行下架)。
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
// LoadTriggerURLFromEnv 讀 ARCRUN_TRIGGER_URLsync 模式必填)。
func LoadTriggerURLFromEnv() (string, error) {
u := os.Getenv("ARCRUN_TRIGGER_URL")
if u == "" {
return "", fmt.Errorf("sync 缺環境變數:ARCRUN_TRIGGER_URLnamed-webhook 觸發完整 URL,設定只走環境變數,絕不寫進 repo/code")
}
if !strings.HasPrefix(u, "http://") && !strings.HasPrefix(u, "https://") {
return "", fmt.Errorf("ARCRUN_TRIGGER_URL 必須是完整 URLhttp/https),得到:%s", u)
}
return u, nil
}
// TriggerResult 是本輪觸發的結果,隨 sync 輸出 stdout。
type TriggerResult struct {
Status string `json:"status"` // sent | skipped_no_changes | failed | planned--dry-run
HTTPStatus int `json:"http_status,omitempty"`
Error string `json:"error,omitempty"`
MarkedCount int `json:"marked_count"` // 本輪成功回寫 ingested_hash 的檔數
DroppedPaths []string `json:"dropped_paths,omitempty"` // 因上傳失敗被擋下、未隨 payload 送出的事件路徑
}
// BuildSendablePayload 依上傳結果過濾掃描 payload:
// added/modified 只有上傳成功(uploaded/skipped_exists)才隨 payload 送出——
// schema 的 r2_key 語意=「原稿已在 R2」,上傳失敗還送=叫消費端去 404。
// renamed/removed 不依賴 R2 物件,一律保留;warnings 原樣保留(防呆輪照送)。
// 回傳(可送出的 payload 副本, 被擋下的路徑清單)。
func BuildSendablePayload(p *TriggerPayload, uploads []UploadResult) (*TriggerPayload, []string) {
okPaths := map[string]bool{}
for _, u := range uploads {
if u.Status == "uploaded" || u.Status == "skipped_exists" {
okPaths[u.Path] = true
}
}
sendable := *p
sendable.Events = []Event{}
var dropped []string
for _, ev := range p.Events {
if ev.Type == "added" || ev.Type == "modified" {
if !okPaths[ev.Path] {
dropped = append(dropped, ev.Path)
continue
}
}
sendable.Events = append(sendable.Events, ev)
}
return &sendable, dropped
}
// SendTrigger 把 payload POST 到 named-webhook。回傳 HTTP 狀態碼;非 2xx 視為錯誤。
// timeout 放寬到 300snamed-webhook 觸發的 ingest workflow 可能同步跑(demo 實測 20-30s+)。
func SendTrigger(url string, p *TriggerPayload, hc *http.Client) (int, error) {
if hc == nil {
hc = &http.Client{Timeout: 300 * time.Second}
}
body, err := json.Marshal(p)
if err != nil {
return 0, err
}
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return 0, err
}
req.Header.Set("Content-Type", "application/json")
resp, err := hc.Do(req)
if err != nil {
return 0, fmt.Errorf("觸發 POST 失敗(不回寫 manifest,下輪自然重試):%w", err)
}
defer resp.Body.Close()
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return resp.StatusCode, fmt.Errorf("觸發回 HTTP %d(不回寫 manifest,下輪自然重試):%s",
resp.StatusCode, strings.TrimSpace(string(snippet)))
}
return resp.StatusCode, nil
}
// MarkIngestedEvents 在觸發成功(2xx)後,對「實際送出」的 added/modified/renamed 事件
// 回寫 ingested_hash。droppedPaths=本輪因上傳失敗被擋下的路徑——同路徑若另有 renamed
// 事件(改名+內容從未成功 ingest 的檔會同輪補發 added),該 renamed 也不得回寫,
// 否則原稿永遠上不了 R2 卻被標成已 ingest。回傳成功回寫的檔數。
func MarkIngestedEvents(m *Manifest, sentEvents []Event, droppedPaths []string, at int64) int {
droppedSet := map[string]bool{}
for _, p := range droppedPaths {
droppedSet[p] = true
}
n := 0
for _, ev := range sentEvents {
switch ev.Type {
case "added", "modified", "renamed":
if droppedSet[ev.Path] {
continue
}
if m.MarkIngested(ev.Path, ev.SourceHash, at) {
n++
}
}
}
return n
}
+251
View File
@@ -0,0 +1,251 @@
package main
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"sync"
"testing"
"time"
)
// mockTrigger 用 httptest 模擬 arcrun named-webhook 觸發端點。
type mockTrigger struct {
mu sync.Mutex
status int // 回應狀態碼(預設 200
payloads []*TriggerPayload
}
func newMockTrigger(t *testing.T) (*httptest.Server, *mockTrigger) {
t.Helper()
m := &mockTrigger{status: http.StatusOK}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
if ct := r.Header.Get("Content-Type"); !strings.HasPrefix(ct, "application/json") {
t.Errorf("Content-Type 應為 application/json,得到 %q", ct)
}
body, err := io.ReadAll(r.Body)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
var p TriggerPayload
if err := json.Unmarshal(body, &p); err != nil {
t.Errorf("payload 不是合法 JSON: %v", err)
}
m.mu.Lock()
m.payloads = append(m.payloads, &p)
st := m.status
m.mu.Unlock()
w.WriteHeader(st)
if st >= 500 {
io.WriteString(w, `{"error":"mock 觸發失敗"}`)
} else {
io.WriteString(w, `{"success":true,"execution_id":"mock-exec"}`)
}
}))
t.Cleanup(srv.Close)
return srv, m
}
// syncRound 模擬 sync 的一輪:scan → upload(真 mock R2)→ 過濾 → POST → 2xx 才回寫。
// 與 main.run 的 sync 分支同邏輯(單測不跑 CLI 殼,直接組核心函式)。
func syncRound(t *testing.T, root string, m *Manifest, r2 *R2Client, url string) (*TriggerPayload, []UploadResult, *TriggerResult) {
t.Helper()
p := mustScan(t, root, m)
uploads := UploadChanged(root, p.Events, r2)
sendable, dropped := BuildSendablePayload(p, uploads)
res := &TriggerResult{DroppedPaths: dropped}
if len(sendable.Events) == 0 && len(sendable.Warnings) == 0 {
res.Status = "skipped_no_changes"
return sendable, uploads, res
}
status, err := SendTrigger(url, sendable, nil)
res.HTTPStatus = status
if err != nil {
res.Status = "failed"
res.Error = err.Error()
return sendable, uploads, res
}
res.Status = "sent"
res.MarkedCount = MarkIngestedEvents(m, sendable.Events, dropped, time.Now().Unix())
return sendable, uploads, res
}
// 情境 1:成功回寫——上傳全成、觸發 2xx → added 事件的檔回寫 ingested_hash
// 下一輪掃描歸零(不重發);伺服器收到的 payload 帶 schema_version/folder_id/r2_key。
func TestSyncSuccessMarksIngested(t *testing.T) {
_, _, r2 := newMockR2(t)
srv, trig := newMockTrigger(t)
root := t.TempDir()
content := "sync 測試內容\n"
writeFile(t, root, "a.md", content, baseTime)
m := newTestManifest()
_, uploads, res := syncRound(t, root, m, r2, srv.URL)
if len(uploads) != 1 || uploads[0].Status != "uploaded" {
t.Fatalf("要 1 筆 uploaded: %+v", uploads)
}
if res.Status != "sent" || res.MarkedCount != 1 || len(res.DroppedPaths) != 0 {
t.Fatalf("要 sentmarked 1: %+v", res)
}
e := m.Entries["a.md"]
if e.IngestedHash != hashOf(content) || e.IngestedAt == 0 {
t.Fatalf("觸發 2xx 後應回寫 ingested_hash: %+v", e)
}
// 伺服器收到的 payload 符合約定
if len(trig.payloads) != 1 {
t.Fatalf("要恰好 1 發觸發: %d", len(trig.payloads))
}
got := trig.payloads[0]
if got.SchemaVersion != 1 || got.FolderID == "" {
t.Fatalf("payload 頭欄位不對: %+v", got)
}
if len(got.Events) != 1 || got.Events[0].Type != "added" || got.Events[0].R2Key == "" {
t.Fatalf("要 1 筆帶 r2_key 的 added: %+v", got.Events)
}
// 下一輪:無變更=零事件、不再觸發
_, _, res2 := syncRound(t, root, m, r2, srv.URL)
if res2.Status != "skipped_no_changes" || len(trig.payloads) != 1 {
t.Fatalf("無變更輪不該再觸發: %+v(觸發數 %d)", res2, len(trig.payloads))
}
}
// 情境 2:觸發失敗(500)——不回寫 ingested_hash,下一輪同檔重發=自然重試;
// 修好後(200)重試成功才回寫。
func TestSyncFailureNoMarkThenRetry(t *testing.T) {
_, _, r2 := newMockR2(t)
srv, trig := newMockTrigger(t)
trig.status = http.StatusInternalServerError
root := t.TempDir()
content := "會先失敗的內容\n"
writeFile(t, root, "b.md", content, baseTime)
m := newTestManifest()
_, _, res := syncRound(t, root, m, r2, srv.URL)
if res.Status != "failed" || res.Error == "" || res.HTTPStatus != 500 {
t.Fatalf("要 failed+錯誤訊息: %+v", res)
}
if m.Entries["b.md"].IngestedHash != "" {
t.Fatal("觸發失敗不得回寫 ingested_hash")
}
// 修好 → 下一輪重發(R2 端 skipped_exists no-op)→ 2xx → 回寫
trig.status = http.StatusOK
sendable, uploads, res2 := syncRound(t, root, m, r2, srv.URL)
if res2.Status != "sent" || res2.MarkedCount != 1 {
t.Fatalf("重試輪應 sentmarked 1: %+v", res2)
}
if len(uploads) != 1 || uploads[0].Status != "skipped_exists" {
t.Fatalf("重試輪 R2 應 no-op: %+v", uploads)
}
if len(sendable.Events) != 1 || sendable.Events[0].Type != "added" {
t.Fatalf("重試輪應重發 added: %+v", sendable.Events)
}
if m.Entries["b.md"].IngestedHash != hashOf(content) {
t.Fatal("重試成功後應回寫 ingested_hash")
}
}
// 情境 3:防呆警告輪(mass_delete_guard)——removed 事件被壓下,但 payload 連同
// warnings 照送(消費端看得到警告、不執行下架);沒有任何回寫。
func TestSyncGuardRoundStillSendsWarnings(t *testing.T) {
_, _, r2 := newMockR2(t)
srv, trig := newMockTrigger(t)
root := t.TempDir()
m := newTestManifest()
// 先建 3 檔並完成一輪成功 sync(全部標 ingested
for _, n := range []string{"a.md", "b.md", "c.md"} {
writeFile(t, root, n, n+" content\n", baseTime)
}
if _, _, res := syncRound(t, root, m, r2, srv.URL); res.Status != "sent" || res.MarkedCount != 3 {
t.Fatalf("前置輪應全成: %+v", res)
}
// 刪 2/367% > 40% 門檻)→ 防呆
os.Remove(root + "/a.md")
os.Remove(root + "/b.md")
sendable, _, res := syncRound(t, root, m, r2, srv.URL)
if res.Status != "sent" {
t.Fatalf("警告輪應照送: %+v", res)
}
if len(sendable.Events) != 0 {
t.Fatalf("防呆輪不得夾帶任何事件(removed 已壓下): %+v", sendable.Events)
}
if len(sendable.Warnings) != 1 || sendable.Warnings[0].Code != "mass_delete_guard" {
t.Fatalf("要 mass_delete_guard 警告: %+v", sendable.Warnings)
}
got := trig.payloads[len(trig.payloads)-1]
if len(got.Warnings) != 1 || got.Warnings[0].Code != "mass_delete_guard" || len(got.Events) != 0 {
t.Fatalf("伺服器端收到的警告輪不對: %+v", got)
}
if res.MarkedCount != 0 {
t.Fatalf("警告輪無內容事件,不該回寫: %+v", res)
}
}
// 情境 4(純函式):上傳失敗的 added/modified 不隨 payload 送出;同路徑的 renamed
// 也不得回寫(否則內容從未上 R2 卻被標 ingested)。
func TestBuildSendableDropsFailedUploads(t *testing.T) {
h1, h2 := hashOf("one"), hashOf("two")
p := &TriggerPayload{
SchemaVersion: 1, FolderID: "f",
Events: []Event{
{Type: "added", Path: "ok.md", SourceHash: h1, R2Key: r2KeyOf(h1)},
{Type: "added", Path: "bad.md", SourceHash: h2, R2Key: r2KeyOf(h2)},
{Type: "renamed", Path: "bad.md", OldPath: "old-bad.md", SourceHash: h2},
{Type: "removed", Path: "gone.md", SourceHash: hashOf("g")},
},
}
uploads := []UploadResult{
{Path: "ok.md", Status: "uploaded"},
{Path: "bad.md", Status: "failed", Error: "mock"},
}
sendable, dropped := BuildSendablePayload(p, uploads)
if len(dropped) != 1 || dropped[0] != "bad.md" {
t.Fatalf("要擋下 bad.md: %v", dropped)
}
types := []string{}
for _, ev := range sendable.Events {
types = append(types, ev.Type+":"+ev.Path)
}
want := "added:ok.md renamed:bad.md removed:gone.md"
if strings.Join(types, " ") != want {
t.Fatalf("送出清單不對:%v(要 %s", types, want)
}
// 回寫:ok.md 回寫;bad.md 的 renamed 因同路徑被擋下也不回寫
m := newTestManifest()
m.Entries["ok.md"] = &ManifestEntry{ContentHash: h1}
m.Entries["bad.md"] = &ManifestEntry{ContentHash: h2}
n := MarkIngestedEvents(m, sendable.Events, dropped, 42)
if n != 1 || m.Entries["ok.md"].IngestedHash != h1 {
t.Fatalf("只該回寫 ok.md: n=%d %+v", n, m.Entries["ok.md"])
}
if m.Entries["bad.md"].IngestedHash != "" {
t.Fatal("上傳失敗檔的 renamed 不得回寫 ingested_hash")
}
}
// 附加:ARCRUN_TRIGGER_URL 缺漏/格式錯,清楚報錯。
func TestLoadTriggerURL(t *testing.T) {
t.Setenv("ARCRUN_TRIGGER_URL", "")
if _, err := LoadTriggerURLFromEnv(); err == nil || !strings.Contains(err.Error(), "ARCRUN_TRIGGER_URL") {
t.Fatalf("缺 env 應報含變數名的錯: %v", err)
}
t.Setenv("ARCRUN_TRIGGER_URL", "not-a-url")
if _, err := LoadTriggerURLFromEnv(); err == nil {
t.Fatal("非完整 URL 應報錯")
}
t.Setenv("ARCRUN_TRIGGER_URL", "https://example.com/webhooks/named/demo/rag_ingest/trigger")
u, err := LoadTriggerURLFromEnv()
if err != nil || u == "" {
t.Fatalf("合法 URL 應通過: %v", err)
}
}