feat(km-wiki-ingest): 重做成跑在 cypher 上的 workflow(走 cypher binding,汰換 service-binding drainer,D28)
把知識庫 ingest 從 standalone drainer(誤用 Service Bindings)重表達為 cypher workflow:
- workflow.yaml(Phase 0 cron drain):watch_cron→load_cursor→list_cards→pick_card(code)
→fetch_card→parse_card(code)→upsert_entry→save_cursor→post_envelopes→post_one_envelope。
線性 pipe,跨-worker 全走 cypher binding(零件節點),零 service binding。
- workflow.delta.yaml(Gitea webhook 穩態):collect_changed(code)→foreach card→fetch/parse/upsert/foreach envelope。
- code/kbdb/graph 接法:code=canonical `code` 零件(arcrun-code);kbdb/graph=http_request 零件打
/entries/ingest、/triplets/ingest(server 端冪等);Gitea=http_request。
- 平台端最小補丁(各需 gated 部署):
1) cypher-executor component-loader:WASM_HTTP_RUNNER_IDS 加 'code'(canonical→arcrun-code,cypher binding 正解)。
2) kbdb base:POST /entries/ingest(page_name+content_hash 冪等 upsert,對稱 graph /triplets/ingest;
因 flow DSL 無資料條件分支,把 create/patch/skip 冪等推到 server 端)。
- drainer/DEPRECATED.md:標舊 standalone worker 退役計畫(新版穩定後 wrangler delete)。
- DEPLOY.md:部署順序、cron/webhook 掛法、subdomain 對齊、驗收與退役。
本輪不部署(待總管/leo 審架構)。
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJiLCRUU2o3aSpPEzVCt2o
This commit is contained in:
@@ -36,6 +36,11 @@ import type { Bindings, ComponentRunner, ServiceBinding } from '../types';
|
||||
const WASM_HTTP_RUNNER_IDS: ReadonlySet<string> = new Set([
|
||||
// 通用 HTTP 零件
|
||||
'http_request',
|
||||
// 通用 code 零件(arcrun-code,QuickJS 沙箱;D27 逃生口)。canonical `code` →
|
||||
// arcrun-code.{WORKER_SUBDOMAIN}.workers.dev。這是「cypher binding」到 code 零件的正解
|
||||
// (workflow 節點以 component: code 引用;不走 service binding,符 D28)。
|
||||
// TODO(同下白名單架構債):改為 registry 動態查,免每加零件都改本檔。
|
||||
'code',
|
||||
// gmail / telegram / line_notify / google_sheets 已降級為 recipe(2026-05-29 Phase 2):
|
||||
// recipe:gmail_send / telegram_send / line_notify_send / google_sheets_read|append
|
||||
// 走 step 6 KV recipe 解析,不再是零件。零件目錄已刪。
|
||||
|
||||
@@ -23,6 +23,68 @@ entryRoutes.post('/', async (c) => {
|
||||
return c.json({ success: true, entry });
|
||||
});
|
||||
|
||||
// POST /entries/ingest — server 端冪等 upsert(page_name 當鍵 + content_hash skip-if-unchanged)。
|
||||
// 語義對稱 graph 的 POST /triplets/ingest:讓「工作流只打一發、冪等由 server 保證」,
|
||||
// 使 km_wiki_ingest_drain cypher workflow 不必在 flow DSL 裡做 lookup→decide→create/patch 分支
|
||||
// (DSL 無資料條件分支)。owner_id 走 query。body 收 parse_card 產的 entry 形狀:
|
||||
// { page_name, entry_type, content, source, tags?(array|json), metadata?(object)|metadata_json?, content_hash? }
|
||||
// 行為:無此 (page_name,owner_id) → create;有且 content_hash 同 → skip(不重嵌);有且不同/無 hash → update+重嵌。
|
||||
// 回:{ success, action:'created'|'updated'|'skipped', entry }。
|
||||
entryRoutes.post('/ingest', async (c) => {
|
||||
const body = await c.req.json().catch(() => null) as Record<string, unknown> | null;
|
||||
if (!body || !body.entry_type || !body.page_name) {
|
||||
return c.json({ success: false, error: 'entry_type 與 page_name 必填' }, 400);
|
||||
}
|
||||
const owner_id = c.req.query('owner_id') || (body.owner_id as string | undefined) || undefined;
|
||||
const page_name = String(body.page_name);
|
||||
|
||||
// 正規化 tags / metadata → *_json 字串(同時容忍呼叫端直接給 *_json)。
|
||||
const tags_json =
|
||||
typeof body.tags_json === 'string' ? body.tags_json
|
||||
: Array.isArray(body.tags) ? JSON.stringify(body.tags)
|
||||
: undefined;
|
||||
const metaObj = (body.metadata && typeof body.metadata === 'object') ? body.metadata as Record<string, unknown> : undefined;
|
||||
const metadata_json =
|
||||
typeof body.metadata_json === 'string' ? body.metadata_json
|
||||
: metaObj ? JSON.stringify(metaObj)
|
||||
: undefined;
|
||||
const newHash =
|
||||
(typeof body.content_hash === 'string' && body.content_hash)
|
||||
|| (metaObj && typeof metaObj.content_hash === 'string' ? metaObj.content_hash : undefined);
|
||||
|
||||
// 查現有(page_name 是冪等鍵;owner_id 隔離租戶)。
|
||||
const { entries } = await listEntries(c.env.DB, { page_name, owner_id, limit: 1 });
|
||||
const existing = entries[0];
|
||||
|
||||
if (existing) {
|
||||
let storedHash: string | undefined;
|
||||
try { storedHash = existing.metadata_json ? (JSON.parse(existing.metadata_json).content_hash as string) : undefined; } catch { /* ignore */ }
|
||||
if (newHash && storedHash && storedHash === newHash) {
|
||||
return c.json({ success: true, action: 'skipped', entry: existing });
|
||||
}
|
||||
const entry = await updateEntry(c.env.DB, existing.id, {
|
||||
content: body.content as string | undefined,
|
||||
...(tags_json !== undefined ? { tags_json } : {}),
|
||||
...(metadata_json !== undefined ? { metadata_json } : {}),
|
||||
});
|
||||
if (embedEnabled(c.env) && body.content !== undefined && entry) {
|
||||
c.executionCtx.waitUntil(embedOnWrite(c.env, entry).catch(() => {}));
|
||||
}
|
||||
return c.json({ success: true, action: 'updated', entry });
|
||||
}
|
||||
|
||||
const entry = await createEntry(c.env.DB, {
|
||||
entry_type: String(body.entry_type),
|
||||
content: body.content as string | undefined,
|
||||
owner_id,
|
||||
page_name,
|
||||
tags_json,
|
||||
metadata_json,
|
||||
});
|
||||
if (embedEnabled(c.env)) c.executionCtx.waitUntil(embedOnWrite(c.env, entry).catch(() => {}));
|
||||
return c.json({ success: true, action: 'created', entry });
|
||||
});
|
||||
|
||||
// GET /entries — list with filters (entry_type, owner_id, parent_id, page_name, source, q/search)
|
||||
// e.g. list workflows under a project: ?parent_id=PROJECT&entry_type=workflow
|
||||
// e.g. get one by idempotency key: ?page_name=skill-rag_with_arcrun
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# km-wiki-ingest — 部署(cypher workflow 版,走 cypher binding)
|
||||
|
||||
> **架構(D28)**:知識庫 ingest =**跑在 cypher-executor 上的 workflow**,跨-worker 呼叫全走
|
||||
> **cypher binding**(零件節點,由 component-loader 解析),**零 Service Bindings**。
|
||||
> 汰換舊的 standalone `arcrun-km-wiki-drainer`(`drainer/`,已誤用 service binding → 見 `drainer/DEPRECATED.md`)。
|
||||
>
|
||||
> **本輪不部署**(等總管/leo 審架構)。以下是過閘後的部署順序。
|
||||
|
||||
## 0) 先決條件(兩個平台端小改,皆本分支已含,需各自 gated 部署)
|
||||
|
||||
這兩處是「讓 workflow 能純走 cypher binding」的最小平台補丁,**不是** ingest 專屬邏輯:
|
||||
|
||||
1. **cypher-executor:`code` 零件進 loader 白名單**
|
||||
- 檔:`cypher-executor/src/lib/component-loader.ts` → `WASM_HTTP_RUNNER_IDS` 加 `'code'`(本分支已加 1 行)。
|
||||
- 效果:workflow 節點 `component: code` → 解析成 `arcrun-code.{WORKER_SUBDOMAIN}.workers.dev`(cypher binding)。
|
||||
- 部署:`cd cypher-executor && npx wrangler deploy`(要 leo 過閘;動到共用 executor)。
|
||||
- **替代(不改 executor)**:把兩個 workflow 裡 `component: code` 改成 code 的完整 URL
|
||||
`component: "https://arcrun-code.<subdomain>.workers.dev"`(走 loader step 2 外部 URL)。省一次 executor 部署,
|
||||
但少了「canonical 零件」的乾淨語義。建議用白名單版(這才是 D28 說的 cypher binding)。
|
||||
|
||||
2. **KBDB base:`POST /entries/ingest`(server 端冪等 upsert)**
|
||||
- 檔:`kbdb/src/routes/entries.ts`(本分支已加)。語義對稱 graph 的 `/triplets/ingest`:
|
||||
page_name 當鍵、`content_hash` 同 → skip、不同/無 → update、無則 create。
|
||||
- 為何需要它:cypher 的 flow DSL **無資料條件分支**(只有 ON_SUCCESS/ON_FAIL/FOREACH),
|
||||
無法在 workflow 內做「查→比對→create 或 patch」三岔。把冪等推到 server 端,workflow 只打一發。
|
||||
- 部署:`cd kbdb && npx wrangler deploy`(要過閘)。
|
||||
|
||||
> ⚠️ **subdomain 對齊**:兩個 workflow 內的 URL 用 `uncle6-me`(官方 `WORKER_SUBDOMAIN`,見 `cypher-executor/wrangler.toml`)。
|
||||
> 若你的 arcrun-kbdb / kbdb-graph-plugin / arcrun-code 部在別的 workers.dev subdomain(例 `leo21c`),
|
||||
> 把兩檔的 URL 一起改。**舊 standalone drainer 誤用 `leo21c` 是坑之一。**
|
||||
|
||||
## 1) CLI 設定
|
||||
|
||||
```bash
|
||||
# self-hosted(namespace 明碼分區):
|
||||
acr config set mode self-hosted
|
||||
acr config set cypher_url https://cypher.arcrun.dev # 或你自架的 executor
|
||||
echo "NAMESPACE=leo" >> .env
|
||||
# standard(平台多租戶):acr init 取 api_key
|
||||
```
|
||||
|
||||
## 2) 設 Gitea token(機密;節點以 {{credential.gitea_token}} 引用)
|
||||
|
||||
```bash
|
||||
acr creds set gitea_token <你的 GITEA_TOKEN> # 讀 Leo/notes 用
|
||||
```
|
||||
> graph 的 `X-Arcrun-API-Key: leo` 是 namespace 字串(非機密),已內嵌節點,不需設 credential。
|
||||
|
||||
## 3) 部署 Phase 0 drain(cron 觸發)
|
||||
|
||||
```bash
|
||||
acr push workflow.yaml
|
||||
```
|
||||
- `watch_cron` 的 `cron_expr: "*/2 * * * *"` 會被 `webhooks-named` 的 `extractCronExpr` 抓出,
|
||||
寫進 `cron-idx:_all`;cypher `scheduled()` 每分鐘比對觸發本 workflow(**不需另設 CF cron**)。
|
||||
- 游標**不需 seed**:首跑 `load_cursor` GET 回空 → `pick_card` 從頭;`save_cursor` 首 tick 自動 create。
|
||||
- 手動補跑一 tick(驗收):`acr run km_wiki_ingest_drain`
|
||||
|
||||
## 4) 部署穩態 delta(Gitea webhook 觸發)
|
||||
|
||||
```bash
|
||||
acr push workflow.delta.yaml
|
||||
```
|
||||
- 取得 trigger URL(self-hosted 例):
|
||||
`https://cypher.arcrun.dev/webhooks/named/leo/km_wiki_ingest_delta/trigger`
|
||||
- Gitea repo `Leo/notes` → **Settings → Webhooks → Add Webhook → Gitea**:
|
||||
- Target URL:上面的 trigger URL
|
||||
- Content-Type:`application/json`
|
||||
- Trigger:Push events
|
||||
- (standard 模式改帶 `X-Arcrun-API-Key` header;self-hosted 免 header,namespace 在 path)
|
||||
- 這是 Gitea→Cloudflare(cypher),**非 GitHub Actions** → 不觸 GitHub flag(D4/D20)。
|
||||
|
||||
## 5) 驗收
|
||||
|
||||
```bash
|
||||
# entry 落地?
|
||||
curl -s "https://arcrun-kbdb.uncle6-me.workers.dev/entries?owner_id=leo&entry_type=wiki_card&limit=5" | jq '.count'
|
||||
# triplet 落地?
|
||||
curl -s "https://kbdb-graph-plugin.uncle6-me.workers.dev/graph?owner_id=leo" | jq '.nodes|length'
|
||||
# 游標?
|
||||
curl -s "https://arcrun-kbdb.uncle6-me.workers.dev/entries?owner_id=leo&page_name=cursor:km_wiki_ingest_drain:Leo/notes" | jq '.entries[0].content'
|
||||
```
|
||||
冪等自證:連跑兩次 `acr run km_wiki_ingest_drain`,第二次同卡 entry `action:skipped`、graph per-source no-op。
|
||||
|
||||
## 6) 舊 standalone drainer 退役
|
||||
|
||||
- 全 drain 完 + delta webhook 穩定收斂後,退役 `arcrun-km-wiki-drainer`:`wrangler delete arcrun-km-wiki-drainer`。
|
||||
- 保留 `drainer/` 原始碼一個週期(含 `DEPRECATED.md`)供對照,確認新版逐卡輸出與舊版一致再刪。
|
||||
- 見 `drainer/DEPRECATED.md`。
|
||||
|
||||
## 節點圖速覽
|
||||
|
||||
**drain(cron)**:`watch_cron(cron)` → `load_cursor(http GET kbdb)` → `list_cards(http GET Gitea tree)`
|
||||
→ `pick_card(code)` → `fetch_card(http GET Gitea raw)` → `parse_card(code)`
|
||||
→ `upsert_entry(http POST kbdb /entries/ingest)` → `save_cursor(http POST kbdb /entries/ingest)`
|
||||
→ `post_envelopes(foreach)` → `post_one_envelope(http POST graph /triplets/ingest)`
|
||||
|
||||
**delta(webhook)**:`collect_changed(code)` →〔對每個 card〕→ `fetch_card_d` → `parse_card_d(code)`
|
||||
→ `upsert_entry_d(http)` → `post_envelopes_d(foreach)` → `post_one_envelope_d(http)`
|
||||
|
||||
零件對應的 cypher binding:`code`→arcrun-code;`http_request`→arcrun-http-request(打 Gitea/kbdb/graph);
|
||||
`cron`→觸發登記;`foreach_control`→平台 logic primitive(唯一允許的 service-binding 用法:零件等級,非本工作流編排)。
|
||||
**全程無工作流等級的 service binding。**
|
||||
@@ -0,0 +1,27 @@
|
||||
# ⛔ DEPRECATED — 此 standalone drainer 架構錯誤(D28),改用 cypher workflow
|
||||
|
||||
**別部署這個。** `arcrun-km-wiki-drainer` 是 standalone Worker,對同帳號 worker 用
|
||||
**Service Bindings**(`SVC_CODE`/`SVC_KBDB`/`SVC_GRAPH`)呼叫 code/kbdb/graph。
|
||||
|
||||
leo 拍板(D28):**Service Bindings 一般禁用**,唯一例外是「零件等級:把幾個 wasm 綁成複合零件」。
|
||||
**工作流/recipe 等級的跨-worker 編排**(ingest 串 code+kbdb+graph 正是)→ **走 cypher binding
|
||||
=跑成 cypher 上的 workflow**。這也是 D27「Arcrun workflow」的本意。
|
||||
|
||||
前一 subagent 因 `workflow.yaml` 當時是 skeleton,就抄捷徑自建這個 standalone worker
|
||||
(worker-to-worker 撞 CF 1042 → 又錯用 service binding 補),一步歪步步歪。
|
||||
|
||||
## 正解(本目錄上一層)
|
||||
- `../workflow.yaml`(Phase 0 cron drain)+ `../workflow.delta.yaml`(Gitea webhook 穩態)
|
||||
=跑在 cypher-executor 上的 workflow,全走 cypher binding,零 service binding。
|
||||
- 部署見 `../DEPLOY.md`。
|
||||
|
||||
## 為什麼 standalone 版當初「看起來能動」卻是錯的
|
||||
- 它把 1042 歸因成「同帳號 worker-to-worker 必須用 service binding」——**錯**。
|
||||
cypher-executor 靠 `global_fetch_strictly_public` compat flag 讓 fetch 走公網前門,
|
||||
sibling worker 呼叫本就通;standalone worker 沒這個 flag 才撞牆,而正解是「別自建 standalone worker,
|
||||
把編排交回 cypher」,不是加 service binding。
|
||||
- 它還把 URL 寫成 `*.leo21c.workers.dev`,與 cypher 的 `WORKER_SUBDOMAIN=uncle6-me` 不一致。
|
||||
|
||||
## 退役步驟
|
||||
新版全 drain 完 + delta 穩定後:`wrangler delete arcrun-km-wiki-drainer`,再刪本 `drainer/` 目錄。
|
||||
保留一個週期供逐卡輸出對照。
|
||||
@@ -0,0 +1,429 @@
|
||||
name: km_wiki_ingest_delta
|
||||
description: >
|
||||
穩態 delta ingest —— **cypher workflow,Gitea push webhook 觸發**(非 cron,非 standalone worker)。
|
||||
只處理本次 commit 動到的 system-dev/wiki/cards/**/*.md(不重掃全庫),量小、天生不撞 subrequest 頂、
|
||||
不需限速。同 drain 走 cypher binding(code 解析、http_request 打 kbdb/graph),零 Service Bindings。
|
||||
觸發=Gitea repo Settings → Webhooks → 指向 cypher 的 named-webhook trigger URL:
|
||||
https://cypher.arcrun.dev/webhooks/named/{namespace}/km_wiki_ingest_delta/trigger
|
||||
(self-hosted:namespace 在 path 明碼;standard:改帶 X-Arcrun-API-Key header。見 DEPLOY.md)
|
||||
⚠️ 這是 Gitea → Cloudflare(cypher),**非 GitHub Actions** → 不觸 GitHub flag 紅線(D4/D20)。
|
||||
|
||||
# 觸發 context = Gitea webhook payload({ commits:[{added,modified,removed}], ... })直接當初始 ctx。
|
||||
# collect_changed(code) 從 payload 濾出動到的卡片 → 外層 FOREACH 逐卡跑
|
||||
# fetch → parse → upsert entry → 內層 FOREACH 逐 envelope POST graph。全鏈冪等(未變檔自動 skip)。
|
||||
|
||||
flow:
|
||||
- "collect_changed >> 對每個 card >> fetch_card_d" # code:payload → 變更卡路徑陣列
|
||||
- "fetch_card_d >> ON_SUCCESS >> parse_card_d" # 抓卡片全文(Gitea raw)
|
||||
- "parse_card_d >> ON_SUCCESS >> upsert_entry_d" # code:卡片 md → entry + envelope
|
||||
- "upsert_entry_d >> ON_SUCCESS >> post_envelopes_d" # 卡片 → base entry(冪等 upsert)
|
||||
- "post_envelopes_d >> 對每個 envelope >> post_one_envelope_d" # 逐段 POST graph /triplets/ingest
|
||||
|
||||
config:
|
||||
# ── 1) 從 webhook payload 收集變更卡片(純函式 code)──
|
||||
# output(→ .data):{ cards:[relPath...], removed:[...] }。removed 卡的 graph 清理未實作(另議)。
|
||||
collect_changed:
|
||||
component: code
|
||||
code: |
|
||||
const root = (input.cards_root || 'system-dev/wiki/cards').replace(/\/$/, '') + '/';
|
||||
const isCard = (p) => typeof p === 'string' && p.startsWith(root) && p.endsWith('.md') && !((p.split('/').pop() || '').startsWith('00-INDEX'));
|
||||
const changed = new Set(), removed = new Set();
|
||||
for (const cm of (input.commits || [])) {
|
||||
for (const p of [...(cm.added || []), ...(cm.modified || [])]) if (isCard(p)) changed.add(p);
|
||||
for (const p of (cm.removed || [])) if (isCard(p)) removed.add(p);
|
||||
}
|
||||
return { cards: [...changed], removed: [...removed] };
|
||||
input:
|
||||
commits: "{{commits}}"
|
||||
cards_root: "system-dev/wiki/cards"
|
||||
limits:
|
||||
timeout_ms: 2000
|
||||
max_output_bytes: 1048576
|
||||
|
||||
# ── 2) 抓卡片全文(外層 FOREACH 注入的 {{card}} = 一條卡路徑)──
|
||||
fetch_card_d:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "https://git.uncle6.me/api/v1/repos/Leo/notes/raw/{{card}}?ref=main"
|
||||
headers:
|
||||
Authorization: "token {{credential.gitea_token}}"
|
||||
|
||||
parse_card_d:
|
||||
component: code
|
||||
code: |
|
||||
// km-wiki-ingest — 機械式卡片→(entry + triplet envelope) 轉換核心(無 LLM、純函式)
|
||||
// ---------------------------------------------------------------------------
|
||||
// 取代舊 `kbdb-ingest-plugin/scripts/ingest-cli.mjs` 的 raw→Haiku 路:
|
||||
// 舊路 = 讀裸筆記 → 呼叫 Haiku 萃 (s,p,o) → envelope(有 LLM、非決定性、耗 token)。
|
||||
// 新路 = 讀「已精耕卡片」(`system-dev/wiki/cards/**/*.md`)→ 直接解析卡片內既有的
|
||||
// `## 實體`(節點)、`## 關聯` 的 typed-edge(`A >> 關係 >> B`)與 `[[wikilink]]`
|
||||
// → entry + triplet envelope。純機械、決定性、零 token。
|
||||
//
|
||||
// 這支=通用 `code` 零件(Arcrun#10,sandbox inline JS)承載的解析邏輯本體。
|
||||
// workflow.yaml 的 parse_card 節點把本檔的 planCard 邏輯內聯進 code 零件的 config
|
||||
// (去 import/export、raw NUL 分隔符改 u0000 escape、改用 code 沙箱注入的 sha256);
|
||||
// 不再鑄 domain 零件 km_wiki_card_parse(Arcrun#10 裁定:一次性解析走通用逃生口)。
|
||||
// 本檔續留作「該內聯 JS 的權威來源 + 可單元測試的參考實作」(純函式、stdin→stdout JSON、無 fs/網路)。
|
||||
//
|
||||
// 對齊契約:kbdb-ingest-plugin/contracts/ingest-candidate.json(envelope 形狀 / 禁止欄位)。
|
||||
// 對齊頂層 SDD:卡片→entry(metadata.embed=true,走 base API)、wikilink→triplet(走 graph)。
|
||||
//
|
||||
// 鐵律:不碰儲存、不算向量、不建表。這支只「產出將寫入什麼」,實際 HTTP 由 workflow 打。
|
||||
|
||||
// (import 移除:code 沙箱提供注入的 sha256 builtin)
|
||||
// --- CF subrequest 預算(防「Too many subrequests by single Worker invocation」,07_01 根因)---
|
||||
//
|
||||
// graph worker 處理一次 POST /triplets/ingest 時,對 base 的每次 fetch = 1 subrequest。
|
||||
// 精確拆帳(讀 kbdb-graph-plugin/src/actions/triplet-ingest.ts + triplet-crud.ts + templates.ts):
|
||||
// ingestEnvelope = ensurePluginTemplates(3) + listRecordsByTemplate(1)
|
||||
// + Σ triplet [ createTriplet → ensurePluginTemplates(3) + createRecord(1) = 4 ]
|
||||
// + persistNodes [ ensurePluginTemplates(3) + Σ node createRecord(1) ]
|
||||
// + Σ deprecated updateRecord(1)
|
||||
// ⟹ subreq(envelope) = 7 + 4*N_triplets + M_nodes + D_deprecated
|
||||
//
|
||||
// 07_01 實測炸點:N=11, M=10, D=0 → 7+44+10 = 61 > 50(CF 免費/bundled 上限)→ 炸半殘。
|
||||
//
|
||||
// 對策 = 「一卡一 tick、每 envelope 壓在預算下、超大檔以 source_uri anchor 分段」。
|
||||
const SUBREQ_CEILING = 50; // CF 單次 Worker invocation subrequest 硬上限(bundled)
|
||||
const SUBREQ_BUDGET = 40; // 我們的目標上限(留 10 給 D_deprecated 等變動)
|
||||
|
||||
/** 精確估算「一個 envelope 打進 graph /triplets/ingest」會在 graph worker 內產生幾個 subrequest。 */
|
||||
function estimateEnvelopeSubrequests(nTriplets, mNodes, dDeprecated = 0) {
|
||||
return 7 + 4 * nTriplets + mNodes + dDeprecated;
|
||||
}
|
||||
|
||||
// --- sha256(content_hash 冪等鍵)---
|
||||
// (sha256 移除:使用 code 沙箱注入的 curated builtin sha256)
|
||||
|
||||
// --- frontmatter 解析(極簡 YAML:只吃我們卡片用到的 tags / gloss / pipeline_candidate)---
|
||||
function parseFrontmatter(md) {
|
||||
const m = md.match(/^---\n([\s\S]*?)\n---\n?/);
|
||||
if (!m) return { data: {}, body: md };
|
||||
const body = md.slice(m[0].length);
|
||||
const data = {};
|
||||
for (const line of m[1].split('\n')) {
|
||||
const kv = line.match(/^([A-Za-z_][\w-]*):\s*(.*)$/);
|
||||
if (!kv) continue;
|
||||
const key = kv[1];
|
||||
let val = kv[2].trim();
|
||||
if (val.startsWith('[') && val.endsWith(']')) {
|
||||
// inline list: [a, b, c]
|
||||
data[key] = val.slice(1, -1).split(',').map((s) => s.trim()).filter(Boolean);
|
||||
} else if (val === 'true' || val === 'false') {
|
||||
data[key] = val === 'true';
|
||||
} else {
|
||||
data[key] = val;
|
||||
}
|
||||
}
|
||||
return { data, body };
|
||||
}
|
||||
|
||||
// --- 取某個 `## 標題` / `### 標題` 區塊的內文(到下一個同級或更高級標題為止)---
|
||||
function sectionBody(md, heading) {
|
||||
// heading 例:'## 實體'、'### 內文知識關係'
|
||||
const level = heading.match(/^#+/)[0].length;
|
||||
const lines = md.split('\n');
|
||||
const out = [];
|
||||
let inSec = false;
|
||||
for (const line of lines) {
|
||||
const h = line.match(/^(#+)\s+(.*)$/);
|
||||
if (h) {
|
||||
const thisLevel = h[1].length;
|
||||
if (inSec) {
|
||||
// 遇到同級或更高級標題 → 區塊結束
|
||||
if (thisLevel <= level) break;
|
||||
}
|
||||
// 標題文字「開頭相符」即算命中(容忍標題後帶括號補述)
|
||||
if (!inSec && thisLevel === level && line.replace(/^#+\s+/, '').startsWith(heading.replace(/^#+\s+/, ''))) {
|
||||
inSec = true;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (inSec) out.push(line);
|
||||
}
|
||||
return out.join('\n');
|
||||
}
|
||||
|
||||
// --- 實體行解析:`- **正規名**(別名1/別名2)— 描述`(別名、描述皆選填)---
|
||||
function parseEntities(md) {
|
||||
const sec = sectionBody(md, '## 實體');
|
||||
const entities = [];
|
||||
for (const raw of sec.split('\n')) {
|
||||
const line = raw.trim();
|
||||
if (!line.startsWith('- ')) continue;
|
||||
if (line.startsWith('- >') || line.startsWith('> ')) continue; // 跳過引言說明行
|
||||
const m = line.match(/^- \*\*(.+?)\*\*(?:((.+?)))?\s*(?:[—–\-]\s*(.*))?$/);
|
||||
if (!m) continue;
|
||||
const name = m[1].trim();
|
||||
if (!name) continue;
|
||||
const aliases = m[2]
|
||||
? m[2].split(/[//、,]/).map((s) => s.trim()).filter((s) => s && s !== name)
|
||||
: [];
|
||||
const gloss = (m[3] || '').trim();
|
||||
entities.push({ name, aliases, gloss });
|
||||
}
|
||||
return entities;
|
||||
}
|
||||
|
||||
// --- typed-edge 行解析:`A >> 謂詞 >> B`(端點可為裸實體名或 [[wikilink]])---
|
||||
function parseTypedEdges(sectionText) {
|
||||
const edges = [];
|
||||
for (const raw of (sectionText || '').split('\n')) {
|
||||
const line = raw.trim();
|
||||
if (!line.startsWith('- ')) continue;
|
||||
const body = line.slice(2).trim();
|
||||
if (body.startsWith('(') || body.startsWith('(')) continue; // 「(暫無…)」占位行
|
||||
const parts = body.split('>>');
|
||||
if (parts.length !== 3) continue;
|
||||
const subject = stripWikilink(parts[0].trim());
|
||||
const predicate = parts[1].trim();
|
||||
const object = stripWikilink(parts[2].trim());
|
||||
if (!subject || !predicate || !object) continue;
|
||||
edges.push({ subject, predicate, object });
|
||||
}
|
||||
return edges;
|
||||
}
|
||||
|
||||
// [[notes/00-INDEX]] → notes/00-INDEX ;純字串則原樣回。
|
||||
function stripWikilink(s) {
|
||||
const m = s.match(/^\[\[(.+?)\]\]$/);
|
||||
return m ? m[1].trim() : s;
|
||||
}
|
||||
|
||||
// --- 抽所有 inline [[wikilink]](含 header 的 ← [[notes/00-INDEX]] 與內文)---
|
||||
function extractInlineWikilinks(md) {
|
||||
const out = [];
|
||||
const re = /\[\[(.+?)\]\]/g;
|
||||
let m;
|
||||
while ((m = re.exec(md)) !== null) out.push(m[1].trim());
|
||||
return out;
|
||||
}
|
||||
|
||||
// --- 卡片 canonical id:以檔名(去副檔名)為準,對齊 `## 卡片關係` 用的 [[基名]] 慣例 ---
|
||||
function cardCanonical(relPath) {
|
||||
const base = relPath.split('/').pop().replace(/\.md$/, '');
|
||||
return base;
|
||||
}
|
||||
|
||||
/**
|
||||
* 解析一張卡片 → { entry, nodes, triplets, meta }(尚未分段的原始產物)。
|
||||
* relPath:卡片相對 repo 根路徑(如 system-dev/wiki/cards/notes/Xxx.md)。
|
||||
* repo:如 'Leo/notes'。
|
||||
*/
|
||||
function parseCard(md, relPath, repo = 'Leo/notes') {
|
||||
const { data: fm } = parseFrontmatter(md);
|
||||
const canonical = cardCanonical(relPath);
|
||||
const titleMatch = md.match(/^#\s+(.+)$/m);
|
||||
const title = titleMatch ? titleMatch[1].trim() : canonical;
|
||||
|
||||
// 1) 節點:## 實體 的正規名 + 別名 + gloss。
|
||||
const entities = parseEntities(md);
|
||||
|
||||
// 2) 邊:內文知識關係(實體↔實體)+ 卡片關係(卡↔卡)+ inline wikilink(卡→卡 導覽/引用)。
|
||||
const intraEdges = parseTypedEdges(sectionBody(md, '### 內文知識關係'))
|
||||
.map((e) => ({ ...e, confidence: 1.0 }));
|
||||
const cardEdges = parseTypedEdges(sectionBody(md, '### 卡片關係'))
|
||||
.map((e) => ({ ...e, confidence: 1.0 }));
|
||||
|
||||
// inline wikilink(← [[notes/00-INDEX]] 等)→ 卡→卡「連結至」邊,去重、排除自環與已被 typed 邊覆蓋者。
|
||||
const typedPairs = new Set(
|
||||
[...cardEdges].map((e) => `${e.subject}\u0000${e.object}`),
|
||||
);
|
||||
const seenRef = new Set();
|
||||
const refEdges = [];
|
||||
for (const target of extractInlineWikilinks(md)) {
|
||||
const t = stripWikilink(target);
|
||||
if (t === canonical || t === title) continue; // 自環
|
||||
if (typedPairs.has(`${canonical}\u0000${t}`)) continue; // 已有明確謂詞邊
|
||||
const key = `${canonical}\u0000${t}`;
|
||||
if (seenRef.has(key)) continue;
|
||||
seenRef.add(key);
|
||||
refEdges.push({ subject: canonical, predicate: '連結至', object: t, confidence: 0.5 });
|
||||
}
|
||||
|
||||
const triplets = [...intraEdges, ...cardEdges, ...refEdges];
|
||||
|
||||
// 3) 節點清單:卡片本身(canonical,帶 frontmatter gloss)+ 內文實體。
|
||||
// 卡對卡邊指到的「別張卡」不在此補 node —— 那張卡自己被 ingest 時會補自己的 node。
|
||||
const nodes = [];
|
||||
const seenNode = new Set();
|
||||
const pushNode = (n) => {
|
||||
const k = n.name.toLowerCase();
|
||||
if (!n.name || seenNode.has(k)) return;
|
||||
seenNode.add(k);
|
||||
nodes.push(n);
|
||||
};
|
||||
pushNode({ name: canonical, gloss: fm.gloss || '', aliases: title && title !== canonical ? [title] : [] });
|
||||
for (const e of entities) pushNode({ name: e.name, gloss: e.gloss, aliases: e.aliases });
|
||||
|
||||
return {
|
||||
entry: {
|
||||
// base POST /entries(或 kbdb_upsert_block)用。metadata.embed=true → 語意可搜。
|
||||
page_name: `wikicard:${repo}/${canonical}`, // idempotency key(穩定)
|
||||
entry_type: 'wiki_card',
|
||||
content: md, // 卡片全文逐字(embed 對象)
|
||||
tags: Array.isArray(fm.tags) ? fm.tags : [],
|
||||
metadata: {
|
||||
embed: true, // ★ base embed 模組讀此旗標
|
||||
source: `gitea:${repo}@${relPath}`,
|
||||
content_hash: sha256(md),
|
||||
kind: 'wiki_card',
|
||||
repo,
|
||||
canonical,
|
||||
pipeline_candidate: fm.pipeline_candidate === true,
|
||||
},
|
||||
},
|
||||
nodes,
|
||||
triplets,
|
||||
meta: { canonical, title, relPath, repo, contentHash: sha256(md) },
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* 把一張卡片的 (nodes, triplets) 打包成「一個或多個」ingest envelope,
|
||||
* 使每個 envelope 打進 graph 後的 subrequest 都 ≤ SUBREQ_BUDGET。
|
||||
*
|
||||
* 分段規則(對應頂層 SDD R4 / issue #8 第4點):
|
||||
* - 單 envelope 夠塞(7+4N+M ≤ budget)→ 不分段,uri = 基 uri(無 anchor)。
|
||||
* - 需分段 → 每段 uri = `<基uri>#seg{NN}`、anchor = `seg{NN}`。
|
||||
* 每段是「獨立 source_uri」→ 各自獨立冪等,繞開 graph 的 per-source content_hash 整包 skip
|
||||
* (否則同 uri 第 2 段起會被 line 65 的 content_hash 命中而整包跳過)。
|
||||
* - 節點只放進「第一個引用到它的段」,跨段不重送(避免 graph persistNodes 重建 entity record)。
|
||||
*/
|
||||
function planEnvelopes(parsed, opts = {}) {
|
||||
const budget = opts.budget ?? SUBREQ_BUDGET;
|
||||
const repo = parsed.meta.repo;
|
||||
const relPath = parsed.meta.relPath;
|
||||
const baseUri = `gitea:${repo}@${relPath}`;
|
||||
const contentHash = parsed.meta.contentHash;
|
||||
const commit = opts.commit;
|
||||
|
||||
const extractor = {
|
||||
model: opts.extractorModel ?? 'mechanical/km-wiki-card-parse@1',
|
||||
tier: 'deep', // 人工精耕卡=deep(決定性、非淺萃)
|
||||
extracted_at: Math.floor(Date.now() / 1000),
|
||||
};
|
||||
|
||||
const nodeByName = new Map(parsed.nodes.map((n) => [n.name, n]));
|
||||
|
||||
// 貪婪打包:逐條 triplet 累進,段成本 = 7 + 4*(段內邊數) + (段內首見節點數)。
|
||||
const segments = [];
|
||||
let cur = null;
|
||||
const startSeg = () => {
|
||||
cur = { triplets: [], nodeNames: new Set() };
|
||||
segments.push(cur);
|
||||
};
|
||||
const segCost = (seg, extraEdges = 0, extraNodes = 0) =>
|
||||
estimateEnvelopeSubrequests(seg.triplets.length + extraEdges, seg.nodeNames.size + extraNodes);
|
||||
|
||||
startSeg();
|
||||
for (const t of parsed.triplets) {
|
||||
// 這條邊會新引入哪些節點(subject/object 命中 nodeByName 且本段尚未收)
|
||||
const cand = [t.subject, t.object].filter(
|
||||
(nm) => nodeByName.has(nm) && !cur.nodeNames.has(nm) && !anySegHas(segments, cur, nm),
|
||||
);
|
||||
// 放得下?(含新增這條邊 + 新引入節點)
|
||||
if (cur.triplets.length > 0 && segCost(cur, 1, cand.length) > budget) {
|
||||
startSeg();
|
||||
}
|
||||
cur.triplets.push(t);
|
||||
for (const nm of [t.subject, t.object]) {
|
||||
if (nodeByName.has(nm) && !anySegHas(segments, null, nm)) cur.nodeNames.add(nm);
|
||||
}
|
||||
}
|
||||
|
||||
const multi = segments.length > 1;
|
||||
const envelopes = segments.map((seg, i) => {
|
||||
const anchor = multi ? `seg${String(i + 1).padStart(2, '0')}` : undefined;
|
||||
const uri = multi ? `${baseUri}#${anchor}` : baseUri;
|
||||
const nodes = [...seg.nodeNames].map((nm) => {
|
||||
const n = nodeByName.get(nm);
|
||||
const out = { name: n.name };
|
||||
if (n.gloss) out.gloss = n.gloss;
|
||||
if (n.aliases && n.aliases.length) out.aliases = n.aliases;
|
||||
out.embed = true;
|
||||
return out;
|
||||
});
|
||||
const source = { uri, content_hash: contentHash };
|
||||
if (anchor) source.anchor = anchor;
|
||||
if (commit) source.commit = commit;
|
||||
return {
|
||||
source,
|
||||
extractor,
|
||||
nodes,
|
||||
triplets: seg.triplets.map((t) => ({
|
||||
subject: t.subject,
|
||||
predicate: t.predicate,
|
||||
object: t.object,
|
||||
confidence: t.confidence ?? 1.0,
|
||||
})),
|
||||
_estSubrequests: estimateEnvelopeSubrequests(seg.triplets.length, seg.nodeNames.size),
|
||||
};
|
||||
});
|
||||
|
||||
// triplets≥1 是契約硬性;無邊的卡不產 envelope(仍會建 entry)。
|
||||
return envelopes.filter((e) => e.triplets.length >= 1);
|
||||
}
|
||||
|
||||
function anySegHas(segments, exclude, name) {
|
||||
for (const s of segments) {
|
||||
if (s === exclude) continue;
|
||||
if (s.nodeNames.has(name)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** 一張卡片 → 完整 ingest 計畫(entry + envelopes)。planCard = parseCard + planEnvelopes。 */
|
||||
function planCard(md, relPath, repo = 'Leo/notes', opts = {}) {
|
||||
const parsed = parseCard(md, relPath, repo);
|
||||
const envelopes = planEnvelopes(parsed, opts);
|
||||
return { entry: parsed.entry, envelopes, meta: parsed.meta, nodeCount: parsed.nodes.length, tripletCount: parsed.triplets.length };
|
||||
}
|
||||
|
||||
// graph /triplets/ingest 是 strict schema,拒絕未知鍵。planEnvelopes 於 envelope 上掛的
|
||||
// 診斷鍵 _estSubrequests 不在 ingest-candidate 契約內 → post 前剝除所有 _-前綴鍵,
|
||||
// 讓 post_one_envelope 的 body_json={{envelope}} 為契約乾淨 payload。
|
||||
// (leo21c 實測:不剝除會回 422 unrecognized_keys "_estSubrequests"。)
|
||||
const __plan = planCard(input.md, input.relPath, input.repo, input.opts || {});
|
||||
__plan.envelopes = __plan.envelopes.map(function (e) {
|
||||
const clean = {};
|
||||
for (const k in e) { if (k.charAt(0) !== '_') clean[k] = e[k]; }
|
||||
return clean;
|
||||
});
|
||||
return __plan;
|
||||
|
||||
input:
|
||||
md: "{{fetch_card_d.data.body}}"
|
||||
relPath: "{{card}}"
|
||||
repo: "Leo/notes"
|
||||
opts:
|
||||
budget: 40
|
||||
limits:
|
||||
timeout_ms: 3000
|
||||
max_output_bytes: 4194304
|
||||
|
||||
# ── 3) 卡片 → base entry(server 端冪等 upsert,同 drain)──
|
||||
upsert_entry_d:
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "https://arcrun-kbdb.uncle6-me.workers.dev/entries/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
body_json: "{{parse_card_d.data.entry}}"
|
||||
|
||||
# ── 4) triplet envelope(可能多段)→ 內層 FOREACH 逐段 POST graph ──
|
||||
post_envelopes_d:
|
||||
component: foreach_control
|
||||
items: "{{parse_card_d.data.envelopes}}"
|
||||
item_key: envelope
|
||||
|
||||
post_one_envelope_d:
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "https://kbdb-graph-plugin.uncle6-me.workers.dev/triplets/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
X-Arcrun-API-Key: "leo"
|
||||
body_json: "{{envelope}}"
|
||||
|
||||
# 部署變數同 workflow.yaml(drain)。差別:無 cron/游標;由 Gitea webhook 觸發 named-webhook trigger。
|
||||
@@ -1,61 +1,121 @@
|
||||
name: km_wiki_ingest_drain
|
||||
description: >
|
||||
Phase 0 限速 drain:cron 每 tick 只處理「一張卡」→ 機械解析成 entry + triplet envelope
|
||||
→ 冪等寫 KBDB(base entry / graph triplet)。反覆跑直到全庫 drain 完。
|
||||
來源=repo 的 system-dev/wiki/cards/**/*.md(人工精耕卡,非裸筆記,無 LLM)。
|
||||
解析由通用 code 零件(sandbox inline JS)承載,不再鑄 domain 零件(Arcrun#10 裁定)。
|
||||
穩態(Gitea push webhook 只處理 delta)見檔尾 §穩態變體。
|
||||
Phase 0 限速 drain(cron 每 tick 一卡)——**跑在 cypher-executor 上的 workflow,全程走 cypher binding,
|
||||
零 Service Bindings**(D28)。來源=Gitea repo 的 system-dev/wiki/cards/**/*.md(人工精耕卡,無 LLM)。
|
||||
卡片 → 通用 `code` 零件機械解析成 entry + triplet envelope → 冪等寫 KBDB base(/entries/ingest)
|
||||
與 graph(/triplets/ingest)。反覆跑直到全庫 drain 完;穩態改走 webhook(見 workflow.delta.yaml)。
|
||||
|
||||
# ── 為什麼「一 tick 一卡」=根治 07_01 的 Too many subrequests ──
|
||||
# graph worker 處理一次 POST /triplets/ingest 的 subrequest = 7 + 4*N_triplets + M_nodes + D_deprecated。
|
||||
# 07_01 炸點:單一 envelope 吞整檔 N=11,M=10 → 61 > 50(CF bundled 上限)→ 半殘。
|
||||
# 對策:① 一卡一 tick(天然小批,notes 卡 ~N4/M5 → est 28~33,穩壓 50 下)
|
||||
# ② code 節點的內聯解析會自動把超大卡以 source_uri anchor 分段(每段獨立冪等)。
|
||||
# ⟹ 任何單一 graph 呼叫都不會再破頂。
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# 架構(D28 訂正):這**不是** standalone drainer worker,是 cypher workflow。
|
||||
# - 跨-worker 呼叫一律走 **cypher binding**=把每個外部呼叫表達成一個「零件節點」,
|
||||
# 由 cypher-executor 的 component-loader 解析執行。**不用 Service Bindings**
|
||||
# (service binding 只准「零件內部把幾個 wasm 綁成複合零件」的零件等級,見 D28)。
|
||||
# - code/kbdb/graph 的接法(皆 cypher binding,非 service binding):
|
||||
# code → canonical 零件 `code`(arcrun-code,QuickJS 沙箱;純函式解析)
|
||||
# kbdb → `http_request` 零件打 {{kbdb_url}}/entries/ingest(base 的 server 端冪等 upsert)
|
||||
# graph → `http_request` 零件打 {{graph_url}}/triplets/ingest(graph 的 server 端 per-source 冪等)
|
||||
# Gitea → `http_request` 零件(外部 API,本就走公網)
|
||||
# - 觸發:cron 零件節點(cypher scheduled() 每分鐘掃 cron-idx 觸發本 workflow)。
|
||||
#
|
||||
# ⚠️ 部署前置(見 DEPLOY.md):
|
||||
# 1) cypher-executor 需能解析 `code` 零件 —— 目前 component-loader 的 WASM_HTTP_RUNNER_IDS
|
||||
# 白名單未含 `code`(本分支已補 1 行;或改用 code 的 workers.dev 完整 URL 當 component)。
|
||||
# 2) KBDB base 需有 POST /entries/ingest(page_name+content_hash 冪等 upsert)——本分支已加,
|
||||
# 與 graph /triplets/ingest 對稱。若不部署它:改用「lookup→decide→create/patch」多節點版(較繁,
|
||||
# 且 flow DSL 無資料條件分支,需靠 ON_SUCCESS/ON_FAIL 兩路 hack,故不建議)。
|
||||
#
|
||||
# ── 為什麼「一 tick 一卡」=根治 07_01 的 Too many subrequests(同原設計,不變)──
|
||||
# graph 處理一次 /triplets/ingest 的 subrequest = 7 + 4*N_triplets + M_nodes + D_deprecated;
|
||||
# 07_01 炸點 N=11,M=10 → 61 > 50。對策:一卡一 tick(天然小批)+ code 節點超大卡以 source_uri anchor 分段。
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
flow:
|
||||
- "watch_cron >> ON_SUCCESS >> pick_next_card"
|
||||
- "pick_next_card >> ON_SUCCESS >> fetch_card"
|
||||
- "fetch_card >> ON_SUCCESS >> parse_card"
|
||||
- "parse_card >> ON_SUCCESS >> upsert_entry" # 卡片 → base entry(embed=true),冪等
|
||||
- "upsert_entry >> ON_SUCCESS >> post_envelopes" # wikilink/typed-edge → graph triplet
|
||||
- "post_envelopes >> 對每個 envelope >> post_one_envelope" # 分段時多段,各段獨立冪等
|
||||
# 線性 pipe(無資料條件分支——flow DSL 不支援 IF 條件,僅 ON_SUCCESS/ON_FAIL/FOREACH)。
|
||||
- "watch_cron >> ON_SUCCESS >> load_cursor" # cron tick 觸發
|
||||
- "load_cursor >> ON_SUCCESS >> list_cards" # 讀游標(kbdb entry,字串 body)
|
||||
- "list_cards >> ON_SUCCESS >> pick_card" # 列卡(Gitea git tree recursive)
|
||||
- "pick_card >> ON_SUCCESS >> fetch_card" # code:游標+tree → 下一張卡(到底回捲)
|
||||
- "fetch_card >> ON_SUCCESS >> parse_card" # 抓卡片全文(Gitea raw)
|
||||
- "parse_card >> ON_SUCCESS >> upsert_entry" # code:卡片 md → entry + envelope(純函式)
|
||||
- "upsert_entry >> ON_SUCCESS >> save_cursor" # 卡片 → base entry(冪等 upsert,embed=true)
|
||||
- "save_cursor >> ON_SUCCESS >> post_envelopes" # 進游標(樂觀;冪等使其僅為掃描指標)
|
||||
- "post_envelopes >> 對每個 envelope >> post_one_envelope" # 逐段 POST graph /triplets/ingest
|
||||
|
||||
config:
|
||||
# 1) 排程 tick:慢推。每 2 分鐘一張卡=限速(Phase 0 唯一需要 rate-limit 之處)。
|
||||
# ── 1) cron 觸發節點 ─────────────────────────────────────────────────────────
|
||||
# webhooks-named 部署時 extractCronExpr 會抓這裡的 cron_expr 註冊進 cron-idx,
|
||||
# cypher scheduled() 每分鐘比對觸發(不需另設 CF cron;cypher 本身已有每分鐘 tick)。
|
||||
watch_cron:
|
||||
component: cron
|
||||
cron_expr: "*/2 * * * *"
|
||||
description: "每 2 分鐘 drain 一張卡(限速慢推,避免 CF 額度與 subrequest 壓力)"
|
||||
|
||||
# 2) 取下一張待處理卡(cursor drain)。用 Gitea contents API 列 cards 目錄 + 一個游標 block
|
||||
# 記「處理到哪」。回傳單一 { rel_path, download_url, content_hash?(git blob sha) }。
|
||||
# 註:list + cursor 的細節可用 http_request(Gitea API) + set/string_ops 組;此處給語意佔位。
|
||||
pick_next_card:
|
||||
# ── 2) 讀游標(kbdb base entry,page_name 當鍵)。回應 body 是字串,交由 pick_card(code) JSON.parse。──
|
||||
load_cursor:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "https://git.uncle6.me/api/v1/repos/{{repo}}/contents/system-dev/wiki/cards?ref={{ref}}"
|
||||
headers:
|
||||
Authorization: "token {{gitea_token}}"
|
||||
Accept: "application/json"
|
||||
# 下游用 filter/set 取「游標之後第一張、且 .md、且非 00-INDEX」的一張。
|
||||
url: "https://arcrun-kbdb.uncle6-me.workers.dev/entries?page_name=cursor:km_wiki_ingest_drain:Leo/notes&owner_id=leo"
|
||||
|
||||
# 3) 抓卡片全文(Gitea raw)。
|
||||
# ── 3) 列卡(Gitea git tree recursive)。Authorization 走 credential 注入(見 DEPLOY §creds)。──
|
||||
list_cards:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "https://git.uncle6.me/api/v1/repos/Leo/notes/git/trees/main?recursive=true"
|
||||
headers:
|
||||
Authorization: "token {{credential.gitea_token}}"
|
||||
Accept: "application/json"
|
||||
|
||||
# ── 4) 挑下一張卡(純函式 code 零件)──
|
||||
# input:游標 body + tree body + cards_root。output(→ .data):
|
||||
# { rel_path, cursor_content_json, total, wrapped, has_card }
|
||||
# 到底回捲(continuous drain);靠 entry/graph 冪等,未改卡 cheap skip。
|
||||
# repo 內零卡才 throw(→ 節點 success:false → 下游 ON_SUCCESS 不觸發,該 tick no-op)。
|
||||
pick_card:
|
||||
component: code
|
||||
code: |
|
||||
const root = (input.cards_root || 'system-dev/wiki/cards').replace(/\/$/, '') + '/';
|
||||
let last_path = '', cycle = 0;
|
||||
try {
|
||||
const cj = JSON.parse(input.cursor_body || '{}');
|
||||
const e = (cj.entries || [])[0];
|
||||
if (e && e.content) { const s = JSON.parse(e.content); last_path = s.last_path || ''; cycle = s.cycle || 0; }
|
||||
} catch (_) { /* 游標缺/壞 → 從頭 */ }
|
||||
let tree = {};
|
||||
try { tree = JSON.parse(input.tree_body || '{}'); } catch (_) {}
|
||||
const cards = (tree.tree || [])
|
||||
.filter((x) => x.type === 'blob' && x.path.startsWith(root) && x.path.endsWith('.md'))
|
||||
.filter((x) => { const b = x.path.split('/').pop() || ''; return b !== '.gitkeep' && !b.startsWith('00-INDEX'); })
|
||||
.map((x) => x.path)
|
||||
.sort();
|
||||
if (cards.length === 0) throw new Error('no cards under ' + root + '(repo 空或路徑錯)');
|
||||
let idx = cards.findIndex((p) => p > last_path);
|
||||
let wrapped = false;
|
||||
if (idx < 0) { idx = 0; wrapped = true; }
|
||||
const rel_path = cards[idx];
|
||||
return {
|
||||
rel_path,
|
||||
total: cards.length,
|
||||
wrapped,
|
||||
truncated: tree.truncated === true,
|
||||
has_card: true,
|
||||
cursor_content_json: JSON.stringify({ last_path: rel_path, cycle: cycle + (wrapped ? 1 : 0), processed_at: Math.floor(Date.now() / 1000) }),
|
||||
};
|
||||
input:
|
||||
cursor_body: "{{load_cursor.data.body}}"
|
||||
tree_body: "{{list_cards.data.body}}"
|
||||
cards_root: "system-dev/wiki/cards"
|
||||
limits:
|
||||
timeout_ms: 3000
|
||||
max_output_bytes: 2097152
|
||||
|
||||
# ── 5) 抓卡片全文(Gitea raw)。parse_card 讀 {{fetch_card.data.body}}。──
|
||||
fetch_card:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "{{pick_next_card.next.download_url}}"
|
||||
url: "https://git.uncle6.me/api/v1/repos/Leo/notes/raw/{{pick_card.data.rel_path}}?ref=main"
|
||||
headers:
|
||||
Authorization: "token {{gitea_token}}"
|
||||
Authorization: "token {{credential.gitea_token}}"
|
||||
|
||||
# 4) ★ 機械解析 —— 通用 code 零件(sandbox inline JS,無 LLM、無 fs/網路,stdin→stdout JSON)。
|
||||
# Arcrun#10 裁定:一次性解析邏輯走通用逃生口,不再鑄 domain 零件 km_wiki_card_parse。
|
||||
# 下面 code: 內聯的即 lib/card-to-envelope.mjs 的 planCard 邏輯(去 import/export、
|
||||
# raw NUL 分隔符改 \u0000 escape、改用 code 沙箱注入的 curated builtin sha256;
|
||||
# 已單測證明與原模組輸出逐欄全等)。
|
||||
# input:卡片全文 md + 相對路徑 relPath + repo + opts.budget(subrequest 目標上限)。
|
||||
# output:{ success:true, data:{ entry, envelopes[], meta, nodeCount, tripletCount } }
|
||||
# —— envelope 已分段、已估 subrequest。故下游改引用 parse_card.data.*。
|
||||
parse_card:
|
||||
component: code
|
||||
code: |
|
||||
@@ -402,7 +462,7 @@ config:
|
||||
|
||||
input:
|
||||
md: "{{fetch_card.data.body}}"
|
||||
relPath: "{{pick_next_card.next.rel_path}}"
|
||||
relPath: "{{pick_card.data.rel_path}}"
|
||||
repo: "{{repo}}"
|
||||
opts:
|
||||
budget: 40 # subrequest 目標上限(留 10 給 D_deprecated),超過自動 anchor 分段
|
||||
@@ -410,22 +470,37 @@ config:
|
||||
timeout_ms: 3000 # 純 CPU 解析;大卡也充裕
|
||||
max_output_bytes: 4194304 # envelope 陣列可能較大(4 MiB)
|
||||
|
||||
# 5) 卡片 → base entry,冪等 upsert(page_name 當鍵;找到 PATCH、沒有 POST)。
|
||||
# metadata.embed=true → base embed 模組會補嵌 → 語意可搜。
|
||||
# ── 6) 卡片 → base entry:server 端冪等 upsert(page_name 當鍵 + content_hash skip-if-unchanged)。──
|
||||
# 走 http_request 打 KBDB base 的 POST /entries/ingest(本分支新增;語義對稱 graph /triplets/ingest)。
|
||||
# owner_id 走 query;body_json = parse_card 產的 entry 物件(含 metadata.content_hash / tags / embed:true)。
|
||||
# 端點行為:無此 page_name → create;有且 content_hash 同 → skip(不重嵌);有且不同 → update + 重嵌。
|
||||
upsert_entry:
|
||||
component: kbdb_upsert_block
|
||||
api_key: "{{kbdb_api_key}}"
|
||||
kbdb_url: "{{kbdb_url}}"
|
||||
page_name: "{{parse_card.data.entry.page_name}}"
|
||||
type: "{{parse_card.data.entry.entry_type}}"
|
||||
content: "{{parse_card.data.entry.content}}"
|
||||
source: "{{parse_card.data.entry.metadata.source}}"
|
||||
tags_json: "{{parse_card.data.entry.tags_json}}"
|
||||
# ⚠️ metadata.embed=true / content_hash 需經 base /entries 帶 metadata_json 落地;
|
||||
# 若 kbdb_upsert_block 尚未透傳 metadata_json,改用 http_request 直打 base POST/PATCH /entries
|
||||
# 帶 body_json.metadata_json(見 description.md §entry 冪等)。
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "https://arcrun-kbdb.uncle6-me.workers.dev/entries/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
body_json: "{{parse_card.data.entry}}"
|
||||
|
||||
# 6) triplet envelope(可能多段)→ 逐段 POST graph /triplets/ingest。
|
||||
# ── 7) 進游標(同 /entries/ingest upsert;cursor 專用 page_name,content=pick_card 產的新游標 JSON 字串)。──
|
||||
# 無 content_hash → 端點一律 update(游標每 tick 都變)。樂觀進游標:即使後續 envelope 失敗,
|
||||
# 靠 graph 冪等 + 回捲重掃自癒,故游標僅為「掃到哪」的加速指標,非正確性關鍵。
|
||||
save_cursor:
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "https://arcrun-kbdb.uncle6-me.workers.dev/entries/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
body_json:
|
||||
entry_type: "ingest_cursor"
|
||||
page_name: "cursor:km_wiki_ingest_drain:Leo/notes"
|
||||
content: "{{pick_card.data.cursor_content_json}}"
|
||||
source: "gitea:Leo/notes"
|
||||
metadata:
|
||||
kind: "ingest_cursor"
|
||||
embed: false
|
||||
|
||||
# ── 8) triplet envelope(可能多段)→ 逐段 POST graph /triplets/ingest。──
|
||||
# graph 端 per-source(uri+content_hash) 冪等:同 hash 整包 no-op;分段各段 uri 不同 → 各自獨立冪等。
|
||||
post_envelopes:
|
||||
component: foreach_control
|
||||
@@ -435,24 +510,19 @@ config:
|
||||
post_one_envelope:
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "{{graph_url}}/triplets/ingest"
|
||||
url: "https://kbdb-graph-plugin.uncle6-me.workers.dev/triplets/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
X-Arcrun-API-Key: "{{graph_api_key}}"
|
||||
body_json: "{{envelope}}" # envelope 已符合 ingest-candidate.json 契約(禁止欄位已排除)
|
||||
X-Arcrun-API-Key: "leo"
|
||||
body_json: "{{envelope}}" # envelope 已符合 ingest-candidate.json 契約(_-前綴診斷鍵已由 code 節點剝除)
|
||||
|
||||
# ── 執行環境變數(部署時注入;此檔不放密鑰)──
|
||||
# repo=Leo/notes ref=main gitea_token=<GITEA_TOKEN>
|
||||
# kbdb_url=https://arcrun-kbdb.leo21c.workers.dev kbdb_api_key=<partner key>
|
||||
# graph_url=<graph plugin base url on leo21c> graph_api_key=leo
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# §穩態變體(km_wiki_ingest_delta):Gitea push webhook → 只處理 delta 檔
|
||||
# 部署變數(皆已內嵌節點 data;此處僅列出以便 self-hosted fork 修改):
|
||||
# repo=Leo/notes ref=main owner=leo cards_root=system-dev/wiki/cards
|
||||
# kbdb_url = https://arcrun-kbdb.uncle6-me.workers.dev (WORKER_SUBDOMAIN=uncle6-me,見 cypher wrangler)
|
||||
# graph_url = https://kbdb-graph-plugin.uncle6-me.workers.dev
|
||||
# graph_api_key = leo(namespace 字串,非機密)
|
||||
# 機密:gitea_token → acr creds set gitea_token(節點以 {{credential.gitea_token}} 引用,部署時注入)。
|
||||
# ⚠️ WORKER_SUBDOMAIN 須與實際部署帳號一致:官方=uncle6-me;若你的 kbdb/graph 部在別的
|
||||
# workers.dev subdomain(例 leo21c),把上面 URL 一起改。舊 standalone drainer 誤用 leo21c。
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# 觸發 = Gitea repo Settings → Webhooks → 指向 arcrun(cypher-executor) 的 workflow webhook URL。
|
||||
# ⚠️ 這是 Gitea → Cloudflare(arcrun),非 GitHub Actions → 不觸 GitHub flag 紅線(D4/D20)。
|
||||
# 只把上面 flow 的 watch_cron/pick_next_card 換成:
|
||||
# input(webhook payload)>> ON_SUCCESS >> collect_changed
|
||||
# collect_changed = 從 payload.commits[].{added,modified} 濾出 system-dev/wiki/cards/**/*.md
|
||||
# >> foreach 檔 >> fetch_card >> parse_card >> upsert_entry >> post_envelopes(同上)
|
||||
# 量小、天生不撞頂、不需限速;靠 graph/entry 冪等自動 skip 未變檔。
|
||||
|
||||
Reference in New Issue
Block a user