feat(km-wiki-ingest): 重做成跑在 cypher 上的 workflow(走 cypher binding,汰換 service-binding drainer,D28)
把知識庫 ingest 從 standalone drainer(誤用 Service Bindings)重表達為 cypher workflow:
- workflow.yaml(Phase 0 cron drain):watch_cron→load_cursor→list_cards→pick_card(code)
→fetch_card→parse_card(code)→upsert_entry→save_cursor→post_envelopes→post_one_envelope。
線性 pipe,跨-worker 全走 cypher binding(零件節點),零 service binding。
- workflow.delta.yaml(Gitea webhook 穩態):collect_changed(code)→foreach card→fetch/parse/upsert/foreach envelope。
- code/kbdb/graph 接法:code=canonical `code` 零件(arcrun-code);kbdb/graph=http_request 零件打
/entries/ingest、/triplets/ingest(server 端冪等);Gitea=http_request。
- 平台端最小補丁(各需 gated 部署):
1) cypher-executor component-loader:WASM_HTTP_RUNNER_IDS 加 'code'(canonical→arcrun-code,cypher binding 正解)。
2) kbdb base:POST /entries/ingest(page_name+content_hash 冪等 upsert,對稱 graph /triplets/ingest;
因 flow DSL 無資料條件分支,把 create/patch/skip 冪等推到 server 端)。
- drainer/DEPRECATED.md:標舊 standalone worker 退役計畫(新版穩定後 wrangler delete)。
- DEPLOY.md:部署順序、cron/webhook 掛法、subdomain 對齊、驗收與退役。
本輪不部署(待總管/leo 審架構)。
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJiLCRUU2o3aSpPEzVCt2o
This commit is contained in:
@@ -1,61 +1,121 @@
|
||||
name: km_wiki_ingest_drain
|
||||
description: >
|
||||
Phase 0 限速 drain:cron 每 tick 只處理「一張卡」→ 機械解析成 entry + triplet envelope
|
||||
→ 冪等寫 KBDB(base entry / graph triplet)。反覆跑直到全庫 drain 完。
|
||||
來源=repo 的 system-dev/wiki/cards/**/*.md(人工精耕卡,非裸筆記,無 LLM)。
|
||||
解析由通用 code 零件(sandbox inline JS)承載,不再鑄 domain 零件(Arcrun#10 裁定)。
|
||||
穩態(Gitea push webhook 只處理 delta)見檔尾 §穩態變體。
|
||||
Phase 0 限速 drain(cron 每 tick 一卡)——**跑在 cypher-executor 上的 workflow,全程走 cypher binding,
|
||||
零 Service Bindings**(D28)。來源=Gitea repo 的 system-dev/wiki/cards/**/*.md(人工精耕卡,無 LLM)。
|
||||
卡片 → 通用 `code` 零件機械解析成 entry + triplet envelope → 冪等寫 KBDB base(/entries/ingest)
|
||||
與 graph(/triplets/ingest)。反覆跑直到全庫 drain 完;穩態改走 webhook(見 workflow.delta.yaml)。
|
||||
|
||||
# ── 為什麼「一 tick 一卡」=根治 07_01 的 Too many subrequests ──
|
||||
# graph worker 處理一次 POST /triplets/ingest 的 subrequest = 7 + 4*N_triplets + M_nodes + D_deprecated。
|
||||
# 07_01 炸點:單一 envelope 吞整檔 N=11,M=10 → 61 > 50(CF bundled 上限)→ 半殘。
|
||||
# 對策:① 一卡一 tick(天然小批,notes 卡 ~N4/M5 → est 28~33,穩壓 50 下)
|
||||
# ② code 節點的內聯解析會自動把超大卡以 source_uri anchor 分段(每段獨立冪等)。
|
||||
# ⟹ 任何單一 graph 呼叫都不會再破頂。
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# 架構(D28 訂正):這**不是** standalone drainer worker,是 cypher workflow。
|
||||
# - 跨-worker 呼叫一律走 **cypher binding**=把每個外部呼叫表達成一個「零件節點」,
|
||||
# 由 cypher-executor 的 component-loader 解析執行。**不用 Service Bindings**
|
||||
# (service binding 只准「零件內部把幾個 wasm 綁成複合零件」的零件等級,見 D28)。
|
||||
# - code/kbdb/graph 的接法(皆 cypher binding,非 service binding):
|
||||
# code → canonical 零件 `code`(arcrun-code,QuickJS 沙箱;純函式解析)
|
||||
# kbdb → `http_request` 零件打 {{kbdb_url}}/entries/ingest(base 的 server 端冪等 upsert)
|
||||
# graph → `http_request` 零件打 {{graph_url}}/triplets/ingest(graph 的 server 端 per-source 冪等)
|
||||
# Gitea → `http_request` 零件(外部 API,本就走公網)
|
||||
# - 觸發:cron 零件節點(cypher scheduled() 每分鐘掃 cron-idx 觸發本 workflow)。
|
||||
#
|
||||
# ⚠️ 部署前置(見 DEPLOY.md):
|
||||
# 1) cypher-executor 需能解析 `code` 零件 —— 目前 component-loader 的 WASM_HTTP_RUNNER_IDS
|
||||
# 白名單未含 `code`(本分支已補 1 行;或改用 code 的 workers.dev 完整 URL 當 component)。
|
||||
# 2) KBDB base 需有 POST /entries/ingest(page_name+content_hash 冪等 upsert)——本分支已加,
|
||||
# 與 graph /triplets/ingest 對稱。若不部署它:改用「lookup→decide→create/patch」多節點版(較繁,
|
||||
# 且 flow DSL 無資料條件分支,需靠 ON_SUCCESS/ON_FAIL 兩路 hack,故不建議)。
|
||||
#
|
||||
# ── 為什麼「一 tick 一卡」=根治 07_01 的 Too many subrequests(同原設計,不變)──
|
||||
# graph 處理一次 /triplets/ingest 的 subrequest = 7 + 4*N_triplets + M_nodes + D_deprecated;
|
||||
# 07_01 炸點 N=11,M=10 → 61 > 50。對策:一卡一 tick(天然小批)+ code 節點超大卡以 source_uri anchor 分段。
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
flow:
|
||||
- "watch_cron >> ON_SUCCESS >> pick_next_card"
|
||||
- "pick_next_card >> ON_SUCCESS >> fetch_card"
|
||||
- "fetch_card >> ON_SUCCESS >> parse_card"
|
||||
- "parse_card >> ON_SUCCESS >> upsert_entry" # 卡片 → base entry(embed=true),冪等
|
||||
- "upsert_entry >> ON_SUCCESS >> post_envelopes" # wikilink/typed-edge → graph triplet
|
||||
- "post_envelopes >> 對每個 envelope >> post_one_envelope" # 分段時多段,各段獨立冪等
|
||||
# 線性 pipe(無資料條件分支——flow DSL 不支援 IF 條件,僅 ON_SUCCESS/ON_FAIL/FOREACH)。
|
||||
- "watch_cron >> ON_SUCCESS >> load_cursor" # cron tick 觸發
|
||||
- "load_cursor >> ON_SUCCESS >> list_cards" # 讀游標(kbdb entry,字串 body)
|
||||
- "list_cards >> ON_SUCCESS >> pick_card" # 列卡(Gitea git tree recursive)
|
||||
- "pick_card >> ON_SUCCESS >> fetch_card" # code:游標+tree → 下一張卡(到底回捲)
|
||||
- "fetch_card >> ON_SUCCESS >> parse_card" # 抓卡片全文(Gitea raw)
|
||||
- "parse_card >> ON_SUCCESS >> upsert_entry" # code:卡片 md → entry + envelope(純函式)
|
||||
- "upsert_entry >> ON_SUCCESS >> save_cursor" # 卡片 → base entry(冪等 upsert,embed=true)
|
||||
- "save_cursor >> ON_SUCCESS >> post_envelopes" # 進游標(樂觀;冪等使其僅為掃描指標)
|
||||
- "post_envelopes >> 對每個 envelope >> post_one_envelope" # 逐段 POST graph /triplets/ingest
|
||||
|
||||
config:
|
||||
# 1) 排程 tick:慢推。每 2 分鐘一張卡=限速(Phase 0 唯一需要 rate-limit 之處)。
|
||||
# ── 1) cron 觸發節點 ─────────────────────────────────────────────────────────
|
||||
# webhooks-named 部署時 extractCronExpr 會抓這裡的 cron_expr 註冊進 cron-idx,
|
||||
# cypher scheduled() 每分鐘比對觸發(不需另設 CF cron;cypher 本身已有每分鐘 tick)。
|
||||
watch_cron:
|
||||
component: cron
|
||||
cron_expr: "*/2 * * * *"
|
||||
description: "每 2 分鐘 drain 一張卡(限速慢推,避免 CF 額度與 subrequest 壓力)"
|
||||
|
||||
# 2) 取下一張待處理卡(cursor drain)。用 Gitea contents API 列 cards 目錄 + 一個游標 block
|
||||
# 記「處理到哪」。回傳單一 { rel_path, download_url, content_hash?(git blob sha) }。
|
||||
# 註:list + cursor 的細節可用 http_request(Gitea API) + set/string_ops 組;此處給語意佔位。
|
||||
pick_next_card:
|
||||
# ── 2) 讀游標(kbdb base entry,page_name 當鍵)。回應 body 是字串,交由 pick_card(code) JSON.parse。──
|
||||
load_cursor:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "https://git.uncle6.me/api/v1/repos/{{repo}}/contents/system-dev/wiki/cards?ref={{ref}}"
|
||||
headers:
|
||||
Authorization: "token {{gitea_token}}"
|
||||
Accept: "application/json"
|
||||
# 下游用 filter/set 取「游標之後第一張、且 .md、且非 00-INDEX」的一張。
|
||||
url: "https://arcrun-kbdb.uncle6-me.workers.dev/entries?page_name=cursor:km_wiki_ingest_drain:Leo/notes&owner_id=leo"
|
||||
|
||||
# 3) 抓卡片全文(Gitea raw)。
|
||||
# ── 3) 列卡(Gitea git tree recursive)。Authorization 走 credential 注入(見 DEPLOY §creds)。──
|
||||
list_cards:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "https://git.uncle6.me/api/v1/repos/Leo/notes/git/trees/main?recursive=true"
|
||||
headers:
|
||||
Authorization: "token {{credential.gitea_token}}"
|
||||
Accept: "application/json"
|
||||
|
||||
# ── 4) 挑下一張卡(純函式 code 零件)──
|
||||
# input:游標 body + tree body + cards_root。output(→ .data):
|
||||
# { rel_path, cursor_content_json, total, wrapped, has_card }
|
||||
# 到底回捲(continuous drain);靠 entry/graph 冪等,未改卡 cheap skip。
|
||||
# repo 內零卡才 throw(→ 節點 success:false → 下游 ON_SUCCESS 不觸發,該 tick no-op)。
|
||||
pick_card:
|
||||
component: code
|
||||
code: |
|
||||
const root = (input.cards_root || 'system-dev/wiki/cards').replace(/\/$/, '') + '/';
|
||||
let last_path = '', cycle = 0;
|
||||
try {
|
||||
const cj = JSON.parse(input.cursor_body || '{}');
|
||||
const e = (cj.entries || [])[0];
|
||||
if (e && e.content) { const s = JSON.parse(e.content); last_path = s.last_path || ''; cycle = s.cycle || 0; }
|
||||
} catch (_) { /* 游標缺/壞 → 從頭 */ }
|
||||
let tree = {};
|
||||
try { tree = JSON.parse(input.tree_body || '{}'); } catch (_) {}
|
||||
const cards = (tree.tree || [])
|
||||
.filter((x) => x.type === 'blob' && x.path.startsWith(root) && x.path.endsWith('.md'))
|
||||
.filter((x) => { const b = x.path.split('/').pop() || ''; return b !== '.gitkeep' && !b.startsWith('00-INDEX'); })
|
||||
.map((x) => x.path)
|
||||
.sort();
|
||||
if (cards.length === 0) throw new Error('no cards under ' + root + '(repo 空或路徑錯)');
|
||||
let idx = cards.findIndex((p) => p > last_path);
|
||||
let wrapped = false;
|
||||
if (idx < 0) { idx = 0; wrapped = true; }
|
||||
const rel_path = cards[idx];
|
||||
return {
|
||||
rel_path,
|
||||
total: cards.length,
|
||||
wrapped,
|
||||
truncated: tree.truncated === true,
|
||||
has_card: true,
|
||||
cursor_content_json: JSON.stringify({ last_path: rel_path, cycle: cycle + (wrapped ? 1 : 0), processed_at: Math.floor(Date.now() / 1000) }),
|
||||
};
|
||||
input:
|
||||
cursor_body: "{{load_cursor.data.body}}"
|
||||
tree_body: "{{list_cards.data.body}}"
|
||||
cards_root: "system-dev/wiki/cards"
|
||||
limits:
|
||||
timeout_ms: 3000
|
||||
max_output_bytes: 2097152
|
||||
|
||||
# ── 5) 抓卡片全文(Gitea raw)。parse_card 讀 {{fetch_card.data.body}}。──
|
||||
fetch_card:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "{{pick_next_card.next.download_url}}"
|
||||
url: "https://git.uncle6.me/api/v1/repos/Leo/notes/raw/{{pick_card.data.rel_path}}?ref=main"
|
||||
headers:
|
||||
Authorization: "token {{gitea_token}}"
|
||||
Authorization: "token {{credential.gitea_token}}"
|
||||
|
||||
# 4) ★ 機械解析 —— 通用 code 零件(sandbox inline JS,無 LLM、無 fs/網路,stdin→stdout JSON)。
|
||||
# Arcrun#10 裁定:一次性解析邏輯走通用逃生口,不再鑄 domain 零件 km_wiki_card_parse。
|
||||
# 下面 code: 內聯的即 lib/card-to-envelope.mjs 的 planCard 邏輯(去 import/export、
|
||||
# raw NUL 分隔符改 \u0000 escape、改用 code 沙箱注入的 curated builtin sha256;
|
||||
# 已單測證明與原模組輸出逐欄全等)。
|
||||
# input:卡片全文 md + 相對路徑 relPath + repo + opts.budget(subrequest 目標上限)。
|
||||
# output:{ success:true, data:{ entry, envelopes[], meta, nodeCount, tripletCount } }
|
||||
# —— envelope 已分段、已估 subrequest。故下游改引用 parse_card.data.*。
|
||||
parse_card:
|
||||
component: code
|
||||
code: |
|
||||
@@ -402,7 +462,7 @@ config:
|
||||
|
||||
input:
|
||||
md: "{{fetch_card.data.body}}"
|
||||
relPath: "{{pick_next_card.next.rel_path}}"
|
||||
relPath: "{{pick_card.data.rel_path}}"
|
||||
repo: "{{repo}}"
|
||||
opts:
|
||||
budget: 40 # subrequest 目標上限(留 10 給 D_deprecated),超過自動 anchor 分段
|
||||
@@ -410,22 +470,37 @@ config:
|
||||
timeout_ms: 3000 # 純 CPU 解析;大卡也充裕
|
||||
max_output_bytes: 4194304 # envelope 陣列可能較大(4 MiB)
|
||||
|
||||
# 5) 卡片 → base entry,冪等 upsert(page_name 當鍵;找到 PATCH、沒有 POST)。
|
||||
# metadata.embed=true → base embed 模組會補嵌 → 語意可搜。
|
||||
# ── 6) 卡片 → base entry:server 端冪等 upsert(page_name 當鍵 + content_hash skip-if-unchanged)。──
|
||||
# 走 http_request 打 KBDB base 的 POST /entries/ingest(本分支新增;語義對稱 graph /triplets/ingest)。
|
||||
# owner_id 走 query;body_json = parse_card 產的 entry 物件(含 metadata.content_hash / tags / embed:true)。
|
||||
# 端點行為:無此 page_name → create;有且 content_hash 同 → skip(不重嵌);有且不同 → update + 重嵌。
|
||||
upsert_entry:
|
||||
component: kbdb_upsert_block
|
||||
api_key: "{{kbdb_api_key}}"
|
||||
kbdb_url: "{{kbdb_url}}"
|
||||
page_name: "{{parse_card.data.entry.page_name}}"
|
||||
type: "{{parse_card.data.entry.entry_type}}"
|
||||
content: "{{parse_card.data.entry.content}}"
|
||||
source: "{{parse_card.data.entry.metadata.source}}"
|
||||
tags_json: "{{parse_card.data.entry.tags_json}}"
|
||||
# ⚠️ metadata.embed=true / content_hash 需經 base /entries 帶 metadata_json 落地;
|
||||
# 若 kbdb_upsert_block 尚未透傳 metadata_json,改用 http_request 直打 base POST/PATCH /entries
|
||||
# 帶 body_json.metadata_json(見 description.md §entry 冪等)。
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "https://arcrun-kbdb.uncle6-me.workers.dev/entries/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
body_json: "{{parse_card.data.entry}}"
|
||||
|
||||
# 6) triplet envelope(可能多段)→ 逐段 POST graph /triplets/ingest。
|
||||
# ── 7) 進游標(同 /entries/ingest upsert;cursor 專用 page_name,content=pick_card 產的新游標 JSON 字串)。──
|
||||
# 無 content_hash → 端點一律 update(游標每 tick 都變)。樂觀進游標:即使後續 envelope 失敗,
|
||||
# 靠 graph 冪等 + 回捲重掃自癒,故游標僅為「掃到哪」的加速指標,非正確性關鍵。
|
||||
save_cursor:
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "https://arcrun-kbdb.uncle6-me.workers.dev/entries/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
body_json:
|
||||
entry_type: "ingest_cursor"
|
||||
page_name: "cursor:km_wiki_ingest_drain:Leo/notes"
|
||||
content: "{{pick_card.data.cursor_content_json}}"
|
||||
source: "gitea:Leo/notes"
|
||||
metadata:
|
||||
kind: "ingest_cursor"
|
||||
embed: false
|
||||
|
||||
# ── 8) triplet envelope(可能多段)→ 逐段 POST graph /triplets/ingest。──
|
||||
# graph 端 per-source(uri+content_hash) 冪等:同 hash 整包 no-op;分段各段 uri 不同 → 各自獨立冪等。
|
||||
post_envelopes:
|
||||
component: foreach_control
|
||||
@@ -435,24 +510,19 @@ config:
|
||||
post_one_envelope:
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "{{graph_url}}/triplets/ingest"
|
||||
url: "https://kbdb-graph-plugin.uncle6-me.workers.dev/triplets/ingest?owner_id=leo"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
X-Arcrun-API-Key: "{{graph_api_key}}"
|
||||
body_json: "{{envelope}}" # envelope 已符合 ingest-candidate.json 契約(禁止欄位已排除)
|
||||
X-Arcrun-API-Key: "leo"
|
||||
body_json: "{{envelope}}" # envelope 已符合 ingest-candidate.json 契約(_-前綴診斷鍵已由 code 節點剝除)
|
||||
|
||||
# ── 執行環境變數(部署時注入;此檔不放密鑰)──
|
||||
# repo=Leo/notes ref=main gitea_token=<GITEA_TOKEN>
|
||||
# kbdb_url=https://arcrun-kbdb.leo21c.workers.dev kbdb_api_key=<partner key>
|
||||
# graph_url=<graph plugin base url on leo21c> graph_api_key=leo
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# §穩態變體(km_wiki_ingest_delta):Gitea push webhook → 只處理 delta 檔
|
||||
# 部署變數(皆已內嵌節點 data;此處僅列出以便 self-hosted fork 修改):
|
||||
# repo=Leo/notes ref=main owner=leo cards_root=system-dev/wiki/cards
|
||||
# kbdb_url = https://arcrun-kbdb.uncle6-me.workers.dev (WORKER_SUBDOMAIN=uncle6-me,見 cypher wrangler)
|
||||
# graph_url = https://kbdb-graph-plugin.uncle6-me.workers.dev
|
||||
# graph_api_key = leo(namespace 字串,非機密)
|
||||
# 機密:gitea_token → acr creds set gitea_token(節點以 {{credential.gitea_token}} 引用,部署時注入)。
|
||||
# ⚠️ WORKER_SUBDOMAIN 須與實際部署帳號一致:官方=uncle6-me;若你的 kbdb/graph 部在別的
|
||||
# workers.dev subdomain(例 leo21c),把上面 URL 一起改。舊 standalone drainer 誤用 leo21c。
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# 觸發 = Gitea repo Settings → Webhooks → 指向 arcrun(cypher-executor) 的 workflow webhook URL。
|
||||
# ⚠️ 這是 Gitea → Cloudflare(arcrun),非 GitHub Actions → 不觸 GitHub flag 紅線(D4/D20)。
|
||||
# 只把上面 flow 的 watch_cron/pick_next_card 換成:
|
||||
# input(webhook payload)>> ON_SUCCESS >> collect_changed
|
||||
# collect_changed = 從 payload.commits[].{added,modified} 濾出 system-dev/wiki/cards/**/*.md
|
||||
# >> foreach 檔 >> fetch_card >> parse_card >> upsert_entry >> post_envelopes(同上)
|
||||
# 量小、天生不撞頂、不需限速;靠 graph/entry 冪等自動 skip 未變檔。
|
||||
|
||||
Reference in New Issue
Block a user