feat(ingest): 機械式 wiki 卡片→KBDB ingest(Arcrun#8 Phase A,dry-run 驗證)
新 ingest 模型(無 LLM,取代舊 raw→Haiku 路): - 來源 = system-dev/wiki/cards/**/*.md(精耕卡)+ ## 實體/## 關聯 typed-edge + [[wikilink]] - 卡片→base entry(metadata.embed=true);typed-edge/wikilink→graph triplet - 純機械決定性,零 token 形式 = Arcrun workflow(cron drain + Gitea webhook delta)+ 新自訂零件 km_wiki_card_parse 不撞頂設計:一卡一 tick + 超大卡以 source_uri anchor 自動分段; graph fan-out 精確 = 7+4N+M+D,dry-run 對 notes 三卡上限 33 subrequest(<50)。 Phase A:不部署、不寫 live。含純核心 + dry-run 證據(3 entries/15 triplets/16 nodes)。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJiLCRUU2o3aSpPEzVCt2o
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# 提案:新自訂 arcrun 零件 km_wiki_card_parse 的契約(Phase A 提案,尚未部署)。
|
||||
# 定位:機械把「已精耕 wiki 卡片」解析成 base entry + graph triplet envelope(無 LLM、無 fs/網路)。
|
||||
# 完全符合 arcrun 元件形狀(stdin_stdout_json、no_network_syscall、no_filesystem_syscall)。
|
||||
# 參考實作:lib/card-to-envelope.mjs(決定性純函式;部署時移植成 Go/WASM 或以 workerd JS 元件承載)。
|
||||
canonical_id: "km_wiki_card_parse"
|
||||
display_name: "KM Wiki 卡片機械解析"
|
||||
category: "transform"
|
||||
version: "v1"
|
||||
wasi_target: "preview1"
|
||||
stability: "floating"
|
||||
runtime_compat:
|
||||
- "cf-workers"
|
||||
- "workerd"
|
||||
- "wazero"
|
||||
constraints:
|
||||
max_size_kb: 2048
|
||||
max_cold_start_ms: 50
|
||||
no_network_syscall: true # 純轉換,不打網路
|
||||
no_filesystem_syscall: true # 卡片全文由上游 http_request 餵入,不讀檔
|
||||
io_model: "stdin_stdout_json"
|
||||
input_schema:
|
||||
type: object
|
||||
required: [content, rel_path]
|
||||
properties:
|
||||
content:
|
||||
type: string
|
||||
description: 卡片 Markdown 全文(上游 http_request 從 Gitea raw 抓)。
|
||||
rel_path:
|
||||
type: string
|
||||
description: 卡片相對 repo 根的路徑(如 system-dev/wiki/cards/notes/Xxx.md)。
|
||||
repo:
|
||||
type: string
|
||||
description: 來源 repo(如 Leo/notes),組 source.uri / page_name 用。預設 Leo/notes。
|
||||
budget:
|
||||
type: number
|
||||
description: 單 envelope subrequest 目標上限(預設 40,CF 硬頂 50)。超過則自動 anchor 分段。
|
||||
commit:
|
||||
type: string
|
||||
description: git commit sha(可追溯,寫進 source.commit)。選填。
|
||||
output_schema:
|
||||
type: object
|
||||
properties:
|
||||
entry:
|
||||
type: object
|
||||
description: 卡片→base entry。含 page_name(冪等鍵)、entry_type、content、tags、metadata(embed=true, source, content_hash)。
|
||||
envelopes:
|
||||
type: array
|
||||
description: >
|
||||
卡片→graph triplet envelope 陣列(已分段、已符合 ingest-candidate.json 契約)。
|
||||
每個 envelope 附 _estSubrequests;planner 保證每段 ≤ budget。無邊的卡回空陣列。
|
||||
meta:
|
||||
type: object
|
||||
description: canonical / title / content_hash 等追溯資訊。
|
||||
gherkin_tests:
|
||||
- scenario: "typed-edge 卡片"
|
||||
given: '{"content":"## 實體\n- **A** — a\n- **B** — b\n## 關聯\n### 內文知識關係\n- A >> 關係 >> B","rel_path":"system-dev/wiki/cards/notes/t.md"}'
|
||||
then_contains: '"subject":"A"'
|
||||
- scenario: "embed 打標"
|
||||
given: '{"content":"# t\n## 實體\n- **A** — a\n## 關聯\n### 內文知識關係\n- A >> r >> A","rel_path":"x/t.md"}'
|
||||
then_contains: '"embed":true'
|
||||
tags: [transform, wiki, ingest, mechanical, kbdb, primitive]
|
||||
description: |
|
||||
取代 kbdb-ingest-plugin 舊 raw→Haiku 路的機械版:讀「已精耕卡」的 ## 實體(節點)、
|
||||
## 關聯 的 typed-edge(A >> 關係 >> B)與 [[wikilink]],直接映射 entry + triplet,零 LLM、零 token。
|
||||
自動以 source_uri anchor 分段,保證每個 graph /triplets/ingest 呼叫壓在 CF subrequest 上限下。
|
||||
@@ -0,0 +1,74 @@
|
||||
# km-wiki-ingest — 機械式 wiki 卡片 → KBDB ingest(Arcrun#8 / 頂層 SDD T2–T4)
|
||||
|
||||
> Phase A 產物:機械 ingest 邏輯 + 乾跑證據 + workflow 設計。**不部署、不寫 live KBDB。**
|
||||
|
||||
## 解決什麼問題
|
||||
|
||||
把各 repo 的 `system-dev/wiki/cards/**/*.md`(人工精耕卡)**機械地**(無 LLM)灌進 leo21c KBDB:
|
||||
|
||||
- **卡片 → base entry**(`metadata.embed=true`,供語意搜尋)。
|
||||
- **`## 實體` → graph node**;**`## 關聯` 的 typed-edge(`A >> 關係 >> B`)與 `[[wikilink]]` → graph triplet**。
|
||||
|
||||
取代舊 `kbdb-ingest-plugin/scripts/ingest-cli.mjs` 的 `raw → Haiku → 三元組` 路:新路純解析卡片內既有結構,**決定性、零 token、零幻覺**。
|
||||
|
||||
## 形式選擇與理由(給總管)
|
||||
|
||||
**形式 = Arcrun workflow(YAML 編排)+ 1 個新自訂零件 `km_wiki_card_parse`(機械解析)+ 現成零件(`cron` / `http_request` / `foreach_control` / `kbdb_upsert_block`)。**
|
||||
|
||||
理由:
|
||||
|
||||
1. **編排本來就是 arcrun 的主場**:cron 限速 drain、Gitea webhook 只吃 delta、foreach 小批、冪等 upsert——這些跟現成零件 1:1 對得上,且 leo 要「Arcrun workflow 慢慢做」、arcrun 哲學禁一次性腳本。
|
||||
2. **arcrun 唯一缺的是「卡片 → envelope」的解析**。那是一段**決定性純轉換**(無 LLM、無網路、無檔案)——剛好是自訂零件的理想形狀(`stdin_stdout_json` + `no_network_syscall` + `no_filesystem_syscall`)。把它做成零件(而非在 YAML 裡塞 `string_ops` 正則)讓 workflow 可讀、解析可單元測試,貼合 arcrun「零件」複用哲學。
|
||||
3. **小批是結構性的,不是靠祈禱**:一卡一 tick,每卡在 graph worker 的 fan-out ≈ `7+4N+M` subrequest(notes 卡 N≈4/M≈5 → est 28~33,穩壓 CF 50 頂下);解析零件會**預先把超大卡以 `source_uri` anchor 分段**,任何單一 graph 呼叫都不破頂。
|
||||
|
||||
**Phase A 交付**:純解析+打包核心(`lib/card-to-envelope.mjs`,現在就能跑)+乾跑驗證器(`lib/dry-run.mjs`,印出「將寫入什麼」)+本 workflow.yaml +零件契約(部署被閘控,故 live 接線是「設計而非執行」)。
|
||||
|
||||
## 診斷小結:fan-out 精確來源 + 小批為何解得掉
|
||||
|
||||
讀 `kbdb-graph-plugin` 現役寫入路徑(`triplet-ingest.ts` / `triplet-crud.ts` / `templates.ts` / `kbdb-client.ts`)逐行拆帳:
|
||||
|
||||
```
|
||||
POST /triplets/ingest(graph worker 單次 invocation)對 base 的 subrequest:
|
||||
ensurePluginTemplates(3) # 頂層一次
|
||||
+ listRecordsByTemplate(1) # 抓同 source 現存 active(冪等分組)
|
||||
+ Σ_triplet [ createTriplet → ensurePluginTemplates(3) + createRecord(1) ] # ★ 每條邊重跑 ensure!
|
||||
+ persistNodes [ ensurePluginTemplates(3) + Σ_node createRecord(1) ]
|
||||
+ Σ_deprecated updateRecord(1)
|
||||
= 7 + 4*N_triplets + M_nodes + D_deprecated
|
||||
```
|
||||
|
||||
- **精確炸點還原**:07_01 單一 envelope 吞 `N=11, M=10, D=0` → `7+44+10 = 61 > 50` → 破頂半殘。**放大器=`createTriplet` 內每條邊都重呼 `ensurePluginTemplates`(3 個 GET)**,佔了 33/61。
|
||||
- **小批為何解得掉**:把「整檔一 envelope」改成「一卡一 envelope、必要時再 anchor 分段」,把 `N` 壓到讓 `7+4N+M ≤ 40`。notes 三卡實測 est 上限 = 33,全綠。超大卡(自測 20 邊/22 節點=114)→ 自動分 4 段,每段 ≤ 38。
|
||||
- **附帶建議(非本 Phase 必改)**:graph 端把 `createTriplet`/`persistNodes` 內重複的 `ensurePluginTemplates` 提到 ingest 入口只跑一次,可把每 envelope 省下 `3*(N+1)` 個 subrequest(單卡 est 33→約 18),批量還能更大。此為 graph-plugin 的可選優化,記此存查。
|
||||
|
||||
## 冪等設計
|
||||
|
||||
| 對象 | 冪等鍵 | 行為 |
|
||||
|---|---|---|
|
||||
| **entry** | `page_name`(穩定:`wikicard:<repo>/<canonical>`)+ `metadata.content_hash` | 找到同 page_name:hash 相同 → skip;不同 → PATCH content(觸發重嵌)。沒有 → POST 新建。 |
|
||||
| **triplet envelope** | `source.uri` + `source.content_hash` | graph 現役 per-source 冪等:同 hash 整包 no-op(`triplet-ingest.ts:65`)。 |
|
||||
| **分段** | 各段 `source.uri = <基uri>#segNN` | 各段獨立 uri → 各自獨立冪等,**繞開 per-source content_hash 整包 skip**(否則同 uri 第 2 段起會被判定「已落地」而整包跳過)。節點只放進「首次引用它的段」,跨段不重送(避免 graph 重建 entity)。 |
|
||||
|
||||
## 觸發(兩階段,對齊 SDD R3)
|
||||
|
||||
- **Phase 0(一次性 backfill)**:`cron */2` 每 tick drain 一張卡(限速慢推),反覆跑到全庫清空。冪等 → 可續傳、重跑零寫入。
|
||||
- **穩態(日常增量)**:**Gitea push webhook → arcrun workflow**,只吃 `commits[].{added,modified}` 中的 `system-dev/wiki/cards/**/*.md`。⚠️ Gitea → Cloudflare(arcrun),**非 GitHub Actions**,不觸 GitHub flag 紅線(D4/D20)。量小、不撞頂、不限速。
|
||||
|
||||
## 乾跑證據(Phase A,不寫 live)
|
||||
|
||||
```
|
||||
node lib/dry-run.mjs --repo-path <notes clone> --repo Leo/notes --self-test
|
||||
```
|
||||
|
||||
對 `Leo/notes` 的 3 張卡實測:3 entries(embed=true)+ 3 envelopes、15 triplets、16 nodes;
|
||||
**單次 graph 呼叫 subrequest 上限 = 33(< 50),無任一 envelope 破頂**。
|
||||
self-test 合成超大卡(不分段 est=114 會炸)→ 自動分 4 段、每段 ≤ 38,全綠。
|
||||
|
||||
## 待 live 部署 + 寫入(總管過 leo 閘用)
|
||||
|
||||
1. **部署新零件 `km_wiki_card_parse`**:移植 `lib/card-to-envelope.mjs` → arcrun 元件(Go/WASM 或 workerd JS 承載)→ `acr` 註冊。
|
||||
2. **部署 workflow**:`km_wiki_ingest_drain`(cron drain);`wrangler` 直推 leo21c(**禁 `acr update`**——codeload 綁 GitHub 假綠,Arcrun#4)。
|
||||
3. **注入環境變數**(不放 repo):`repo=Leo/notes ref=main gitea_token kbdb_url=https://arcrun-kbdb.leo21c.workers.dev kbdb_api_key graph_url graph_api_key=leo`。`CLOUDFLARE_ACCOUNT_ID=leo21c`(別讓官方 58309b 污染)。
|
||||
4. **entry 寫入路徑確認**:若 `kbdb_upsert_block` 尚不透傳 `metadata_json`(需 `embed:true`/`content_hash`),entry 改用 `http_request` 直打 base `POST/PATCH /entries` 帶 `body_json.metadata_json`。
|
||||
5. **預期寫入量(Leo/notes 現況 3 卡)**:3 entries + 15 triplets + 16 node records(去重後更少);分 3 次 graph 呼叫(每次 ≤ 33 subrequest)+ 3 次 entry upsert。全庫鋪開時照 cron 一卡一 tick 慢推。
|
||||
6. **驗收**:ingest 後 `GET /embed/backfill/status` 應見 pending 上升→drain 後歸零、embedded 增加;三模式(關鍵字/語意/圖)curl 驗。
|
||||
@@ -0,0 +1,176 @@
|
||||
{
|
||||
"repo": "Leo/notes",
|
||||
"commit": "4b9a53c1c99d596c23b1b449fd79728610f6995b",
|
||||
"budget": 40,
|
||||
"ceiling": 50,
|
||||
"cards": [
|
||||
{
|
||||
"relPath": "system-dev/wiki/cards/notes/Gitea當後端編輯器全CF化網站構想.md",
|
||||
"canonical": "Gitea當後端編輯器全CF化網站構想",
|
||||
"entry": {
|
||||
"page_name": "wikicard:Leo/notes/Gitea當後端編輯器全CF化網站構想",
|
||||
"entry_type": "wiki_card",
|
||||
"metadata.embed": true,
|
||||
"content_hash": "64b402952fe0…",
|
||||
"content_bytes": 2324,
|
||||
"tags": [
|
||||
"系統設計",
|
||||
"工具教學"
|
||||
]
|
||||
},
|
||||
"envelopeCount": 1,
|
||||
"envelopes": [
|
||||
{
|
||||
"source.uri": "gitea:Leo/notes@system-dev/wiki/cards/notes/Gitea當後端編輯器全CF化網站構想.md",
|
||||
"source.anchor": null,
|
||||
"nodes": 6,
|
||||
"triplets": 5,
|
||||
"est_subrequests": 33,
|
||||
"under_ceiling": true,
|
||||
"sample_triplets": [
|
||||
"Gitea >> 類比於 >> WordPress (1)",
|
||||
"Gitea >> 充當後端供稿給 >> Cloudflare Pages (1)",
|
||||
"Quartz >> 目前負責轉譯給 >> Cloudflare Pages (1)",
|
||||
"Cloudflare Artifacts >> 若提供 git 倉庫則可取代 >> Gitea (1)"
|
||||
],
|
||||
"sample_nodes": [
|
||||
"Gitea — 可自架的 git 平台,編輯體驗近似 WordPress 後…",
|
||||
"WordPress — 常見的內容管理後端,作為 Gitea 編輯體驗的類比對象。…",
|
||||
"Cloudflare Pages — Cloudflare 的靜態站前端託管。…",
|
||||
"Quartz — 目前把筆記轉成網站前端的工具。…"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"relPath": "system-dev/wiki/cards/notes/Prompt能力即拆解自己邏輯的能力.md",
|
||||
"canonical": "Prompt能力即拆解自己邏輯的能力",
|
||||
"entry": {
|
||||
"page_name": "wikicard:Leo/notes/Prompt能力即拆解自己邏輯的能力",
|
||||
"entry_type": "wiki_card",
|
||||
"metadata.embed": true,
|
||||
"content_hash": "63296a663227…",
|
||||
"content_bytes": 2109,
|
||||
"tags": [
|
||||
"AI協作",
|
||||
"工具教學",
|
||||
"觀點主張"
|
||||
]
|
||||
},
|
||||
"envelopeCount": 1,
|
||||
"envelopes": [
|
||||
{
|
||||
"source.uri": "gitea:Leo/notes@system-dev/wiki/cards/notes/Prompt能力即拆解自己邏輯的能力.md",
|
||||
"source.anchor": null,
|
||||
"nodes": 5,
|
||||
"triplets": 5,
|
||||
"est_subrequests": 32,
|
||||
"under_ceiling": true,
|
||||
"sample_triplets": [
|
||||
"Prompt 能力 >> 本質上等於 >> 邏輯拆解能力 (1)",
|
||||
"邏輯拆解能力 >> 產出 >> pseudo code (1)",
|
||||
"pseudo code >> 足以教會 >> AI (1)",
|
||||
"Prompt能力即拆解自己邏輯的能力 >> 呼應 >> 程式化邏輯可圖解任何主題不限AI (1)"
|
||||
],
|
||||
"sample_nodes": [
|
||||
"Prompt 能力 — 把腦中意圖轉成能指揮 AI 的指令的能力。…",
|
||||
"邏輯拆解能力 — 把腦中隱性流程外顯成可陳述步驟的能力。…",
|
||||
"pseudo code — 用類程式的步驟描述邏輯、尚未綁定特定語法的表達。…",
|
||||
"AI — 需被人以指令/範例指揮才產出的生成模型。…"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"relPath": "system-dev/wiki/cards/notes/程式化邏輯可圖解任何主題不限AI.md",
|
||||
"canonical": "程式化邏輯可圖解任何主題不限AI",
|
||||
"entry": {
|
||||
"page_name": "wikicard:Leo/notes/程式化邏輯可圖解任何主題不限AI",
|
||||
"entry_type": "wiki_card",
|
||||
"metadata.embed": true,
|
||||
"content_hash": "a9dbf5fc0f9a…",
|
||||
"content_bytes": 2577,
|
||||
"tags": [
|
||||
"工具教學",
|
||||
"觀點主張",
|
||||
"系統設計"
|
||||
]
|
||||
},
|
||||
"envelopeCount": 1,
|
||||
"envelopes": [
|
||||
{
|
||||
"source.uri": "gitea:Leo/notes@system-dev/wiki/cards/notes/程式化邏輯可圖解任何主題不限AI.md",
|
||||
"source.anchor": null,
|
||||
"nodes": 5,
|
||||
"triplets": 5,
|
||||
"est_subrequests": 32,
|
||||
"under_ceiling": true,
|
||||
"sample_triplets": [
|
||||
"程式化邏輯 >> 可圖解 >> 亞洲金融風暴 (1)",
|
||||
"流程圖解 >> 奠基於 >> 程式化邏輯 (1)",
|
||||
"系統動力學 >> 類同於 >> 流程圖解 (1)",
|
||||
"程式化邏輯可圖解任何主題不限AI >> 呼應 >> Prompt能力即拆解自己邏輯的能力 (1)"
|
||||
],
|
||||
"sample_nodes": [
|
||||
"程式化邏輯 — 以程式的因果鏈結構來表述任一領域的邏輯。…",
|
||||
"亞洲金融風暴 — 講者小 Lin 用長邏輯鏈敘述的金融事件案例。…",
|
||||
"流程圖解 — 用 n8n 這類流程工具把邏輯視覺化講解的方法。…",
|
||||
"系統動力學 — 以存量流量與回饋環圖解因果的建模工具。…"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"totals": {
|
||||
"cards": 3,
|
||||
"entries_to_upsert": 3,
|
||||
"triplet_envelopes": 3,
|
||||
"total_triplets": 15,
|
||||
"total_nodes": 16,
|
||||
"max_est_subrequests_single_call": 33,
|
||||
"ceiling": 50,
|
||||
"budget": 40,
|
||||
"any_envelope_over_ceiling": 0
|
||||
},
|
||||
"self_test": {
|
||||
"note": "合成 20 邊 / 22 節點 的超大卡",
|
||||
"if_single_envelope_est_subrequests": 114,
|
||||
"would_crash_single": true,
|
||||
"segmented_into": 4,
|
||||
"per_segment": [
|
||||
{
|
||||
"uri": "gitea:Leo/notes@system-dev/wiki/cards/notes/合成超大卡.md#seg01",
|
||||
"anchor": "seg01",
|
||||
"triplets": 6,
|
||||
"nodes": 7,
|
||||
"est_subrequests": 38,
|
||||
"under_ceiling": true
|
||||
},
|
||||
{
|
||||
"uri": "gitea:Leo/notes@system-dev/wiki/cards/notes/合成超大卡.md#seg02",
|
||||
"anchor": "seg02",
|
||||
"triplets": 6,
|
||||
"nodes": 6,
|
||||
"est_subrequests": 37,
|
||||
"under_ceiling": true
|
||||
},
|
||||
{
|
||||
"uri": "gitea:Leo/notes@system-dev/wiki/cards/notes/合成超大卡.md#seg03",
|
||||
"anchor": "seg03",
|
||||
"triplets": 6,
|
||||
"nodes": 6,
|
||||
"est_subrequests": 37,
|
||||
"under_ceiling": true
|
||||
},
|
||||
{
|
||||
"uri": "gitea:Leo/notes@system-dev/wiki/cards/notes/合成超大卡.md#seg04",
|
||||
"anchor": "seg04",
|
||||
"triplets": 3,
|
||||
"nodes": 3,
|
||||
"est_subrequests": 22,
|
||||
"under_ceiling": true
|
||||
}
|
||||
],
|
||||
"all_segments_under_ceiling": true
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,178 @@
|
||||
#!/usr/bin/env node
|
||||
// km-wiki-ingest 乾跑(dry-run)驗證器 — 不寫 live、不部署。
|
||||
// -------------------------------------------------------------
|
||||
// 對某 repo 的 system-dev/wiki/cards/**/*.md 跑機械解析 + envelope 打包,
|
||||
// 輸出「將寫入什麼」:① entry 清單(page_name / entry_type / metadata.embed / content_hash)
|
||||
// ② triplet envelope 清單(每段的 nodes / triplets / source.uri+anchor)
|
||||
// ③ 每個 graph /triplets/ingest 呼叫的 subrequest 估算(證明壓在 CF 上限下)
|
||||
// ④ 冪等鍵設計(entry=page_name+content_hash;triplet=source.uri+content_hash)。
|
||||
//
|
||||
// 用法:node dry-run.mjs --repo-path <clone路徑> [--repo Leo/notes] [--budget 40] [--json] [--full]
|
||||
// --self-test 額外跑「合成超大卡」證明分段生效。
|
||||
|
||||
import { readFileSync, existsSync, readdirSync, statSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { planCard, estimateEnvelopeSubrequests, SUBREQ_CEILING, SUBREQ_BUDGET } from './card-to-envelope.mjs';
|
||||
|
||||
function parseArgs(argv) {
|
||||
const out = { repo: 'Leo/notes', budget: SUBREQ_BUDGET };
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--repo-path') out.repoPath = argv[++i];
|
||||
else if (a === '--repo') out.repo = argv[++i];
|
||||
else if (a === '--budget') out.budget = Number(argv[++i]);
|
||||
else if (a === '--json') out.json = true;
|
||||
else if (a === '--full') out.full = true;
|
||||
else if (a === '--self-test') out.selfTest = true;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function walkCards(dir) {
|
||||
const out = [];
|
||||
if (!existsSync(dir)) return out;
|
||||
for (const e of readdirSync(dir, { withFileTypes: true })) {
|
||||
const p = path.join(dir, e.name);
|
||||
if (e.isDirectory()) out.push(...walkCards(p));
|
||||
else if (e.name.endsWith('.md') && e.name !== '.gitkeep' && !e.name.startsWith('00-INDEX')) out.push(p);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function gitCommit(repoPath) {
|
||||
try {
|
||||
return execFileSync('git', ['-C', repoPath, 'rev-parse', 'HEAD'], { encoding: 'utf8' }).trim();
|
||||
} catch { return undefined; }
|
||||
}
|
||||
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (!args.repoPath) {
|
||||
console.error('用法: node dry-run.mjs --repo-path <clone路徑> [--repo Leo/notes] [--budget 40] [--json] [--full] [--self-test]');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const cardsRoot = path.join(args.repoPath, 'system-dev', 'wiki', 'cards');
|
||||
const cardPaths = walkCards(cardsRoot);
|
||||
const commit = gitCommit(args.repoPath);
|
||||
|
||||
const report = { repo: args.repo, commit, budget: args.budget, ceiling: SUBREQ_CEILING, cards: [], totals: {} };
|
||||
let totEntries = 0, totEnvelopes = 0, totTriplets = 0, totNodes = 0, maxSub = 0, over = 0;
|
||||
|
||||
for (const p of cardPaths) {
|
||||
const rel = path.relative(args.repoPath, p);
|
||||
const md = readFileSync(p, 'utf8');
|
||||
const plan = planCard(md, rel, args.repo, { budget: args.budget, commit });
|
||||
totEntries++;
|
||||
totEnvelopes += plan.envelopes.length;
|
||||
const cardTriplets = plan.envelopes.reduce((s, e) => s + e.triplets.length, 0);
|
||||
const cardNodes = plan.envelopes.reduce((s, e) => s + e.nodes.length, 0);
|
||||
totTriplets += cardTriplets;
|
||||
totNodes += cardNodes;
|
||||
for (const e of plan.envelopes) {
|
||||
maxSub = Math.max(maxSub, e._estSubrequests);
|
||||
if (e._estSubrequests > SUBREQ_CEILING) over++;
|
||||
}
|
||||
report.cards.push({
|
||||
relPath: rel,
|
||||
canonical: plan.meta.canonical,
|
||||
entry: {
|
||||
page_name: plan.entry.page_name,
|
||||
entry_type: plan.entry.entry_type,
|
||||
'metadata.embed': plan.entry.metadata.embed,
|
||||
content_hash: plan.entry.metadata.content_hash.slice(0, 12) + '…',
|
||||
content_bytes: Buffer.byteLength(plan.entry.content, 'utf8'),
|
||||
tags: plan.entry.tags,
|
||||
},
|
||||
envelopeCount: plan.envelopes.length,
|
||||
envelopes: plan.envelopes.map((e) => ({
|
||||
'source.uri': e.source.uri,
|
||||
'source.anchor': e.source.anchor ?? null,
|
||||
nodes: e.nodes.length,
|
||||
triplets: e.triplets.length,
|
||||
est_subrequests: e._estSubrequests,
|
||||
under_ceiling: e._estSubrequests <= SUBREQ_CEILING,
|
||||
sample_triplets: e.triplets.slice(0, args.full ? 999 : 4).map((t) => `${t.subject} >> ${t.predicate} >> ${t.object} (${t.confidence})`),
|
||||
sample_nodes: e.nodes.slice(0, args.full ? 999 : 4).map((n) => `${n.name}${n.gloss ? ' — ' + n.gloss.slice(0, 30) + '…' : ''}`),
|
||||
})),
|
||||
});
|
||||
}
|
||||
|
||||
report.totals = {
|
||||
cards: totEntries,
|
||||
entries_to_upsert: totEntries,
|
||||
triplet_envelopes: totEnvelopes,
|
||||
total_triplets: totTriplets,
|
||||
total_nodes: totNodes,
|
||||
max_est_subrequests_single_call: maxSub,
|
||||
ceiling: SUBREQ_CEILING,
|
||||
budget: args.budget,
|
||||
any_envelope_over_ceiling: over,
|
||||
};
|
||||
|
||||
// --- self-test:合成一張「超大卡」(20 邊 + 22 節點)證明單 envelope 會爆、分段後每段壓在預算下 ---
|
||||
if (args.selfTest) {
|
||||
const entities = [];
|
||||
const edges = [];
|
||||
for (let i = 0; i < 22; i++) entities.push(`- **實體${i}**(別名${i})— 這是實體 ${i} 的一句描述。`);
|
||||
for (let i = 0; i < 20; i++) edges.push(`- 實體${i} >> 關聯到 >> 實體${i + 1}`);
|
||||
const bigCard = `---\ntags: [壓測]\ngloss: 合成超大卡,測分段。\n---\n# 合成超大卡\n\n← [[notes/00-INDEX]]\n\n## 實體\n${entities.join('\n')}\n\n## 關聯\n### 內文知識關係\n${edges.join('\n')}\n`;
|
||||
const plan = planCard(bigCard, 'system-dev/wiki/cards/notes/合成超大卡.md', args.repo, { budget: args.budget });
|
||||
const single = estimateEnvelopeSubrequests(plan.tripletCount, plan.nodeCount);
|
||||
report.self_test = {
|
||||
note: '合成 20 邊 / 22 節點 的超大卡',
|
||||
if_single_envelope_est_subrequests: single,
|
||||
would_crash_single: single > SUBREQ_CEILING,
|
||||
segmented_into: plan.envelopes.length,
|
||||
per_segment: plan.envelopes.map((e) => ({
|
||||
uri: e.source.uri, anchor: e.source.anchor, triplets: e.triplets.length, nodes: e.nodes.length, est_subrequests: e._estSubrequests, under_ceiling: e._estSubrequests <= SUBREQ_CEILING,
|
||||
})),
|
||||
all_segments_under_ceiling: plan.envelopes.every((e) => e._estSubrequests <= SUBREQ_CEILING),
|
||||
};
|
||||
}
|
||||
|
||||
if (args.json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
// --- 人類可讀輸出 ---
|
||||
const L = (s = '') => console.log(s);
|
||||
L(`\n================ km-wiki-ingest DRY-RUN(不寫 live)================`);
|
||||
L(`repo=${report.repo} commit=${(commit || '(none)').slice(0, 12)} budget=${args.budget} CF_ceiling=${SUBREQ_CEILING}`);
|
||||
L(`卡片來源根:${path.relative(args.repoPath, cardsRoot)} 找到 ${cardPaths.length} 張卡\n`);
|
||||
|
||||
for (const c of report.cards) {
|
||||
L(`── 卡片:${c.relPath}`);
|
||||
L(` ENTRY(base POST /entries 或 kbdb_upsert_block,冪等鍵 page_name):`);
|
||||
L(` page_name = ${c.entry.page_name}`);
|
||||
L(` entry_type = ${c.entry.entry_type}`);
|
||||
L(` metadata.embed= ${c.entry['metadata.embed']} content_hash=${c.entry.content_hash} bytes=${c.entry.content_bytes}`);
|
||||
L(` tags = ${JSON.stringify(c.entry.tags)}`);
|
||||
L(` TRIPLET ENVELOPE(s)(POST graph /triplets/ingest;分段數=${c.envelopeCount}):`);
|
||||
for (const e of c.envelopes) {
|
||||
L(` • uri=${e['source.uri']}${e['source.anchor'] ? ' anchor=' + e['source.anchor'] : ''}`);
|
||||
L(` nodes=${e.nodes} triplets=${e.triplets} est_subrequests=${e.est_subrequests} ≤ceiling? ${e.under_ceiling ? 'YES' : 'NO ⚠️'}`);
|
||||
for (const t of e.sample_triplets) L(` - ${t}`);
|
||||
if (e.sample_nodes.length) L(` nodes: ${e.sample_nodes.join(' | ')}`);
|
||||
}
|
||||
L('');
|
||||
}
|
||||
|
||||
L(`================ 彙總 ================`);
|
||||
for (const [k, v] of Object.entries(report.totals)) L(` ${k.padEnd(34)} = ${v}`);
|
||||
L(` 冪等設計:`);
|
||||
L(` entry → page_name(穩定鍵)+ metadata.content_hash(比對是否改動 → 未改 skip、改動 PATCH 重嵌)`);
|
||||
L(` triplet → source.uri + source.content_hash(graph 現役 per-source 冪等;同 hash 整包 no-op)`);
|
||||
L(` 分段 → 各段獨立 source.uri(#segNN)→ 各自獨立冪等,繞開 per-source content_hash 整包 skip`);
|
||||
|
||||
if (report.self_test) {
|
||||
L(`\n================ SELF-TEST:超大檔分段 ================`);
|
||||
const st = report.self_test;
|
||||
L(` ${st.note}`);
|
||||
L(` 若不分段(單 envelope)估算 subrequest = ${st.if_single_envelope_est_subrequests} → 會炸? ${st.would_crash_single ? 'YES(> ' + SUBREQ_CEILING + ')' : 'no'}`);
|
||||
L(` 分段後段數 = ${st.segmented_into},每段:`);
|
||||
for (const s of st.per_segment) L(` - ${s.anchor}: triplets=${s.triplets} nodes=${s.nodes} est=${s.est_subrequests} ≤ceiling? ${s.under_ceiling ? 'YES' : 'NO ⚠️'}`);
|
||||
L(` 全部段壓在上限下? ${st.all_segments_under_ceiling ? 'YES ✅' : 'NO ⚠️'}`);
|
||||
}
|
||||
L('');
|
||||
@@ -0,0 +1 @@
|
||||
["ingest", "kbdb", "wiki", "mechanical", "cron", "webhook", "graph", "triplet", "no-llm"]
|
||||
@@ -0,0 +1,105 @@
|
||||
name: km_wiki_ingest_drain
|
||||
description: >
|
||||
Phase 0 限速 drain:cron 每 tick 只處理「一張卡」→ 機械解析成 entry + triplet envelope
|
||||
→ 冪等寫 KBDB(base entry / graph triplet)。反覆跑直到全庫 drain 完。
|
||||
來源=repo 的 system-dev/wiki/cards/**/*.md(人工精耕卡,非裸筆記,無 LLM)。
|
||||
穩態(Gitea push webhook 只處理 delta)見檔尾 §穩態變體。
|
||||
|
||||
# ── 為什麼「一 tick 一卡」=根治 07_01 的 Too many subrequests ──
|
||||
# graph worker 處理一次 POST /triplets/ingest 的 subrequest = 7 + 4*N_triplets + M_nodes + D_deprecated。
|
||||
# 07_01 炸點:單一 envelope 吞整檔 N=11,M=10 → 61 > 50(CF bundled 上限)→ 半殘。
|
||||
# 對策:① 一卡一 tick(天然小批,notes 卡 ~N4/M5 → est 28~33,穩壓 50 下)
|
||||
# ② km_wiki_card_parse 會自動把超大卡以 source_uri anchor 分段(每段獨立冪等)。
|
||||
# ⟹ 任何單一 graph 呼叫都不會再破頂。
|
||||
|
||||
flow:
|
||||
- "watch_cron >> ON_SUCCESS >> pick_next_card"
|
||||
- "pick_next_card >> ON_SUCCESS >> fetch_card"
|
||||
- "fetch_card >> ON_SUCCESS >> parse_card"
|
||||
- "parse_card >> ON_SUCCESS >> upsert_entry" # 卡片 → base entry(embed=true),冪等
|
||||
- "upsert_entry >> ON_SUCCESS >> post_envelopes" # wikilink/typed-edge → graph triplet
|
||||
- "post_envelopes >> 對每個 envelope >> post_one_envelope" # 分段時多段,各段獨立冪等
|
||||
|
||||
config:
|
||||
# 1) 排程 tick:慢推。每 2 分鐘一張卡=限速(Phase 0 唯一需要 rate-limit 之處)。
|
||||
watch_cron:
|
||||
component: cron
|
||||
cron_expr: "*/2 * * * *"
|
||||
description: "每 2 分鐘 drain 一張卡(限速慢推,避免 CF 額度與 subrequest 壓力)"
|
||||
|
||||
# 2) 取下一張待處理卡(cursor drain)。用 Gitea contents API 列 cards 目錄 + 一個游標 block
|
||||
# 記「處理到哪」。回傳單一 { rel_path, download_url, content_hash?(git blob sha) }。
|
||||
# 註:list + cursor 的細節可用 http_request(Gitea API) + set/string_ops 組;此處給語意佔位。
|
||||
pick_next_card:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "https://git.uncle6.me/api/v1/repos/{{repo}}/contents/system-dev/wiki/cards?ref={{ref}}"
|
||||
headers:
|
||||
Authorization: "token {{gitea_token}}"
|
||||
Accept: "application/json"
|
||||
# 下游用 filter/set 取「游標之後第一張、且 .md、且非 00-INDEX」的一張。
|
||||
|
||||
# 3) 抓卡片全文(Gitea raw)。
|
||||
fetch_card:
|
||||
component: http_request
|
||||
method: GET
|
||||
url: "{{pick_next_card.next.download_url}}"
|
||||
headers:
|
||||
Authorization: "token {{gitea_token}}"
|
||||
|
||||
# 4) ★ 機械解析(新自訂零件,無 LLM、無 fs/網路,stdin→stdout JSON)。
|
||||
# input:卡片全文 + 相對路徑 + repo;output:{ entry, envelopes[] }(envelope 已分段、已估 subrequest)。
|
||||
# 參考實作見 lib/card-to-envelope.mjs(可移植成 Go/WASM 元件)。
|
||||
parse_card:
|
||||
component: km_wiki_card_parse
|
||||
content: "{{fetch_card.data.body}}"
|
||||
rel_path: "{{pick_next_card.next.rel_path}}"
|
||||
repo: "{{repo}}"
|
||||
budget: 40 # subrequest 目標上限(留 10 給 D_deprecated)
|
||||
|
||||
# 5) 卡片 → base entry,冪等 upsert(page_name 當鍵;找到 PATCH、沒有 POST)。
|
||||
# metadata.embed=true → base embed 模組會補嵌 → 語意可搜。
|
||||
upsert_entry:
|
||||
component: kbdb_upsert_block
|
||||
api_key: "{{kbdb_api_key}}"
|
||||
kbdb_url: "{{kbdb_url}}"
|
||||
page_name: "{{parse_card.entry.page_name}}"
|
||||
type: "{{parse_card.entry.entry_type}}"
|
||||
content: "{{parse_card.entry.content}}"
|
||||
source: "{{parse_card.entry.metadata.source}}"
|
||||
tags_json: "{{parse_card.entry.tags_json}}"
|
||||
# ⚠️ metadata.embed=true / content_hash 需經 base /entries 帶 metadata_json 落地;
|
||||
# 若 kbdb_upsert_block 尚未透傳 metadata_json,改用 http_request 直打 base POST/PATCH /entries
|
||||
# 帶 body_json.metadata_json(見 description.md §entry 冪等)。
|
||||
|
||||
# 6) triplet envelope(可能多段)→ 逐段 POST graph /triplets/ingest。
|
||||
# graph 端 per-source(uri+content_hash) 冪等:同 hash 整包 no-op;分段各段 uri 不同 → 各自獨立冪等。
|
||||
post_envelopes:
|
||||
component: foreach_control
|
||||
items: "{{parse_card.envelopes}}"
|
||||
item_key: envelope
|
||||
|
||||
post_one_envelope:
|
||||
component: http_request
|
||||
method: POST
|
||||
url: "{{graph_url}}/triplets/ingest"
|
||||
headers:
|
||||
Content-Type: "application/json"
|
||||
X-Arcrun-API-Key: "{{graph_api_key}}"
|
||||
body_json: "{{envelope}}" # envelope 已符合 ingest-candidate.json 契約(禁止欄位已排除)
|
||||
|
||||
# ── 執行環境變數(部署時注入;此檔不放密鑰)──
|
||||
# repo=Leo/notes ref=main gitea_token=<GITEA_TOKEN>
|
||||
# kbdb_url=https://arcrun-kbdb.leo21c.workers.dev kbdb_api_key=<partner key>
|
||||
# graph_url=<graph plugin base url on leo21c> graph_api_key=leo
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# §穩態變體(km_wiki_ingest_delta):Gitea push webhook → 只處理 delta 檔
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# 觸發 = Gitea repo Settings → Webhooks → 指向 arcrun(cypher-executor) 的 workflow webhook URL。
|
||||
# ⚠️ 這是 Gitea → Cloudflare(arcrun),非 GitHub Actions → 不觸 GitHub flag 紅線(D4/D20)。
|
||||
# 只把上面 flow 的 watch_cron/pick_next_card 換成:
|
||||
# input(webhook payload)>> ON_SUCCESS >> collect_changed
|
||||
# collect_changed = 從 payload.commits[].{added,modified} 濾出 system-dev/wiki/cards/**/*.md
|
||||
# >> foreach 檔 >> fetch_card >> parse_card >> upsert_entry >> post_envelopes(同上)
|
||||
# 量小、天生不撞頂、不需限速;靠 graph/entry 冪等自動 skip 未變檔。
|
||||
Reference in New Issue
Block a user