feat(km-wiki-ingest): 新增無人值守 drainer runtime(cursor + cron + Gitea webhook)
Phase b:把 ingest 從手動直驅升級成真無人值守。drainer/ 是 workflow km_wiki_ingest_drain 的可部署執行體(Cloudflare Worker),復用已 live 的 arcrun-code/kbdb/graph、不改 cypher-executor。 - 兩觸發共用 processCard 核心:(A) cron drain(Gitea tree 列卡→游標後取一小批→ 逐卡處理→進游標→到底回捲;撐 5,083 檔);(B) Gitea push webhook→只處理 delta。 - 游標存 KBDB 一個 ingest_cursor entry(走 API、零建表)。冪等:未改卡 entry (page_name+content_hash)/triplet(uri+hash) 皆 skip,只有新/改卡真寫。 - src/usercode.ts 由 workflow.yaml parse_card.code 抽出(單一真相源 card-to-envelope.mjs)。 - 維持非-auto:wrangler.toml 的 cron trigger 預設註解(不自走),drain 走 POST /drain。 scheduled handler 已在 code,leo 過閘後取消註解即開自走。README 記部署/secret/webhook 設定。 已部署 leo21c:arcrun-km-wiki-drainer.leo21c.workers.dev。以 wrangler dev(本地 miniflare + .dev.vars token,非 secret 寫入)對 live 服務驗證:cron drain 冪等全 skip、游標前進+回捲、 webhook delta 只寫變更卡、三模式命中新卡。GITEA_TOKEN secret 需 leo 於互動模式設(auto 擋 secret 寫入)。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJiLCRUU2o3aSpPEzVCt2o
This commit is contained in:
@@ -67,7 +67,7 @@ self-test 合成超大卡(不分段 est=114 會炸)→ 自動分 4 段、每
|
|||||||
## 待 live 部署 + 寫入(總管過 leo 閘用)
|
## 待 live 部署 + 寫入(總管過 leo 閘用)
|
||||||
|
|
||||||
1. **部署通用 `code` 零件**(Arcrun#10 分支 `feat/issue-10-code-component`,已就緒):`cd registry/components/code && npm install && npx wrangler deploy`(→ `code.arcrun.dev`)+ `register-component.sh code`。本 workflow 的 parse_card 以 `component: code` 引用它,解析 JS 已內聯在 workflow.yaml(= `lib/card-to-envelope.mjs` 邏輯)。不再部署 domain 零件 `km_wiki_card_parse`。
|
1. **部署通用 `code` 零件**(Arcrun#10 分支 `feat/issue-10-code-component`,已就緒):`cd registry/components/code && npm install && npx wrangler deploy`(→ `code.arcrun.dev`)+ `register-component.sh code`。本 workflow 的 parse_card 以 `component: code` 引用它,解析 JS 已內聯在 workflow.yaml(= `lib/card-to-envelope.mjs` 邏輯)。不再部署 domain 零件 `km_wiki_card_parse`。
|
||||||
2. **部署 workflow**:`km_wiki_ingest_drain`(cron drain);`wrangler` 直推 leo21c(**禁 `acr update`**——codeload 綁 GitHub 假綠,Arcrun#4)。
|
2. **部署 workflow runtime**:`km_wiki_ingest_drain` 的**無人值守執行體= `drainer/`**(Cloudflare Worker:cron drain + Gitea webhook delta,游標存 KBDB、復用 code/kbdb/graph);`wrangler` 直推 leo21c(**禁 `acr update`**)。已 live:`arcrun-km-wiki-drainer.leo21c.workers.dev`。部署/觸發/webhook 設定見 `drainer/README.md`。(workflow.yaml 為宣告式 spec;drainer 為其可執行 runtime。)
|
||||||
3. **注入環境變數**(不放 repo):`repo=Leo/notes ref=main gitea_token kbdb_url=https://arcrun-kbdb.leo21c.workers.dev kbdb_api_key graph_url graph_api_key=leo`。`CLOUDFLARE_ACCOUNT_ID=leo21c`(別讓官方 58309b 污染)。
|
3. **注入環境變數**(不放 repo):`repo=Leo/notes ref=main gitea_token kbdb_url=https://arcrun-kbdb.leo21c.workers.dev kbdb_api_key graph_url graph_api_key=leo`。`CLOUDFLARE_ACCOUNT_ID=leo21c`(別讓官方 58309b 污染)。
|
||||||
4. **entry 寫入路徑確認**:若 `kbdb_upsert_block` 尚不透傳 `metadata_json`(需 `embed:true`/`content_hash`),entry 改用 `http_request` 直打 base `POST/PATCH /entries` 帶 `body_json.metadata_json`。
|
4. **entry 寫入路徑確認**:若 `kbdb_upsert_block` 尚不透傳 `metadata_json`(需 `embed:true`/`content_hash`),entry 改用 `http_request` 直打 base `POST/PATCH /entries` 帶 `body_json.metadata_json`。
|
||||||
5. **預期寫入量(Leo/notes 現況 3 卡)**:3 entries + 15 triplets + 16 node records(去重後更少);分 3 次 graph 呼叫(每次 ≤ 33 subrequest)+ 3 次 entry upsert。全庫鋪開時照 cron 一卡一 tick 慢推。
|
5. **預期寫入量(Leo/notes 現況 3 卡)**:3 entries + 15 triplets + 16 node records(去重後更少);分 3 次 graph 呼叫(每次 ≤ 33 subrequest)+ 3 次 entry upsert。全庫鋪開時照 cron 一卡一 tick 慢推。
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
.dev.vars
|
||||||
|
node_modules/
|
||||||
|
.wrangler/
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# arcrun-km-wiki-drainer —— km_wiki_ingest_drain 的無人值守 runtime
|
||||||
|
|
||||||
|
`workflow.yaml` 是宣告式 spec;**本 Worker 是它可真部署、有游標/觸發的執行體**。
|
||||||
|
復用已 live 的零件與服務,**不改 cypher-executor**:
|
||||||
|
|
||||||
|
```
|
||||||
|
Gitea 抓卡 md → arcrun-code 零件(sandbox inline JS 解析)→ /entries 冪等 upsert(embed:true)
|
||||||
|
→ graph /triplets/ingest(server 側 per-source 冪等)
|
||||||
|
```
|
||||||
|
|
||||||
|
- Live:`https://arcrun-km-wiki-drainer.leo21c.workers.dev`
|
||||||
|
- 解析 JS=`src/usercode.ts`(由 `workflow.yaml` 的 `parse_card.code` 抽出,單一真相源 `lib/card-to-envelope.mjs`)。
|
||||||
|
|
||||||
|
## 兩觸發模式(共用同一 `processCard` 核心)
|
||||||
|
|
||||||
|
| 模式 | 觸發 | 行為 |
|
||||||
|
|---|---|---|
|
||||||
|
| **Phase 0 cron drain** | cron tick(見「開自走」)| `drainBatch(BATCH_SIZE)`:Gitea tree 列卡 → 游標之後取一小批 → 逐卡 `processCard` → 進游標(到底**回捲**重掃)。壓 CF subrequest 下、撐得住 5,083 檔(每 tick 小批)。 |
|
||||||
|
| **穩態 webhook** | Gitea push webhook → `POST /webhook` | 只處理本次 commit 動到的 `system-dev/wiki/cards/**.md`(delta),不重掃全庫。**Gitea→Cloudflare,非 GitHub Actions(不觸 D20)**。 |
|
||||||
|
|
||||||
|
**冪等**:未改卡的 entry(`page_name`+`content_hash`)與 triplet(`source.uri`+`content_hash`)皆 skip;只有新/改卡真寫。游標存 KBDB 一個 `ingest_cursor` entry(走 API、零建表)。
|
||||||
|
|
||||||
|
## HTTP 路由
|
||||||
|
|
||||||
|
| 方法 | 路徑 | 用途 |
|
||||||
|
|---|---|---|
|
||||||
|
| GET | `/` | 健康 |
|
||||||
|
| GET | `/cursor` | 看游標狀態 |
|
||||||
|
| POST | `/drain?batch=N` | 手動觸發一批 drain(= 一個 cron tick;驗收/補跑用) |
|
||||||
|
| POST | `/webhook` | Gitea push webhook 入口(delta) |
|
||||||
|
| POST | `/cursor/reset` | 游標歸零(重新全 drain) |
|
||||||
|
|
||||||
|
## ⚠️ 非-auto(預設)
|
||||||
|
|
||||||
|
`wrangler.toml` 的 `[triggers] crons` **預設註解掉**——不自走排程、不無人值守亂寫。
|
||||||
|
drain 以 `POST /drain` 手動觸發(與 cron `scheduled` handler 同核心)。
|
||||||
|
要開自走 Phase 0 backfill:leo 過閘後取消 `wrangler.toml` 那兩行註解再 deploy。
|
||||||
|
|
||||||
|
## 部署(wrangler 直推 leo21c,禁 acr update)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd registry/examples/km-wiki-ingest/drainer
|
||||||
|
npm install
|
||||||
|
npx wrangler deploy
|
||||||
|
# ★ 必須:設 Gitea 讀取權(Leo/notes 為 private repo)——此步需 leo 在互動模式跑(secret 寫入)
|
||||||
|
printf '%s' "$GITEA_TOKEN" | npx wrangler secret put GITEA_TOKEN
|
||||||
|
# 選填:webhook 簽章驗證
|
||||||
|
# printf '%s' "$WEBHOOK_SECRET" | npx wrangler secret put GITEA_WEBHOOK_SECRET
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gitea webhook 設定(穩態 delta)
|
||||||
|
|
||||||
|
`Leo/notes` → Settings → Webhooks → Add Webhook → Gitea:
|
||||||
|
- Target URL:`https://arcrun-km-wiki-drainer.leo21c.workers.dev/webhook`
|
||||||
|
- HTTP Method:POST,Content-Type:`application/json`
|
||||||
|
- Trigger:Push events(可只限 `main`)
|
||||||
|
- Secret:若設,同步 `GITEA_WEBHOOK_SECRET` secret(本 Worker 驗 `X-Gitea-Signature` HMAC-SHA256)
|
||||||
|
- Gitea 的 ping event → Worker 回 `{ok:true,pong:true}`(測連通)
|
||||||
|
|
||||||
|
## config(wrangler.toml [vars])
|
||||||
|
|
||||||
|
`REPO=Leo/notes REF=main OWNER=leo CARDS_ROOT=system-dev/wiki/cards BATCH_SIZE=2`
|
||||||
|
`CODE_WORKER_URL / KBDB_URL / GRAPH_URL / GRAPH_KEY`(皆 leo21c live 服務)。
|
||||||
|
|
||||||
|
## 規模備註(5,083 檔)
|
||||||
|
|
||||||
|
- 列卡走 Gitea `git/trees?recursive=true`(一次全樹);若 `truncated`,回應標記 `truncated:true`,
|
||||||
|
屆時換 tree 分頁列法(現況 notes 卡數小、不截斷)。
|
||||||
|
- 每 tick 只 `BATCH_SIZE` 張,游標續傳;到底回捲 → 全庫變更靠冪等於下一輪捕捉;
|
||||||
|
即時變更走 webhook。
|
||||||
+1528
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"name": "arcrun-km-wiki-drainer",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"description": "km_wiki_ingest_drain 的無人值守 runtime(cron drain + Gitea webhook delta)",
|
||||||
|
"scripts": { "deploy": "wrangler deploy" },
|
||||||
|
"devDependencies": {
|
||||||
|
"@cloudflare/workers-types": "^4.20250408.0",
|
||||||
|
"typescript": "^5.7.0",
|
||||||
|
"wrangler": "^4.0.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,293 @@
|
|||||||
|
/**
|
||||||
|
* arcrun km_wiki_ingest_drainer —— 無人值守卡片 ingest 編排 Worker(Phase b)
|
||||||
|
*
|
||||||
|
* 這是 workflow `km_wiki_ingest_drain` 的「具體 runtime」:workflow.yaml 是宣告式 spec,
|
||||||
|
* 本 Worker 是可真部署、有游標/觸發的執行體。復用已 live 的零件與服務,不動 cypher-executor:
|
||||||
|
* Gitea 抓卡 md → arcrun-code 零件(sandbox inline JS 解析)→ /entries 冪等 upsert(embed:true)
|
||||||
|
* → graph /triplets/ingest(server 側 per-source 冪等)。
|
||||||
|
*
|
||||||
|
* 兩觸發模式共用同一 card-processing 核心(processCard):
|
||||||
|
* (A) Phase 0 cron drain:cron tick → drainBatch(BATCH_SIZE):列卡(Gitea tree recursive)→
|
||||||
|
* 游標(存 KBDB 一個 cursor entry)之後取一小批 → 逐卡 processCard → 進游標(到底回捲)。
|
||||||
|
* 冪等:未改卡的 entry(content_hash)/triplet(uri+hash) 皆 skip,只有新/改卡真寫。
|
||||||
|
* (B) 穩態 webhook:Gitea push webhook(非 GitHub Actions,不觸 D20)→ 只處理 delta(本次
|
||||||
|
* commit 動到的 system-dev/wiki/cards/**.md)→ 逐卡 processCard。不重掃全庫。
|
||||||
|
*
|
||||||
|
* HTTP 路由(fetch):
|
||||||
|
* GET / 健康
|
||||||
|
* GET /cursor 看游標狀態
|
||||||
|
* POST /drain 手動觸發一批 drain(= 一個 cron tick;驗收/補跑用)
|
||||||
|
* POST /webhook Gitea push webhook 入口(delta 模式)
|
||||||
|
* POST /cursor/reset 游標歸零(重新全 drain 用)
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { CARD_TO_ENVELOPE_USERCODE } from './usercode';
|
||||||
|
|
||||||
|
export interface Env {
|
||||||
|
REPO: string; // Leo/notes
|
||||||
|
REF: string; // main
|
||||||
|
OWNER: string; // leo(KBDB 租戶 owner_id)
|
||||||
|
CARDS_ROOT: string; // system-dev/wiki/cards
|
||||||
|
CODE_WORKER_URL: string; // https://arcrun-code.leo21c.workers.dev/
|
||||||
|
KBDB_URL: string; // https://arcrun-kbdb.leo21c.workers.dev
|
||||||
|
GRAPH_URL: string; // https://kbdb-graph-plugin.leo21c.workers.dev
|
||||||
|
GRAPH_KEY: string; // leo
|
||||||
|
BATCH_SIZE: string; // "2"
|
||||||
|
GITEA_TOKEN: string; // secret:讀 repo
|
||||||
|
GITEA_WEBHOOK_SECRET?: string; // secret(選填):驗 webhook 簽章
|
||||||
|
}
|
||||||
|
|
||||||
|
const CURSOR_TYPE = 'ingest_cursor';
|
||||||
|
const cursorPageName = (env: Env) => `cursor:km_wiki_ingest_drain:${env.REPO}`;
|
||||||
|
|
||||||
|
// ── Gitea ────────────────────────────────────────────────────────────────────
|
||||||
|
const giteaApi = (env: Env, path: string) => `https://git.uncle6.me/api/v1/repos/${env.REPO}${path}`;
|
||||||
|
|
||||||
|
/** 列出 CARDS_ROOT 下所有卡片(git tree recursive,排除 00-INDEX/.gitkeep),回 [{path, sha}] 已排序。 */
|
||||||
|
async function listCards(env: Env): Promise<{ path: string; sha: string }[]> {
|
||||||
|
const r = await fetch(giteaApi(env, `/git/trees/${encodeURIComponent(env.REF)}?recursive=true`), {
|
||||||
|
headers: { Authorization: `token ${env.GITEA_TOKEN}` },
|
||||||
|
});
|
||||||
|
if (!r.ok) throw new Error(`gitea tree ${r.status}: ${(await r.text()).slice(0, 200)}`);
|
||||||
|
const data = (await r.json()) as { tree?: { path: string; type: string; sha: string }[]; truncated?: boolean };
|
||||||
|
const root = env.CARDS_ROOT.replace(/\/$/, '') + '/';
|
||||||
|
const cards = (data.tree || [])
|
||||||
|
.filter((e) => e.type === 'blob' && e.path.startsWith(root) && e.path.endsWith('.md'))
|
||||||
|
.filter((e) => {
|
||||||
|
const base = e.path.split('/').pop() || '';
|
||||||
|
return base !== '.gitkeep' && !base.startsWith('00-INDEX');
|
||||||
|
})
|
||||||
|
.map((e) => ({ path: e.path, sha: e.sha }))
|
||||||
|
.sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0));
|
||||||
|
// truncated:5,083 檔規模 Gitea tree 可能截斷 → 記錄以便換分頁列法(現況 notes 卡數小,不截斷)。
|
||||||
|
(cards as unknown as { truncated?: boolean }).truncated = data.truncated === true;
|
||||||
|
return cards;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function giteaRaw(env: Env, relPath: string): Promise<string> {
|
||||||
|
const url = giteaApi(env, `/raw/${relPath.split('/').map(encodeURIComponent).join('/')}?ref=${encodeURIComponent(env.REF)}`);
|
||||||
|
const r = await fetch(url, { headers: { Authorization: `token ${env.GITEA_TOKEN}` } });
|
||||||
|
if (!r.ok) throw new Error(`gitea raw ${relPath} ${r.status}`);
|
||||||
|
return r.text();
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── card-processing 核心 ──────────────────────────────────────────────────────
|
||||||
|
async function parseCard(env: Env, md: string, relPath: string): Promise<any> {
|
||||||
|
const r = await fetch(env.CODE_WORKER_URL, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'content-type': 'application/json' },
|
||||||
|
body: JSON.stringify({
|
||||||
|
code: CARD_TO_ENVELOPE_USERCODE,
|
||||||
|
input: { md, relPath, repo: env.REPO, opts: { budget: 40 } },
|
||||||
|
limits: { timeout_ms: 3000, max_output_bytes: 4 * 1024 * 1024 },
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
const out = (await r.json()) as { success: boolean; data?: any; error?: string };
|
||||||
|
if (!out.success) throw new Error(`code 零件解析失敗 ${relPath}: ${out.error}`);
|
||||||
|
return out.data; // { entry, envelopes, meta, nodeCount, tripletCount }
|
||||||
|
}
|
||||||
|
|
||||||
|
function metaHash(entry: any): string | undefined {
|
||||||
|
try {
|
||||||
|
return typeof entry.metadata_json === 'string' ? JSON.parse(entry.metadata_json).content_hash : undefined;
|
||||||
|
} catch {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function upsertEntry(env: Env, entry: any): Promise<{ action: string; id?: string }> {
|
||||||
|
const pageName = entry.page_name;
|
||||||
|
const metadata_json = JSON.stringify(entry.metadata);
|
||||||
|
const newHash = entry.metadata.content_hash;
|
||||||
|
const lookup = await fetch(
|
||||||
|
`${env.KBDB_URL}/entries?page_name=${encodeURIComponent(pageName)}&owner_id=${encodeURIComponent(env.OWNER)}`,
|
||||||
|
);
|
||||||
|
const existing = ((await lookup.json()) as { entries?: any[] }).entries?.[0];
|
||||||
|
if (existing) {
|
||||||
|
if (metaHash(existing) === newHash) return { action: 'skip', id: existing.id };
|
||||||
|
const r = await fetch(`${env.KBDB_URL}/entries/${existing.id}`, {
|
||||||
|
method: 'PATCH',
|
||||||
|
headers: { 'content-type': 'application/json' },
|
||||||
|
body: JSON.stringify({ content: entry.content, tags_json: JSON.stringify(entry.tags || []), metadata_json }),
|
||||||
|
});
|
||||||
|
return { action: 'patch', id: existing.id, ...(r.ok ? {} : { action: 'patch-failed' }) };
|
||||||
|
}
|
||||||
|
const r = await fetch(`${env.KBDB_URL}/entries`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'content-type': 'application/json' },
|
||||||
|
body: JSON.stringify({
|
||||||
|
entry_type: entry.entry_type,
|
||||||
|
content: entry.content,
|
||||||
|
page_name: pageName,
|
||||||
|
owner_id: env.OWNER,
|
||||||
|
tags_json: JSON.stringify(entry.tags || []),
|
||||||
|
source: entry.metadata.source,
|
||||||
|
metadata_json,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
const out = (await r.json()) as { entry?: { id?: string } };
|
||||||
|
return { action: 'create', id: out.entry?.id };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function postEnvelopes(env: Env, envelopes: any[]): Promise<any[]> {
|
||||||
|
const results = [];
|
||||||
|
for (const envp of envelopes) {
|
||||||
|
// envelopes 已由 code 節點剝除 _-鍵;此處再保險剝一次。
|
||||||
|
const clean = Object.fromEntries(Object.entries(envp).filter(([k]) => !k.startsWith('_')));
|
||||||
|
const r = await fetch(`${env.GRAPH_URL}/triplets/ingest?owner_id=${encodeURIComponent(env.OWNER)}`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'content-type': 'application/json', 'X-Arcrun-API-Key': env.GRAPH_KEY },
|
||||||
|
body: JSON.stringify(clean),
|
||||||
|
});
|
||||||
|
const text = await r.text();
|
||||||
|
let body: any;
|
||||||
|
try { body = JSON.parse(text); } catch { body = text; }
|
||||||
|
results.push({ status: r.status, uri: (envp as any).source?.uri, triplets: (envp as any).triplets?.length, body });
|
||||||
|
}
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 一張卡的完整處理(冪等):fetch → parse → upsert entry → post envelopes。 */
|
||||||
|
async function processCard(env: Env, relPath: string): Promise<any> {
|
||||||
|
const md = await giteaRaw(env, relPath);
|
||||||
|
const plan = await parseCard(env, md, relPath);
|
||||||
|
const entry = await upsertEntry(env, plan.entry);
|
||||||
|
const envelopes = await postEnvelopes(env, plan.envelopes);
|
||||||
|
const wrote = entry.action !== 'skip' || envelopes.some((e) => e.body && e.body.skipped === false);
|
||||||
|
return {
|
||||||
|
relPath,
|
||||||
|
page_name: plan.entry.page_name,
|
||||||
|
content_hash: (plan.entry.metadata.content_hash || '').slice(0, 12),
|
||||||
|
entry: entry.action,
|
||||||
|
envelopes: envelopes.map((e) => ({ status: e.status, triplets: e.triplets, skipped: e.body?.skipped, ingested: e.body?.ingested })),
|
||||||
|
wrote,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── 游標(存 KBDB 一個 ingest_cursor entry;不 embed)─────────────────────────
|
||||||
|
interface CursorState { last_path: string; cycle: number; processed_total: number; updated_at: number; }
|
||||||
|
|
||||||
|
async function getCursor(env: Env): Promise<{ id?: string; state: CursorState }> {
|
||||||
|
const r = await fetch(
|
||||||
|
`${env.KBDB_URL}/entries?page_name=${encodeURIComponent(cursorPageName(env))}&owner_id=${encodeURIComponent(env.OWNER)}`,
|
||||||
|
);
|
||||||
|
const e = ((await r.json()) as { entries?: any[] }).entries?.[0];
|
||||||
|
if (e) {
|
||||||
|
try { return { id: e.id, state: JSON.parse(e.content) }; } catch { /* fallthrough */ }
|
||||||
|
return { id: e.id, state: { last_path: '', cycle: 0, processed_total: 0, updated_at: 0 } };
|
||||||
|
}
|
||||||
|
return { state: { last_path: '', cycle: 0, processed_total: 0, updated_at: 0 } };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function setCursor(env: Env, cur: { id?: string; state: CursorState }): Promise<void> {
|
||||||
|
const content = JSON.stringify(cur.state);
|
||||||
|
if (cur.id) {
|
||||||
|
await fetch(`${env.KBDB_URL}/entries/${cur.id}`, {
|
||||||
|
method: 'PATCH', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ content }),
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
await fetch(`${env.KBDB_URL}/entries`, {
|
||||||
|
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||||
|
body: JSON.stringify({ entry_type: CURSOR_TYPE, page_name: cursorPageName(env), owner_id: env.OWNER, content,
|
||||||
|
metadata_json: JSON.stringify({ kind: 'ingest_cursor', embed: false }) }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** cron / 手動觸發:drain 一小批。游標之後取 BATCH 張;到底回捲(continuous drain)。 */
|
||||||
|
async function drainBatch(env: Env, batchSize: number): Promise<any> {
|
||||||
|
const cards = await listCards(env);
|
||||||
|
const truncated = (cards as unknown as { truncated?: boolean }).truncated;
|
||||||
|
const cur = await getCursor(env);
|
||||||
|
let idx = cards.findIndex((c) => c.path > cur.state.last_path);
|
||||||
|
let wrapped = false;
|
||||||
|
if (idx < 0) { idx = 0; wrapped = true; } // 到底 → 回捲重掃(靠冪等,未改卡 cheap skip)
|
||||||
|
const batch = cards.slice(idx, idx + batchSize);
|
||||||
|
const results = [];
|
||||||
|
for (const c of batch) results.push(await processCard(env, c.path));
|
||||||
|
const newLast = batch.length ? batch[batch.length - 1].path : cur.state.last_path;
|
||||||
|
cur.state = {
|
||||||
|
last_path: newLast,
|
||||||
|
cycle: cur.state.cycle + (wrapped ? 1 : 0),
|
||||||
|
processed_total: cur.state.processed_total + batch.length,
|
||||||
|
updated_at: Math.floor(Date.now() / 1000),
|
||||||
|
};
|
||||||
|
await setCursor(env, cur);
|
||||||
|
return {
|
||||||
|
mode: 'cron-drain', total_cards: cards.length, truncated: !!truncated, wrapped,
|
||||||
|
batch: batch.map((c) => c.path), results,
|
||||||
|
wrote: results.filter((r) => r.wrote).length, skipped: results.filter((r) => !r.wrote).length,
|
||||||
|
cursor: cur.state,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── webhook(Gitea push → delta)───────────────────────────────────────────────
|
||||||
|
async function verifyGiteaSig(env: Env, raw: string, sig: string | null): Promise<boolean> {
|
||||||
|
if (!env.GITEA_WEBHOOK_SECRET) return true; // 未設 secret → 不驗(leo 自有 repo)
|
||||||
|
if (!sig) return false;
|
||||||
|
const key = await crypto.subtle.importKey('raw', new TextEncoder().encode(env.GITEA_WEBHOOK_SECRET),
|
||||||
|
{ name: 'HMAC', hash: 'SHA-256' }, false, ['sign']);
|
||||||
|
const mac = await crypto.subtle.sign('HMAC', key, new TextEncoder().encode(raw));
|
||||||
|
const hex = [...new Uint8Array(mac)].map((b) => b.toString(16).padStart(2, '0')).join('');
|
||||||
|
return hex === sig;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleWebhook(env: Env, raw: string): Promise<any> {
|
||||||
|
const payload = JSON.parse(raw) as { commits?: { added?: string[]; modified?: string[]; removed?: string[] }[] };
|
||||||
|
const root = env.CARDS_ROOT.replace(/\/$/, '') + '/';
|
||||||
|
const isCard = (p: string) => p.startsWith(root) && p.endsWith('.md') && !(p.split('/').pop() || '').startsWith('00-INDEX');
|
||||||
|
const changed = new Set<string>();
|
||||||
|
const removed = new Set<string>();
|
||||||
|
for (const cm of payload.commits || []) {
|
||||||
|
for (const p of [...(cm.added || []), ...(cm.modified || [])]) if (isCard(p)) changed.add(p);
|
||||||
|
for (const p of cm.removed || []) if (isCard(p)) removed.add(p);
|
||||||
|
}
|
||||||
|
const results = [];
|
||||||
|
for (const p of changed) results.push(await processCard(env, p));
|
||||||
|
return {
|
||||||
|
mode: 'webhook-delta', changed: [...changed], removed: [...removed],
|
||||||
|
note: removed.size ? 'removed 卡的清理未實作(graph 有 deprecate 機制,另議)' : undefined,
|
||||||
|
results, wrote: results.filter((r) => r.wrote).length,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Worker 入口 ────────────────────────────────────────────────────────────────
|
||||||
|
export default {
|
||||||
|
async scheduled(_event: ScheduledEvent, env: Env, ctx: ExecutionContext): Promise<void> {
|
||||||
|
ctx.waitUntil(drainBatch(env, Number(env.BATCH_SIZE || '2')).then(() => {}).catch(() => {}));
|
||||||
|
},
|
||||||
|
|
||||||
|
async fetch(req: Request, env: Env, _ctx: ExecutionContext): Promise<Response> {
|
||||||
|
const url = new URL(req.url);
|
||||||
|
const json = (o: unknown, s = 200) => new Response(JSON.stringify(o, null, 2), { status: s, headers: { 'content-type': 'application/json' } });
|
||||||
|
try {
|
||||||
|
if (req.method === 'GET' && url.pathname === '/') {
|
||||||
|
return json({ ok: true, service: 'km_wiki_ingest_drainer', repo: env.REPO, cards_root: env.CARDS_ROOT, batch: env.BATCH_SIZE });
|
||||||
|
}
|
||||||
|
if (req.method === 'GET' && url.pathname === '/cursor') {
|
||||||
|
return json(await getCursor(env));
|
||||||
|
}
|
||||||
|
if (req.method === 'POST' && url.pathname === '/cursor/reset') {
|
||||||
|
const cur = await getCursor(env);
|
||||||
|
cur.state = { last_path: '', cycle: 0, processed_total: 0, updated_at: Math.floor(Date.now() / 1000) };
|
||||||
|
await setCursor(env, cur);
|
||||||
|
return json({ ok: true, cursor: cur.state });
|
||||||
|
}
|
||||||
|
if (req.method === 'POST' && url.pathname === '/drain') {
|
||||||
|
const n = Number(url.searchParams.get('batch') || env.BATCH_SIZE || '2');
|
||||||
|
return json(await drainBatch(env, n));
|
||||||
|
}
|
||||||
|
if (req.method === 'POST' && url.pathname === '/webhook') {
|
||||||
|
const raw = await req.text();
|
||||||
|
const ok = await verifyGiteaSig(env, raw, req.headers.get('X-Gitea-Signature'));
|
||||||
|
if (!ok) return json({ error: 'bad signature' }, 401);
|
||||||
|
// Gitea ping event(測試)→ 回 pong
|
||||||
|
if (req.headers.get('X-Gitea-Event') === 'ping') return json({ ok: true, pong: true });
|
||||||
|
return json(await handleWebhook(env, raw));
|
||||||
|
}
|
||||||
|
return json({ error: 'not found' }, 404);
|
||||||
|
} catch (e) {
|
||||||
|
return json({ error: e instanceof Error ? e.message : String(e) }, 500);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022",
|
||||||
|
"module": "ES2022",
|
||||||
|
"moduleResolution": "Bundler",
|
||||||
|
"lib": ["ES2022"],
|
||||||
|
"types": ["@cloudflare/workers-types"],
|
||||||
|
"strict": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"noEmit": true,
|
||||||
|
"resolveJsonModule": true
|
||||||
|
},
|
||||||
|
"include": ["src/**/*.ts"]
|
||||||
|
}
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
name = "arcrun-km-wiki-drainer"
|
||||||
|
main = "src/index.ts"
|
||||||
|
compatibility_date = "2025-02-19"
|
||||||
|
workers_dev = true
|
||||||
|
|
||||||
|
# ⚠️ 維持「非-auto」:預設不註冊 cron trigger(不自走排程、不無人值守亂寫)。
|
||||||
|
# drain 以 POST /drain 手動觸發(與 scheduled handler 同核心)。
|
||||||
|
# 要開自走 Phase 0 backfill:leo 過閘後,取消下面兩行註解再 deploy(scheduled handler 已在 code)。
|
||||||
|
# [triggers]
|
||||||
|
# crons = ["*/5 * * * *"]
|
||||||
|
|
||||||
|
[vars]
|
||||||
|
REPO = "Leo/notes"
|
||||||
|
REF = "main"
|
||||||
|
OWNER = "leo"
|
||||||
|
CARDS_ROOT = "system-dev/wiki/cards"
|
||||||
|
CODE_WORKER_URL = "https://arcrun-code.leo21c.workers.dev/"
|
||||||
|
KBDB_URL = "https://arcrun-kbdb.leo21c.workers.dev"
|
||||||
|
GRAPH_URL = "https://kbdb-graph-plugin.leo21c.workers.dev"
|
||||||
|
GRAPH_KEY = "leo"
|
||||||
|
BATCH_SIZE = "2"
|
||||||
|
# secret(wrangler secret put):GITEA_TOKEN(必),GITEA_WEBHOOK_SECRET(選)
|
||||||
Reference in New Issue
Block a user