171 lines
4.0 KiB
Markdown
171 lines
4.0 KiB
Markdown
# MAGATAMA Lane-Specific RunPod Adoption + Versioning
|
|
|
|
Date: 2026-05-07
|
|
|
|
## Scope
|
|
|
|
Harden MAGATAMA training automation for:
|
|
|
|
- `magatamallm`
|
|
- `fo_blogllm`
|
|
- `tip_llm`
|
|
|
|
Goal:
|
|
|
|
- lane-specific training pools remain isolated
|
|
- RunPod `COMPLETED` counts only when model return/adoption is real
|
|
- active lane model gets a new release/version marker after successful adoption
|
|
- dashboard status and errors remain truthful
|
|
|
|
## Problem
|
|
|
|
The data/build side of training already worked:
|
|
|
|
- lane-specific RunPod datasets were built
|
|
- RunPod jobs were submitted
|
|
- registry often showed `IN_PROGRESS` / `COMPLETED`
|
|
|
|
But the end of the chain remained weak:
|
|
|
|
1. adoption/version truth still depended on one shared:
|
|
- `~/magatama-llm/fine-tuning/last_run.json`
|
|
2. multiple lanes could therefore overwrite the same success marker
|
|
3. the modal could degrade late-stream adoption failures into a generic `network error`
|
|
4. the user requirement was stricter:
|
|
- training pool -> RunPod -> artifact -> local import -> version bump -> active alias switch
|
|
- all fully automatic
|
|
|
|
## Code changes made locally
|
|
|
|
### 1. Lane-specific last-run metadata
|
|
|
|
File:
|
|
|
|
- `magatama/packages/fine-tuner/training_api.py`
|
|
|
|
Added:
|
|
|
|
- `lane_last_run_file(lane)`
|
|
|
|
Resulting files:
|
|
|
|
- `~/magatama-llm/fine-tuning/magatamallm-last_run.json`
|
|
- `~/magatama-llm/fine-tuning/fo_blogllm-last_run.json`
|
|
- `~/magatama-llm/fine-tuning/tip_llm-last_run.json`
|
|
|
|
Compatibility:
|
|
|
|
- `magatamallm` still mirrors to legacy:
|
|
- `~/magatama-llm/fine-tuning/last_run.json`
|
|
|
|
### 2. Automatic release alias / version step
|
|
|
|
File:
|
|
|
|
- `magatama/packages/fine-tuner/training_api.py`
|
|
|
|
Added:
|
|
|
|
- `next_release_metadata(lane, active_model)`
|
|
- release alias creation
|
|
|
|
New adoption sequence:
|
|
|
|
1. RunPod artifact imported to candidate model
|
|
2. candidate smoke tests pass
|
|
3. release alias is created:
|
|
- example shape: `<active-alias>-rN`
|
|
4. stable active alias is repointed to that release alias
|
|
|
|
This means the lane now receives a concrete new release/version marker after successful adoption.
|
|
|
|
### 3. Dashboard lane status truth
|
|
|
|
File:
|
|
|
|
- `magatama/packages/dashboard/src/server.ts`
|
|
|
|
Changed:
|
|
|
|
- `/api/llm/status` now reads lane-specific last-run metadata first
|
|
- `release_alias` is preferred as visible model version
|
|
- this prevents one lane from falsely inheriting another lane's last successful run marker
|
|
|
|
### 4. Truthful RunPod terminal failure messaging
|
|
|
|
Files:
|
|
|
|
- `magatama/packages/dashboard/src/server.ts`
|
|
- `magatama/packages/dashboard/public/index-v2.html`
|
|
|
|
Changed:
|
|
|
|
- if RunPod says `COMPLETED` but:
|
|
- no model artifact exists
|
|
- no HF repo appears
|
|
- adoption fails
|
|
|
|
the UI now reports that exact reason instead of collapsing into a vague generic failure
|
|
|
|
Frontend hardening:
|
|
|
|
- avoid showing a misleading late `network error` after the server already emitted a terminal training event
|
|
- if the stream dies without a terminal event, the modal says so explicitly
|
|
|
|
### 5. Local training metrics future-proofed
|
|
|
|
File:
|
|
|
|
- `magatama/packages/fine-tuner/train.py`
|
|
|
|
Changed:
|
|
|
|
- metrics now also respect lane-specific last-run files via `TRAINING_LANE`
|
|
|
|
## Local verification
|
|
|
|
Passed:
|
|
|
|
- `python3 -m py_compile .../training_api.py .../train.py`
|
|
- `pnpm -C .../packages/dashboard build`
|
|
|
|
## Live deployment state
|
|
|
|
Not yet completed in this step.
|
|
|
|
Reason:
|
|
|
|
- direct Erik access failed during this block:
|
|
- `ssh: connect to host 82.165.222.127 port 22: Connection refused`
|
|
- later also `Operation timed out`
|
|
|
|
Therefore:
|
|
|
|
- the automation fix is locally ready
|
|
- but not yet verified live against the currently running:
|
|
- `tip_llm`
|
|
- `fo_blogllm`
|
|
|
|
## Operational next step
|
|
|
|
Once Erik SSH is reachable again:
|
|
|
|
1. deploy updated:
|
|
- `training_api.py`
|
|
- `train.py`
|
|
- dashboard build / server bundle
|
|
2. restart:
|
|
- `magatama-dashboard`
|
|
- Mac-side training API if used
|
|
3. verify lane-specific status:
|
|
- `tip_llm`
|
|
- `fo_blogllm`
|
|
- `magatamallm`
|
|
4. verify that a successful RunPod training now results in:
|
|
- artifact found
|
|
- adoption report present
|
|
- lane-specific `*-last_run.json`
|
|
- release alias incremented
|
|
- stable alias repointed
|
|
|