117 lines
3.9 KiB
Markdown
117 lines
3.9 KiB
Markdown
# Methodology
|
|
|
|
## Approach
|
|
|
|
Organ Architecture treats trained AI models as biological organisms with
|
|
transplantable parts. Instead of retraining from scratch (costs billions),
|
|
we perform post-training surgery: extract, measure, graft, reassemble.
|
|
|
|
## Step 1: Extraction (organ_extract.py)
|
|
|
|
Parse GGUF binary format directly:
|
|
- Read magic number, version, metadata, tensor info
|
|
- Classify each tensor by name pattern into anatomical types
|
|
- Extract each tensor as independent .bin file with header
|
|
- Generate manifest.json mapping the full anatomy
|
|
|
|
Classification rules:
|
|
- `attn_q`, `attn_k`, `attn_v`, `attn_output` → skeleton
|
|
- `ffn_gate`, `ffn_up`, `ffn_down` → organ
|
|
- `token_embd`, `output.weight` → embed
|
|
- `*_norm` → norm
|
|
- `lora_*` → adapter
|
|
|
|
## Step 2: Measurement (organ_measure.py)
|
|
|
|
Z-measure: Z = dI/d(log s) * exp(i*theta)
|
|
|
|
For each tensor, sample up to 100,000 values and compute:
|
|
|
|
1. **Entropy** (information density):
|
|
- Histogram-based Shannon entropy
|
|
- Normalized to [0, 1] against maximum entropy
|
|
- High entropy (>0.95) = uniform = noise
|
|
- Moderate entropy (0.3-0.7) = structured information
|
|
|
|
2. **Kurtosis** (structure):
|
|
- Fourth standardized moment minus 3
|
|
- High absolute kurtosis = sharp peaks = organized structure
|
|
- Near-zero = Gaussian-like = less organization
|
|
|
|
3. **Scale coherence** (CV of sorted diffs):
|
|
- Sort sampled values, compute differences
|
|
- Coefficient of variation of these differences
|
|
- High CV = non-uniform spacing = structured signal
|
|
- Low CV = uniform spacing = noise
|
|
|
|
Combined score → theta in [0, 90] degrees.
|
|
|
|
## Step 3: Purification (organ_purify_v2.py)
|
|
|
|
Fractal signal extraction via Haar wavelets:
|
|
|
|
1. Pad tensor to power-of-2 length
|
|
2. Haar wavelet decomposition across N scales
|
|
3. At each scale: approximation + detail coefficients
|
|
4. Cross-scale coherence check:
|
|
- Compare energy at scale s with energy at scale 2s
|
|
- High coherence (pattern exists at both scales) = fractal = signal
|
|
- Low coherence (pattern at one scale only) = noise
|
|
5. Attenuate incoherent components (noise)
|
|
6. Reconstruct from coherent components (signal)
|
|
7. Restore original scale (mean/std preservation)
|
|
|
|
This directly implements dI/d(log s): information that persists across
|
|
logarithmic scales is the signal. Everything else is training artifact.
|
|
|
|
## Step 4: Grafting (organ_graft.py, transplant_935.py)
|
|
|
|
Two methods:
|
|
|
|
### Via .bin intermediaries (organ_graft.py)
|
|
1. Extract both source and target models to organ directories
|
|
2. Match tensors by layer number and type suffix
|
|
3. Verify dimensional compatibility
|
|
4. Copy matching .bin files from donor to recipient directory
|
|
5. Update manifest
|
|
|
|
### Direct GGUF-to-GGUF (transplant_935.py)
|
|
1. Parse both GGUF headers to get tensor name/offset/size maps
|
|
2. Copy base GGUF entirely as starting point
|
|
3. For each FFN tensor in base that has a matching donor tensor:
|
|
- Verify exact byte size match
|
|
- Seek to donor tensor data, read
|
|
- Seek to base tensor offset in output, overwrite
|
|
4. Result: valid GGUF with patched FFN layers
|
|
|
|
Direct method is faster and avoids header format issues.
|
|
|
|
## Step 5: Assembly (organ_assemble.py)
|
|
|
|
Reconstruct GGUF from organ directory:
|
|
1. Read manifest for metadata and tensor ordering
|
|
2. Write GGUF header (magic, version, n_tensors, n_metadata)
|
|
3. Write metadata key-value pairs
|
|
4. Write tensor info (name, dims, dtype, offset) with 32-byte alignment
|
|
5. Write tensor data with padding
|
|
6. Result: standard GGUF loadable by any compatible runtime
|
|
|
|
## Step 6: Validation
|
|
|
|
Run chimera through InferenceX:
|
|
- Load GGUF, validate all tensors
|
|
- Initialize transformer (attention, KV cache, kernel dispatch)
|
|
- Run inference with chat template
|
|
- Verify coherent output
|
|
|
|
## Key Finding
|
|
|
|
Graft success depends on architectural proximity:
|
|
- Same family (Qwen2 base) + same scale (14B) = coherent output
|
|
- Same family + different scale (7B) = PAD token failure
|
|
- The latent space alignment is implicit in shared training lineage
|
|
|
|
## Signature
|
|
|
|
935
|