Agent optimization
Compare workflow optimization on the GEPA suite, skill selection on SkillsBench, and generated skill quality across systems.
Read the tech reportOptimization · 4 tasks · GPT-4.1 Mini
Workflow optimization on the GEPA suite
MEGA's workflow optimizer benchmarked against MIPROv2, TextGrad, GEPA, and Feedback Descent on four compound AI systems from GEPA's released eval code — multi-hop QA, instruction following, retrieval-based claim verification, and privacy-aware delegation. All methods use the same evaluation checkers and grading criteria.
Higher aggregate task score vs. GEPA
GEPA 69.52 → MEGA 76.55 · +7.03 points
| System | HotpotQA | IFBench | HoVer | PUPA | Aggregate |
|---|---|---|---|---|---|
| Baseline | 38.00 | 47.79 | 46.33 | 78.57 | 52.67 |
| MIPROv2 | 58.00 | 49.15 | 48.33 | 83.37 | 59.71 |
| TextGrad | 62.33 | 48.64 | 47.67 | 85.68 | 61.08 |
| Feedback Descent | 68.33 | 54.59 | 57.67 | 85.66 | 66.56 |
| GEPA | 69.00 | 55.95 | 56.67 | 96.46 | 69.52 |
| MEGA | 72.67 | 61.05 | 74.67 | 97.81 | 76.55 |
Aggregate is the mean of the four task scores, not a single-task result. Scores are test-split accuracy. MEGA train/validation/test sizes: HotpotQA 150/100/300, IFBench 150/100/294, HoVer 150/100/300, and PUPA 111/111/221. Baseline, MIPROv2, TextGrad, and GEPA scores are cited from the GEPA paper; Feedback Descent scores are cited from its authors. MEGA used smaller validation sets than GEPA's original splits (100 vs 300 for HotpotQA) to reduce optimization cost.
Read the tech reportSkillsBench · 84 tasks · 11 domains
Skill selection on SkillsBench
Compare skill selection methods on 84 tasks across 11 domains. Each condition uses Gemini 3 Flash, the same 4,207-asset skill pool, and the same tasks and verifiers. Results are checked with deterministic pytest assertions in isolated Docker containers.
Higher task success rate vs. no skills
Without skills 31.5% → MEGA (WG) 46.5% · +15.0 percentage points
| System | Pass Rate (%) | Avg Tokens/Task (k) | Curation Latency (s/task) | Efficiency (score/Mtok) |
|---|---|---|---|---|
| No Skills | 31.5 | 894 | — | 0.353 |
| AgentSkillOS | 41.1 | 1,189 | 403.4 | 0.345 |
| SkillNet | 41.7 | 983 | 37.8 | 0.424 |
| MEGA (WG) | 46.5 | 822 | 11.8 | 0.566 |
The 47.6% relative increase compares MEGA (WG) with the no-skill baseline, not SkillNet. SkillNet’s pass rate is 41.7%. Efficiency is pass rate per megatoken consumed (score/Mtok). Curation latency measures skill discovery and orchestration before execution. MEGA retrieves skills from a pre-indexed graph without LLM calls during retrieval.
Read the tech reportSkill generation
Generated skill quality
Compare token use, task scores, and the structure of generated skill files.
Five skill-generation systems and a no-skill baseline are evaluated on the same task harness. A separate implementation comparison covers MEGA Code and seven other systems.
View the evaluation repository5 skill-generation systems + baseline
Token use by system
MEGA Code generated skills from a 10-round FastAPI, React, and Gemini development session. The other generators received 1–2 sentence task prompts without detailed traces. Four skills per system were evaluated with HF Upskill’s harness: five test cases per skill, run on both Claude Sonnet and Haiku. The no-skill baseline is included for reference.
the tokens, same tasks
169K vs 897K
MEGA Code
0% vs baseline
897K
HF Upskill
0% vs baseline
897K
anthropic-skill-creator
0% vs baseline
897K
Baseline (No Skill)
reference897K
claude-code-skill-factory
+0% vs baseline
897K
skill-builder
+0% vs baseline
897K
Vertical line marks baseline (no skill). Bars exceeding baseline mean the system used more tokens than having no skill at all.
Combined task score
Mean score across all 8 runs per system (4 skills × 2 models). Each skill was scored on 5 test cases, measuring whether the generated skill correctly guided the AI agent to produce the expected output.
Combined (Sonnet + Haiku)
MEGA Code
169K tokens
0%
HF Upskill
763K tokens
0%
anthropic-skill-creator
826K tokens
0%
Baseline (No Skill)
897K tokens
0%
skill-builder
2,024K tokens
0%
claude-code-skill-factory
1,448K tokens
0%
Sonnet Only
MEGA Code
0%
HF Upskill
0%
Baseline
0%
anthropic-skill-creator
0%
skill-builder
0%
claude-code-skill-factory
0%
Haiku Only
MEGA Code
0%
HF Upskill
0%
anthropic-skill-creator
0%
Baseline
0%
skill-builder
0%
claude-code-skill-factory
0%
Generated file structure
Each cell marks circle (full), triangle (partial), or cross (absent) across 8 structural dimensions of generated skill files.
| Structural Element | MEGA Code | HF Upskill | skill-factory | skill-builder |
|---|---|---|---|---|
| Frontmatter completeness | ||||
| Trigger precision | ||||
| Preconditions | ||||
| Workflow specificity | ||||
| Rule reasoning (Why/Effect) | ||||
| Anti-pattern coverage | ||||
| Common Mistakes (why-it-happens) | ||||
| Success Indicators |
Skill-generation results
81% fewer tokens than baseline
MEGA Code used 169K tokens, compared with 897K without skills and 763K–2M for the other tested systems.
78% combined score
Across four skills and two models, MEGA Code scored 78%, compared with 65% for the no-skill baseline.
16/16 on the structural rubric
The generated files received full marks on all eight criteria, including preconditions, rule explanations, and success indicators.
Privacy filtering
Automated filtering covers eight categories of sensitive patterns before session data leaves the local machine.
Methodology
Test harness
HF Upskill eval
Models
Claude Sonnet & Claude Haiku
Test Cases
5 per skill, auto-generated by Opus via HF Upskill
Source Material
10-round full-stack dev session; competitors given minimal prompts only
Skills Evaluated
4 full-stack skills from a FastAPI + React + Gemini project
Systems
MEGA Code vs. HF Upskill, anthropic-skill-creator, claude-code-skill-factory, skill-builder, Baseline (no skill)
Implementation comparison · 8 systems
Inputs, automation, and quality checks
Compare the inputs, automation, strategy output, and quality checks of MEGA Code and seven other systems.
Implementation comparison
| System | Input Source | Automation | Strategy | Quality Control |
|---|---|---|---|---|
| MEGA Code | Auto-captured sessions, multi-session corpus | Fully autonomous | ✓ | LLM judging, gating, privacy masking |
| HF Upskill | Task prompt + optional traces | Semi-automatic | — | Automated tests, threshold gate |
| SkillWeaver | Web exploration (self-generated) | Fully autonomous | Partial | Iterative self-practice |
| skill-creator (Anthropic) | Task prompt + subagent transcripts | Interactive | Partial | LLM judging, A/B testing, human gate |
| skill-builder | Task prompt | Interactive | — | Manual checklist |
| Claude-Skill-Builder | Task prompt + 40+ pre-built skills + marketplace | Interactive / Semi-automatic | — | Structural conventions |
| claude-code-skill-factory | Task prompt via guided Q&A + templates | Semi-automatic | — | Structural validation |
| MakeSkill | Natural-language spec | Semi-automatic | — | None documented |

Optimization demo
Watch MEGA optimize an agent.
Step through an optimization run end-to-end.


