MEGA Code
Contact
Benchmark

Agent optimization

Compare workflow optimization on the GEPA suite, skill selection on SkillsBench, and generated skill quality across systems.

Read the tech report

Optimization · 4 tasks · GPT-4.1 Mini

Workflow optimization on the GEPA suite

MEGA's workflow optimizer benchmarked against MIPROv2, TextGrad, GEPA, and Feedback Descent on four compound AI systems from GEPA's released eval code — multi-hop QA, instruction following, retrieval-based claim verification, and privacy-aware delegation. All methods use the same evaluation checkers and grading criteria.

+10.1%

Higher aggregate task score vs. GEPA

GEPA 69.52 → MEGA 76.55 · +7.03 points

SystemHotpotQAIFBenchHoVerPUPAAggregate
Baseline38.0047.7946.3378.5752.67
MIPROv258.0049.1548.3383.3759.71
TextGrad62.3348.6447.6785.6861.08
Feedback Descent68.3354.5957.6785.6666.56
GEPA69.0055.9556.6796.4669.52
MEGA72.6761.0574.6797.8176.55

Aggregate is the mean of the four task scores, not a single-task result. Scores are test-split accuracy. MEGA train/validation/test sizes: HotpotQA 150/100/300, IFBench 150/100/294, HoVer 150/100/300, and PUPA 111/111/221. Baseline, MIPROv2, TextGrad, and GEPA scores are cited from the GEPA paper; Feedback Descent scores are cited from its authors. MEGA used smaller validation sets than GEPA's original splits (100 vs 300 for HotpotQA) to reduce optimization cost.

Read the tech report

SkillsBench · 84 tasks · 11 domains

Skill selection on SkillsBench

Compare skill selection methods on 84 tasks across 11 domains. Each condition uses Gemini 3 Flash, the same 4,207-asset skill pool, and the same tasks and verifiers. Results are checked with deterministic pytest assertions in isolated Docker containers.

+47.6%

Higher task success rate vs. no skills

Without skills 31.5% → MEGA (WG) 46.5% · +15.0 percentage points

SystemPass Rate (%)Avg Tokens/Task (k)Curation Latency (s/task)Efficiency (score/Mtok)
No Skills31.5894—0.353
AgentSkillOS41.11,189403.40.345
SkillNet41.798337.80.424
MEGA (WG)46.582211.80.566

The 47.6% relative increase compares MEGA (WG) with the no-skill baseline, not SkillNet. SkillNet’s pass rate is 41.7%. Efficiency is pass rate per megatoken consumed (score/Mtok). Curation latency measures skill discovery and orchestration before execution. MEGA retrieves skills from a pre-indexed graph without LLM calls during retrieval.

Read the tech report

Skill generation

Generated skill quality

Compare token use, task scores, and the structure of generated skill files.

Five skill-generation systems and a no-skill baseline are evaluated on the same task harness. A separate implementation comparison covers MEGA Code and seven other systems.

View the evaluation repository

5 skill-generation systems + baseline

Token use by system

MEGA Code generated skills from a 10-round FastAPI, React, and Gemini development session. The other generators received 1–2 sentence task prompts without detailed traces. Four skills per system were evaluated with HF Upskill’s harness: five test cases per skill, run on both Claude Sonnet and Haiku. The no-skill baseline is included for reference.

1/5

the tokens, same tasks

169K vs 897K

MEGA Code

0% vs baseline

897K

HF Upskill

0% vs baseline

897K

anthropic-skill-creator

0% vs baseline

897K

Baseline (No Skill)

reference

897K

claude-code-skill-factory

+0% vs baseline

897K

skill-builder

+0% vs baseline

897K

Vertical line marks baseline (no skill). Bars exceeding baseline mean the system used more tokens than having no skill at all.

Combined task score

Mean score across all 8 runs per system (4 skills × 2 models). Each skill was scored on 5 test cases, measuring whether the generated skill correctly guided the AI agent to produce the expected output.

Combined (Sonnet + Haiku)

MEGA Code

169K tokens

0%

HF Upskill

763K tokens

0%

anthropic-skill-creator

826K tokens

0%

Baseline (No Skill)

897K tokens

0%

skill-builder

2,024K tokens

0%

claude-code-skill-factory

1,448K tokens

0%

Sonnet Only

MEGA Code

0%

HF Upskill

0%

Baseline

0%

anthropic-skill-creator

0%

skill-builder

0%

claude-code-skill-factory

0%

Haiku Only

MEGA Code

0%

HF Upskill

0%

anthropic-skill-creator

0%

Baseline

0%

skill-builder

0%

claude-code-skill-factory

0%

Generated file structure

Each cell marks circle (full), triangle (partial), or cross (absent) across 8 structural dimensions of generated skill files.

Structural ElementMEGA CodeHF Upskillskill-factoryskill-builder
Frontmatter completeness
Trigger precision
Preconditions
Workflow specificity
Rule reasoning (Why/Effect)
Anti-pattern coverage
Common Mistakes (why-it-happens)
Success Indicators

Skill-generation results

81% fewer tokens than baseline

MEGA Code used 169K tokens, compared with 897K without skills and 763K–2M for the other tested systems.

78% combined score

Across four skills and two models, MEGA Code scored 78%, compared with 65% for the no-skill baseline.

16/16 on the structural rubric

The generated files received full marks on all eight criteria, including preconditions, rule explanations, and success indicators.

Privacy filtering

Automated filtering covers eight categories of sensitive patterns before session data leaves the local machine.

Methodology

Test harness

HF Upskill eval

Models

Claude Sonnet & Claude Haiku

Test Cases

5 per skill, auto-generated by Opus via HF Upskill

Source Material

10-round full-stack dev session; competitors given minimal prompts only

Skills Evaluated

4 full-stack skills from a FastAPI + React + Gemini project

Systems

MEGA Code vs. HF Upskill, anthropic-skill-creator, claude-code-skill-factory, skill-builder, Baseline (no skill)

Implementation comparison · 8 systems

Inputs, automation, and quality checks

Compare the inputs, automation, strategy output, and quality checks of MEGA Code and seven other systems.

Implementation comparison

SystemInput SourceAutomationStrategyQuality Control
MEGA CodeAuto-captured sessions, multi-session corpusFully autonomous✓LLM judging, gating, privacy masking
HF UpskillTask prompt + optional tracesSemi-automatic—Automated tests, threshold gate
SkillWeaverWeb exploration (self-generated)Fully autonomousPartialIterative self-practice
skill-creator (Anthropic)Task prompt + subagent transcriptsInteractivePartialLLM judging, A/B testing, human gate
skill-builderTask promptInteractive—Manual checklist
Claude-Skill-BuilderTask prompt + 40+ pre-built skills + marketplaceInteractive / Semi-automatic—Structural conventions
claude-code-skill-factoryTask prompt via guided Q&A + templatesSemi-automatic—Structural validation
MakeSkillNatural-language specSemi-automatic—None documented

More Benchmark Results

Optimization demo

Watch MEGA optimize an agent.

Step through an optimization run end-to-end.

Watch the demo