100%
Merge-ready fixes
MEGA Loop 11/11 · LangSmith Engine 2/11
Benchmark
MEGA Loop vs. LangSmith Engine
Comparing defect detection, fix verification, and mergeability on the same agent trace dataset.
Benchmark results
11 defects. 11 merge-ready fixes.
DeerFlow 2.0–based order-management agent · 320 traces · 11 defects.Both products received the same 320-trace dataset; MEGA Loop analyzed all 320 traces, while LangSmith Engine analyzed 100.
93.8%
Lower execution cost
MEGA Loop $1.58 vs. LangSmith Engine $25.51
| Metric | MEGA Loop | LangSmith Engine |
|---|---|---|
| Recall | 11/11 (100%) | 3/11 (27.3%) |
| Precision | 11/11 (100%) | 3/4 (75%) |
| F1 score | 1.0 | 0.4 |
| False positives | 0 | 1 |
| Merge-ready fixes | 11/11 (100%) | 2/11 (18.2%) |
| Traces analyzed | 320 of 320 | 100 of 320 |
| Execution cost | $1.58 | $25.51 |
| Usage | 1.57 million tokens | 17 LCU |
LCU stands for LangChain Compute Unit, the platform’s billing unit.
Test setup
An order-management agent with eight injected defects.
The agent handled date-based order searches, recent-order retrieval, status filtering, and report generation.
5
Errors that interrupt requests
- Date parsing fails to find orders.
- Search stops when an order has no date.
- A request for three results returns no order list.
- Report generation stops when an order is not yet shipped.
- Filtering stops when an order has no status.
2
Incorrect results without errors
- The same result repeats 4,000 times, obscuring the answer.
- A percentage appears as None% instead of a numeric value.
1
Response delay
- An unnecessary 11-second wait slows every request.
Eight injected defects, plus three existing defects.
MEGA Loop found and fixed three additional defects already present in ByteDance’s DeerFlow 2.0.
Fix verification
Fix verification and merge review.
Verify the behavior
Each fix was tested by running the modified agent to check whether it resolved the defect.
Review the code diff
GPT-5.6 Sol, an LLM judge independent of both products, reviewed each diff against a five-item rubric to determine whether it could be merged without further edits.
Why one LangSmith Engine patch was rejected
Of its three fixes, the date-boundary patch changed unrelated pagination, order-data normalization, and status filtering. The reviewer rejected it for going beyond the defect’s scope.
None of MEGA Loop’s 11 fixes changed code outside the defect’s scope.
Execution cost
From $25.51 to $1.58.
Usage was 1.57 million tokens for MEGA Loop and 17 LCU for LangSmith Engine.
- LangSmith Engine
- $25.51
- MEGA Loop
- $1.58
100 traces analyzed
320 traces analyzed
How Loop works
How MEGA Loop finds and fixes defects.
01
Analyze the full trace set
Analyze traces and cluster failure patterns to identify defects that can actually be reproduced.
02
Group related defects
Classify dependencies as sequential, parallel, mixed, or standalone, then group defects that need to be fixed together into one work unit.
03
Generate and verify fixes
Create patches for each work unit. Compare behavior before and after the change to check that the defect disappears, and run regression checks to confirm existing behavior still works. Revise fixes that fail verification.
More Benchmark Results
MEGA Loop early access
Be among the first to turn failed runs into verified fixes.
Join the early access waitlist. We’ll reach out as pre-launch access becomes available.