ACE's performance on multi-turn tool use and environment interaction tasks
AppWorld is an autonomous agent evaluation benchmark covering API usage, code generation, and environment interaction. Tasks are divided into normal and challenge difficulty levels; as of submission, the best system on the official leaderboard averages only 60.3%.
Evaluated separately on test-normal and test-challenge splits.
Base model: DeepSeek-V3.1, base framework: ReAct.
1. ACE significantly leads in offline scenarios
ReAct + ACE improves over ReAct + ICL / ReAct + GEPA by 12.3% and 11.9% respectively, demonstrating that structured evolving context outperforms fixed examples or single-optimization instruction prompts.
2. Effective even without GT labels
ReAct + ACE (no GT) still improves over baseline by 14.8%. ACE leverages execution feedback (whether code runs successfully) as signals for Reflector and Curator.
3. Small models match top commercial systems
Evaluation Metrics
| Metric | Meaning |
|---|---|
| TGC | Task Goal Completion rate |
| SGC | Scenario Goal Completion rate |
Main Results
Base model: DeepSeek-V3.1, base framework: ReAct.
| Method | GT | Test-Normal | Test-Challenge | Average | ||
|---|---|---|---|---|---|---|
| TGC | SGC | TGC | SGC | |||
| ReAct (baseline) | – | 63.7 | 42.9 | 41.5 | 21.6 | 42.4 |
| Offline | ||||||
| ReAct + ICL | ✓ | 64.3 | 46.4 | 46.0 | 27.3 | 46.0 |
| ReAct + GEPA | ✓ | 64.9 | 44.6 | 46.0 | 30.2 | 46.4 |
| ReAct + ACE | ✓ | 76.2 | 64.3 | 57.3 | 39.6 | 59.4 |
| ReAct + ACE | ✗ | 75.0 | 64.3 | 54.4 | 35.2 | 57.2 |
| Online | ||||||
| ReAct + DC (CU) | ✗ | 65.5 | 58.9 | 52.3 | 30.8 | 51.9 |
| ReAct + ACE | ✗ | 69.6 | 53.6 | 66.0 | 48.9 | 59.5 |
Three Key Findings
1. ACE significantly leads in offline scenarios
ReAct + ACE improves over ReAct + ICL / ReAct + GEPA by 12.3% and 11.9% respectively, demonstrating that structured evolving context outperforms fixed examples or single-optimization instruction prompts.
2. Effective even without GT labels
ReAct + ACE (no GT) still improves over baseline by 14.8%. ACE leverages execution feedback (whether code runs successfully) as signals for Reflector and Curator.
3. Small models match top commercial systems
- Leaderboard (2025-09-20): IBM CUGA (GPT-4.1 driven) averages 60.3%
- ReAct + ACE (DeepSeek-V3.1) averages 59.4%
- On test-challenge, ACE Online + Offline warmup surpasses IBM CUGA by 8.4% TGC / 0.7% SGC
Why Greater Improvement on Challenge Split
- Complex tasks benefit more from strategy reuse
- ACE's Playbook accumulates long-term experience to help reduce common failure patterns
- Differences on simple tasks are overshadowed by base capabilities
Comparison with GEPA / DC
| Method | Limitation | ACE's Improvement |
|---|---|---|
| GEPA | Single genetic optimization | Continuous evolution |
| DC | Full rewrite causes collapse | Delta + Grow-and-Refine |
| ICL | Fixed examples | Context can accumulate |