Automating Empirical Research with an Agent Skill Library: The Stanford REAP Case and Its Limits
The claim that “AI writes a top journal paper in 20 minutes” is tempting, but taking it at face value is dangerous. The agent skill library released by CoPaper.AI, maintained under Stanford REAP (brycewang-stanford/Auto-Empirical-Research-Skills), gathers more than 23,000 empirical research agent skills spanning eight social science fields. The scale is impressive, but a data scientist’s job is to honestly separate what that scale actually means.
Here at ThakiCloud, we work with multi-agent knowledge work workflows on our K8s-based AI/ML SaaS platform. Let’s go through what is genuinely valuable in this library and what has been overstated.
What Is Real and What Is Exaggerated
First, let’s separate the numbers.
- 23,000 is an ecosystem map, not a ready-to-run count: the more than 23,000 skills broadly map the methodological space of empirical social science research. But that entire set is not immediately executable. The honest estimate for what is actually runnable is a much smaller figure, on the order of about 1,000.
- The “20-minute top journal paper” claim is exaggerated: producing a submittable draft quickly and actually getting it published are different problems. Submittable does not equal published.
- The core hard problems remain unsolved: causal identification, reproducibility, and hallucination are still unresolved. An agent can run a statistical analysis, but it cannot guarantee the validity of a causal inference.
Hedging claims this way is basic hygiene for not getting swept up in exaggeration. A large scale does not by itself guarantee quality.
Why It Still Has Value
Even after stripping away the hype, this approach retains real practical value.
- Codifying methodology: packaging the standard procedures of empirical research (data cleaning, descriptive statistics, regression, hypothesis testing) into reusable skills can automate repetitive work and improve consistency.
- A skill as a capability product: unlike a simple prompt, a skill is version controlled and bundled together with scripts, templates, and validation. It becomes a reusable workflow that covers everything from input to output to error recovery.
- A user contribution structure: a structure where users can upload their own skills opens a path for the ecosystem to grow through collective intelligence.
A Multi-Agent Knowledge Work Automation Perspective
The real lesson from this library is “where you build up capability.” When capability is stacked thickly into the skill rather than the harness (the model loop), the same skill can work across multiple environments. Packaging domain knowledge, judgment, templates, and failure cases into a skill is a core principle for how multi-agent systems grow in a scalable way.
But the parts automation cannot solve are just as clear. The validity of causal identification, the reproducibility of results, and blocking hallucination are not fully verifiable by code. This kind of judgment-heavy work still requires strong models and human review. Automation only lightens repetitive work; it does not replace judgment.
The ThakiCloud Perspective
Our work is running skill libraries like this reproducibly on top of K8s. Repetitive work such as exploration and running statistics goes to cheaper models, while judgment work such as causal inference and result verification goes to stronger models. The principle is to keep workers cheap and reserve cost only for the gate. And we run every result through an adversarial verification stage so that hallucination and exaggeration are filtered out by both code and model.
Closing
The Stanford REAP skill library shows a direction toward automating empirical research with agents, but exaggerations like “a top journal paper in 20 minutes” need to be stripped away. The real value lies in codifying and reusing methodology, and causality, reproducibility, and hallucination remain areas that still require human and strong-model judgment. For engineers interested in honestly handling the boundary between automation and judgment, this is a problem they face every day.
Source: brycewang-stanford/Auto-Empirical-Research-Skills (maintained by Stanford REAP / CoPaper.AI). GitHub: https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills