Gricea: An Open Science Platform for Conversational AI Research
Nikhil Sharma, Yunlin Gong, Xinyang Cheng, Ziang Xiao
Johns Hopkins University
Published September 18, 2026 · on the feed September 21, 2026 · ▲ 13 · arXiv 2609.22039
The abstract
We need studies on conversational AI (CAI) at scale to understand human behavior and shape CAI design. However, fragmented reporting of systems and study configurations hinders replication, extension, and knowledge accumulation. We present Gricea, an open-science platform representing studies as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse. Informed by a formative analysis of prior CAI research, Gricea couples study procedures, participant-facing systems, and conversational task behavior in. In a replication study using Gricea, we replicated configurations 93% of eligible CUI 2026 papers; while also flagging missing information in 96% of papers that hinder faithful replication --- further motivating Gricea's need. In a user study, researchers and practitioners from diverse backgrounds successfully constructed runnable studies addressing various open-ended research questions. Together, these findings demonstrate Gricea's support for constructing, reproducing, and extending CAI studies through shared research artifacts, enabling cumulative knowledge building through open science.
The Brief
2:00A short podcast summary of the paper. Written and voiced by AI.
Transcript
Gricea. An Open Science Platform for Conversational AI Research. By Nikhil Sharma and three others, from Johns Hopkins University. Published on the eighteenth of September. No code repository is linked to it.
Gricea is an open-science platform for research on conversational AI. It represents a study as a configurable, deployable research artifact, something researchers can run, inspect, share, and reuse.
Informed by a formative analysis of prior conversational AI research, the platform couples three things together: the study procedure, the systems that participants face, and the behavior of the conversational task. A study packaged this way carries its own configuration with it. Another researcher can open it, see how it was set up, run it as it was, or change one part and run it again.
The motivation is replication. Understanding human behavior and shaping the design of conversational AI needs studies at scale, but fragmented reporting of systems and study configurations gets in the way of replicating, extending, and accumulating knowledge. In a replication study using Gricea, the authors replicated the configurations of ninety-three percent of eligible CUI two thousand twenty-six papers. They also flagged missing information in ninety- six percent of papers, gaps that hinder faithful replication, and they present that as further motivation for the platform.
In a user study, researchers and practitioners from diverse backgrounds successfully constructed runnable studies that addressed various open-ended research questions. That result speaks to who the platform is for. It is meant to be usable by people beyond its builders, across the range of questions conversational AI research asks, and not only for replicating what has already been done. Together, the replication study and the user study cover the three things the authors say Gricea supports: constructing studies, reproducing them, and extending them.
What Gricea changes is the unit of sharing. A study becomes something another group can run, not only read about, and that is how the authors hope knowledge about conversational AI can build on itself through open science.
Paper Quest · 7 chapters · about 34 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Science builds on what came before. Here, what came before is hard to even find.
You want to replicate a published study of how people use a chatbot. The paper reports its results, but not exactly how the interface, the agent's prompts and the study's steps were set up.
The road ahead
- 1 · The problem ~6 min Published chatbot studies report their results but rarely the exact setup, so rebuilding one by hand is costly and uncertain.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Store a whole study as one runnable artifact, with procedure and task kept apart, where the design you inspect is exactly what runs.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: Using Gricea, what share of eligible CUI 2026 papers did the authors manage to replicate in configuration?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.