Unifying Conformal Language Tasks with In-Context Ensembles
Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara, Kin Kwan Leung, Jesse C. Cresswell
Signal 1 AI · Layer 6 AI
Published September 2, 2026 · on the feed September 8, 2026 · ▲ 10 · arXiv 2609.03005
The abstract
Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.
The Brief
2:01A short podcast summary of the paper. Written and voiced by AI.
Transcript
Unifying Conformal Language Tasks with In-Context Ensembles. By Xiao Shi Huang and four others, from Signal 1 A I and Layer 6 A I. Published on the second of September. Its code repository is worked out: about eighty-five source files, with a readme and a licence.
Many language tasks, summarization and extractive question answering among them, come down to pulling relevant content out of documents under two constraints. Coverage means keeping enough pertinent information to achieve the goal. Conciseness means discarding as much irrelevant information as possible. Conformal prediction can guarantee coverage, but conciseness depends on the score function it is given. The best scoring functions so far are hand-engineered prompts asking an LLM to rate how important each piece of content is, and writing those prompts by hand is labour-intensive and has to be redone for every task.
The Conformal Relevance framework replaces the hand-written prompt with curation and ensembling. It builds the score function from carefully chosen in-context learning examples, and combines several of them into an ensemble. The target is a score that keeps the coverage guarantee intact while making the retained content more concise, with minimal manual input.
Two pieces of the approach are named in the abstract. One is the curation of in-context examples, which stands in for prompt engineering. The other is ensembling, which pools several such scorers into one. Because ensembling is central, the authors also study it formally, asking what diversity among ensemble members contributes to a conformal score.
The framework is demonstrated on seven NLP tasks. The theory yields two results. A complementarity condition characterizes when ensembling improves worst-case sentence scores. A saturation bound limits how much improvement an ensemble can deliver at all.
The shift here is from craft to procedure. Where a strong conformal scorer used to depend on someone writing the right prompt for each task, this framework assembles one from examples and ensembles, and the theory states both when combining helps and where the benefit runs out.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Coverage is guaranteed. Conciseness is not, and it hangs on a prompt someone wrote by hand.
You must cut documents down to only what matters: summaries, answers, evidence. A calibration method already guarantees you keep enough of the relevant sentences.
The road ahead
- 1 · The problem ~6 min A calibration step already guarantees you keep enough relevant sentences, but how much junk you cut depends on a hand-prompted scorer that does not scale.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Show the LLM curated labeled examples instead of describing relevance, build several scorers from different example choices, and average them.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Pick examples from a labeled pool with each strategy, score sentences with an LLM per strategy, average the scores, then calibrate a cut-off.
- 6 · The experiment ~6 min You’ll guess: As more LLM scorers are added to the ensemble, what does the theory say happens to the improvement?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.