Original title: Unifying Conformal Language Tasks with In-Context Ensembles
Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara, Kin Kwan Leung +1 · September 2026
Many language tasks, like summarizing text or answering questions, require pulling out the right information. This means balancing coverage, which is keeping enough helpful details, with conciseness, which is cutting out useless fluff. Scientists use a tool called conformal prediction to guarantee coverage, but they must design a score function to keep answers concise. Usually, people spend lots of manual effort writing specific prompts to make large language models score importance. To solve this, researchers built the Conformal Relevance framework. It automatically picks in-context learning examples and combines predictions from multiple model outputs to build a score function. This setup keeps guaranteed coverage while boosting conciseness with minimal manual input across seven language tasks.
Packing a suitcase for a trip involves deciding what to bring and what to leave behind. It's hard to know the right balance without a checklist, and if your plans change, you might pack too much junk or forget something essential. In this paper, the problem is similar: systems need to extract key information from documents without being too verbose or too brief. By using a collection of good packing examples, the framework learns to pick the right content automatically, much like how experience helps you pack better over time.
People building AI tools for text analysis spend weeks writing custom prompts for each new task. This framework lets them skip most of that work while keeping results accurate.
The abstract doesn't explicitly state limitations, but the work is demonstrated on seven NLP tasks and may not generalize to all scenarios without further validation.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Unifying Conformal Language Tasks with In-Context Ensembles — Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.