Original title: Polish ModernBERT: The Long and Short of Polish Language Understanding
Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec · September 2026
Most Polish language models still use older AI architectures that cannot read long pieces of writing at once. To fix this, researchers built Polish ModernBERT. This family of four models comes in two sizes, Base and Large. Each size has a short version that reads 512 tokens of text and a long version that handles 8K context tokens. The team trained them by testing different training recipes in stages. Across 30 language tests, Polish ModernBERT scored the highest overall marks among evaluated Polish encoders. Its long-reading models beat older Polish RoBERTa baselines while running faster and using less memory. The smaller Base model even beat its baseline while using 22% fewer parameters.
Imagine you are packing for a trip with a limited suitcase, but you need to fit all your essentials. Normally, you might have to leave some things behind or make multiple trips, which is slow and wasteful. Here, the researchers designed a smarter way to pack so that more fits with less space, and you can carry everything in one go. In this paper, the models handle long Polish texts more efficiently, using fewer computer resources while getting better results.
For people working with Polish text, like lawyers or literary scholars, these models make AI tools faster and more accurate. This could lower costs and improve accessibility for Polish language technology.
The abstract doesn't say if these models handle all Polish dialects or real-time uses, and it only compares them to other Polish encoders, not global ones.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Polish ModernBERT: The Long and Short of Polish Language Understanding — Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.