NuExtract 1.5 — Multilingual, Infinite context, still small, and better than GPT-4o!

We introduce NuExtract 1.5, the new version of our foundation model for structured extraction. NuExtract 1.5 is multilingual, can handle arbitrarily long documents, and outperforms GPT-4o in English while being 500 times smaller. As usual, we release it under MIT license.

• This post is a continuation of the original NuExtract post.
• You can download NuExtract 1.5 here and try it here.

Why NuExtract?

NuExtract is a family of small open-source models that do only one thing: they extract information from documents and return a structured output (JSON). Because they only do this one thing, they are very good at it — NuExtract 1.5 (3.8B parameters) is better than GPT-4o on our English zero-shot benchmark while being 500× smaller. Moreover, fine-tuning NuExtract gives you performance hard to reach via prompting, even for a frontier LLM.

Such a small open-source model has two main advantages:

1. You can use it privately, without sharing your data.
2. You can fine-tune it on input-output examples to make it excel at a specific task.

For repetitive tasks needing high performance, or for sensitive data, NuExtract is the way to go.

NuExtract 1.5 in a Nutshell

The two main requests we received after NuExtract 1.0 were the ability to process long documents and to handle non-English documents.

In a nutshell, we created a multilingual dataset and trained the latest open-source LLMs on it. We then added a "continuation" functionality, which gives NuExtract an infinite context size with a bounded memory footprint.

Multilingual Abilities

We need both a multilingual dataset and a multilingual base model. Phi-3.5 mini recently made a lot of progress on that side, handling 23 languages. We chose Phi-3.5 mini as the base of NuExtract 1.5.

For the training dataset we again take raw documents from the C4 dataset: 50% English, 50% other languages (mainly French, German, Spanish, Italian, Portuguese), with longer documents than for the original NuExtract.

For half the documents we use an English template (irrespective of the document language), so users can write a unique template in English when processing multilingual documents. The other half uses templates in the document's own language. The dataset stays purely extractive — copy-paste, no generation.

Infinite Context

Thanks to Phi-3.5 mini, NuExtract 1.5 has a 128k token context (~200 pages). But processing long documents with a transformer is memory-intensive: every token attends over every other token. Maxing out 128k tokens requires 1TB of GPU memory.

To solve this we adopted an original solution: we train NuExtract to extract information while being given previously extracted information.

This "continuation" ability allows us to process arbitrarily long documents by iteratively re-injecting the current state of information through a sliding context window — reminiscent of recurrent neural networks. Memory footprint is bounded by the window size (less than 30GB at 10k window size, irrespective of document size).

The downside is that the output is generated several times, and that performance can degrade if the sliding window is too small.

Training & Results

We train Phi-3.5 mini (3.8B) on our dataset to get NuExtract 1.5. We also train Qwen 2.5 0.5B to get a tiny English-only variant.

English Performance

On our 600-example, 12-problem English benchmark:

• Zero-shot: NuExtract 1.5 is quite better than the original NuExtract, and slightly better than GPT-4o.
• Many-shot (45 examples per problem, fine-tuning for NuExtract, in-context for GPT-4o): GPT-4o is slightly better, but not by much. The big gap between NuExtract 1.5 and the tiny variant suggests that a bigger NuExtract could largely beat GPT-4o.

A 500× smaller model rivaling a frontier LLM is surprising. Three reasons: focus on a single task lets NuExtract redirect weights to text understanding; training enforces strict template compliance and JSON-only output; training drastically reduces hallucinations by forcing extraction from the input and teaching empty results when needed.

Multilingual Performance

NuExtract 1.5 is much better than the original NuExtract, but GPT-4o is still better in this case. Model size matters a lot for multilinguality — we expect to fill this gap with a bigger NuExtract.

Long Documents

On 8k-10k token documents (~20 pages), NuExtract 1.5 beats GPT-4o. On 10k-20k token documents with a 10k extraction window, NuExtract 1.5 still beats GPT-4o. We need to go down to a 2k window for NuExtract 1.5 to become worse than GPT-4o — and even then it remains much better than the tiny variant.

The continuation procedure works.

Let's Use It!

You can try NuExtract 1.5 here. Don't hesitate to give us feedback to help us improve the next versions :)