Summarizing many documents with a model trained on one

Based on Zero-cost Transition to Multi-Document Processing in Summarization with Multi-Channel Attention (ECML PKDD 2024, undergraduate work at VNU Hanoi). Paper · Code

The problem

Good single-document summarizers are everywhere (BART fine-tuned on CNN/DailyMail, for example). Multi-document summarization, say a cluster of news articles on one event, is harder for three reasons:

The usual fix is to concatenate all documents and use full attention, whose cost grows quadratically with total length.
You need aligned multi-document training data, which is scarce.
Concatenated inputs can exceed the encoder's position limit, requiring truncation, chunking, or a different long-context strategy.

Documents are never concatenated into one sequence, so there is no quadratic growth in document count at fixed document length. Each document still has the encoder's position limit, and total input size is bounded by memory and compute.

The idea: treat each document as a channel

Instead of one long sequence, keep n separate documents, each its own channel. Scaling the number of documents then happens across channels (I call this vertical scaling) rather than along sequence length (horizontal scaling, e.g. Longformer or BigBird).

The paper motivates this with an approximate factorization: each document is treated as independently interpretable, and summary content is modeled through per-document contributions. Related news articles are not literally statistically independent, so this is a modeling assumption rather than an identity:

log P(Y | X)  ≈  sum over i of [log P(Y, X_i) - log P(X_i)]
                 = sum over i of log P(Y | X_i)  (approximate score)

Each term only needs one document, which is exactly what a single-document model already computes. Cost becomes linear in the number of documents.

The architecture: Multi-Channel Attention

I build this on BART with no new parameters.

Encoder. Run the pretrained BART encoder on each document independently, with a shared encoder. No position encoding is shared across documents, so there's no document-count limit from positions.
Decoder. Replace cross-attention over one long sequence with multi-channel attention: the decoder attends to each channel separately, then averages the results.

mca = mean over channels i of  softmax(Q K_i^T / sqrt(d)) V_i

Same weights as BART's cross-attention, just applied per channel and averaged. That is why the transition is "zero-cost": no fine-tuning, no extra parameters.

Why average the attention outputs, not the final probabilities?

The factorization motivates combining per-document contributions. The paper illustrates the choice with a simplified decoder block containing a ReLU. If the summary draws only on document 1, document 2 should be ignored, yet its negative contribution would get added in and distort the result. By summing before the ReLU-containing block, the ReLU decides on the combined signal, so a channel's rejection depends on context instead of dragging down the sum. I normalize by n to keep scale stable as documents are added.

Results

Cost. Encoding time and memory grow linearly with document count, like BigBird, while full-attention BART grows quadratically. MCA is also faster than BigBird in practice, because BigBird's sparse patterns need a lot of indexing.
Quality, no fine-tuning. On DUC2004 (50 news clusters), BART-large+MCA and distil-BART+MCA come out best on ROUGE-2, ROUGE-L and BERTScore in the fixed-length setting. They beat plain BART on concatenated input, hierarchical summarization and segment-by-segment summarization, and do far better than PEGASUS and BigBird.
More coverage. In a sample, the plain BART summary only reflected the first document, while the MCA summary pulled in facts from several.
I also fine-tuned MCA on a small augmented set for query-driven multi-document summarization, where it reaches full-attention accuracy.

Schematic of the trend only. The paper measures this with real timings: full attention is quadratic, MCA and BigBird are linear, and MCA is faster than BigBird.

Limits

Independent per-document encoding can miss relationships that require joint reasoning across documents. Averaging channel outputs is an architectural approximation, not an exact implementation of a normalized probabilistic factorization. Each document must still fit the encoder's length limit.