Gabriel/diffu_sd35_128

Diffu SD3.5 · 128px — Swedish historical handwriting line generator

Renders a line of text in a target writer's hand. Give it a string + a single-line crop of the target handwriting; it generates a matching handwritten line, trained from scratch on Swedish archive material (the GT-Lejonet / Riksarkivet corpus). Built to synthesise HTR training data — generated lines read back as legible Swedish through the real recognizer.

This is the 128 px model (step_120000 of the exp_sd35_128 run): a 2× line-resolution upgrade over the earlier 64 px release, trained with the classifier-free-guidance null fix (condition + text dropout), so guidance actually works instead of collapsing.

Quality (measured, not claimed)

• ~2% character error rate with guidance (cfg_scale ≈ 5), scored by the Riksarkivet HTR recognizer Riksarkivet/trocr-base-handwritten-hist-swe — i.e. essentially at the ~1.6% real-data CER floor.
• Composed into full pages, synthetic scans read back at ~2–5% CER end-to-end through the Riksarkivet HTRFlow pipeline — comparable to real documents.
• Converged: both validation loss and guided CER plateaued at their floor by ~step 100k; step_120000 is the checkpoint at the lowest, most stable guided-CER band.

Guidance matters: this model is trained for CFG. Use cfg_scale ≈ 5. Without guidance (cfg_scale ≈ 1) quality drops sharply — that is expected, not a defect.

Usage

from diffu.pipeline import DiffuPipeline          # pip install git+https://github.com/Borg93/diffu

pipe = DiffuPipeline.from_pretrained("Gabriel/diffu_sd35_128")
img = pipe("Göteborgs poliskammare den fjärde maj", style="a_line_crop_of_the_target_hand.png", cfg_scale=5.0)
img.save("out.png")

The style reference is any single-line crop of the target hand (it is ImageNet-normalized to the DINO input the model expects). One writer reference conditions every line you generate with it.

Architecture

From-scratch rectified-flow diffusion:

• diffusers SD3.5 MMDiT backbone (qk-norm / RMSNorm) + custom 2D-RoPE joint-attention.
• Frozen Qwen-Image VAE (f8) — 128 px line crops → 16 px-high latents.
• Glyph-line content conditioning (GNU-Unifont rendered text), so the model reads the string, not tokens.
• DINOv3 pooled style embedding from the writer crop.
• Trained at 128 px with condition-dropout 0.1 + text-dropout 0.1 (the CFG-null fix) and an ink-focal loss.

Performance

# one-off lines: default (eager) is fine — ~1 s/line on a modern GPU
pipe = DiffuPipeline.from_pretrained("Gabriel/diffu_sd35_128")

# many lines / serving: compile the backbone once (regional torch.compile, dynamic —
# one compile covers variable line widths). One-time cold start, then sub-second/line.
pipe = DiffuPipeline.from_pretrained("Gabriel/diffu_sd35_128", compile_backbone=True)

# bf16: ~half the VRAM, same legibility
import torch
pipe = DiffuPipeline.from_pretrained("Gabriel/diffu_sd35_128", dtype=torch.bfloat16, compile_backbone=True)

Don't quantize by default. Measured on this model (torchao int8/fp8, CER-gated): legibility is unaffected but inference gets slower — the GEMMs are small and not memory-bound, so weight-quant overhead dominates. bf16 + compile is the recommended serving config.

Requirements

• The diffu package (the modeling code this pipeline wraps).
• DINOv3 access (gated): trained with facebook/dinov3-vitl16-pretrain-lvd1689m — accept its licence on the Hub (free). Don't substitute DINOv2: different style embedding than the backbone was trained on.
• Qwen-Image VAE (Qwen/Qwen-Image, Apache-2.0) — pulled automatically.
• CUDA GPU recommended (~6 GB VRAM / line in float32, less in bf16).

Intended use & limits

• Intended: generating synthetic Swedish-historical handwriting lines to augment HTR training data.
• One writer per reference: the style comes entirely from the crop you pass; vary the crop to vary the hand.
• Line generator, not a page compositor: full-page assembly (layout, labels, scan aging) lives in the diffu_page package.
• Domain: 18th–19th c. Swedish administrative hands. Out-of-domain text/scripts are not guaranteed.

Files

model.safetensors (trained weights, single non-strict state_dict) · config.json (arch: line_height=128, glyph_line=True, style_in_context=False, qk_norm='rms_norm', use_unifont=True).