Replace the handoff notes with a durable spec and ADRs

The handoff was written as a message to relay information; several
decisions lived only there. Split into permanent homes:

- docs/spec.md — scope, geometry model, pipeline order, detection,
  levels, editor, project file, markers, bundle format, noteman's
  changes, reference values
- ADR 0002 — raster only in release 1; SVG slices measured at 40x WebP
  naive, 2.6x with a bounding-box cull, deferred on risk not size
- ADR 0003 — lossless WebP with levels and 16-level alpha; every lossy
  option and alternative format measured larger for line art
- ADR 0004 — detection proposes, the human disposes; no unattended mode
- ADR 0005 — PyMuPDF for all PDF access, accepting AGPL
- ADR 0006 — systems are found by vertical brackets, not row-darkness
  gaps, which miscount every page of a 6-voice score

Also from testing against the hardest score in the repertoire: scanned
PDFs carry their scan as an embedded image and must be extracted at
native resolution rather than re-rendered at 600 DPI, and per-page
deskew is mandatory (skew varies -2.6 to +1.2 within one PDF).
This commit is contained in:
Esa Kataja
2026-07-28 22:05:09 +03:00
parent efc920fdc5
commit 46ae7e813a
8 changed files with 674 additions and 7 deletions
@@ -0,0 +1,61 @@
# Raster only in release 1; the SVG renderer is deferred
Vector PDFs are most of the newer corpus, and keeping them vector all the way to
the viewer was an early goal — sheet music is line art, and SVG stays crisp at any
tablet zoom. We measured it before building it, and decided to **rasterize vector
sources like everything else in release 1** and revisit the SVG renderer once
real songs have been cut.
## The measurement
One real vector song, 6 pages, 65 systems, rendered both ways:
| Approach | Total | vs WebP |
|---|---|---|
| WebP slices (600 DPI → 1920, ink→alpha, lossless) | 1.19 MB | 1× |
| SVG, naive `viewBox` + `clipPath` | 26.0 MB | 40× |
| SVG, `set_cropbox` per band | 26.5 MB | 41× |
| SVG, bounding-box cull + glyph subset | 3.09 MB | 2.6× |
- **The naive cut is unusable.** A `viewBox` + `clipPath` slice contains the
entire page's geometry and merely hides eleven-twelfths of it.
- **`set_cropbox` does not help.** MuPDF renders full page content regardless of
the crop, so there is no free version of the cull.
- **The cull works.** PyMuPDF emits a `<defs>` glyph table (111 KB of a 256 KB
page) referenced by `<use transform="matrix(...)">`, plus body `<path>`
elements. Filter both by y-extent, then keep only the glyphs the survivors
reference. Roughly 50 lines, 15× improvement.
## Why defer, given the cull works
**Not size.** At 3.1 MB vs 1.2 MB per song — 225 MB vs 87 MB across a 73-song
corpus — both are nothing on a homelab. The measurement killed the lazy
implementation, not the idea.
What defers it is risk and missing evidence:
- The cull is **heuristic parsing**: glyph extents bounded at baseline ±14pt,
path extents read from raw `d` coordinates. It is over-inclusive by design, so
it fails safe — but "fails safe" still means a slice quietly carrying a
neighbour's slur, or a hairline dropped because the y-window was wrong on some
publisher's output. That needs eyeballing per song, a QA loop the raster path
doesn't have.
- Rendering 65 complex SVGs in a scrolling column may be slower than 65 WebPs.
Unmeasured.
- **The deciding question is unanswerable from here**: does 1920px WebP actually
feel insufficient when pinch-zooming on a tablet? Cutting real songs answers
it; more measurement doesn't.
Vector PDFs are also the *clean* case for the raster path — deskew is a no-op,
detection works best, there are no scan artefacts — so rasterizing them is not a
degraded fallback.
## Consequences
- The geometry model stays **renderer-agnostic**, in normalised page coordinates,
so adding the SVG renderer later is an output stage rather than a redesign.
- **Re-export from the project file** regenerates every song's bundle without
repeating human work, so songs cut before the SVG renderer exists are not
stranded.
- noteman needs no SVG support (`image/svg+xml`, `.svg` content type, CSP header
on SVG responses) until the renderer ships.
@@ -0,0 +1,58 @@
# Lossless WebP, with levels and alpha quantised to 16 levels
Slice images are encoded as **lossless WebP**, with the levels adjustment applied
and the alpha channel quantised to 16 levels. About 7 KB per slice, ~450 KB for a
65-system song. Every lossy option and every alternative format measured
*larger* for this content, which is the opposite of the usual intuition — hence
this record.
## The measurement
20 slices of one real song, levels applied throughout, relative to plain lossless
WebP:
| | vs baseline | |
|---|---|---|
| **WebP lossless + alpha quantised to 16** | **68%** | chosen |
| AVIF q60 | 90% | lossy, for 10% |
| WebP lossless | 100% | baseline |
| WebP lossy q85 (alpha) | 107% | |
| AVIF q85 | 114% | |
| JXL lossless | 130133% | |
| WebP lossy q85 (opaque ink-on-white) | 158% | |
| PNG grayscale + alpha | 165% | |
| AVIF lossless | 188% | |
Separately, before levels: applying levels alone takes 338 KB → 211 KB, a 38%
reduction.
## Four results that contradict an instinct
- **Lossy is bigger than lossless here.** Not a quality problem — the measured
difference between q85 and lossless is max 12/255, mean 0.33, i.e. invisible.
Lossy VP8 simply spends more bits on sharp black/white edges than VP8L's
palette and predictor transforms do, and notation is nothing but sharp edges.
The "q85 looks fine" intuition comes from photographs and inverts here.
- **AVIF and JXL both lose**, AVIF lossless by nearly 2×. Their lossless modes
are afterthoughts on photo codecs. WebP's VP8L is close to purpose-built for
flat two-tone line art — sheet music is the content type it is best at. JXL
additionally has no path forward in Chrome.
- **Alpha costs nothing.** Opaque ink-on-white and black-plus-alpha are within
0.1% at lossless, so paper-tint removal and future non-rectangular slices are
free.
- **Levels is the single biggest lever** — 38%, as a side effect of a control
that exists for quality reasons anyway. Pushing the white point below the
paper's luminance sets vast regions to exactly `alpha = 0`, which costs almost
nothing to encode.
Alpha quantisation to 16 levels is imperceptible: antialiased edges span 23 px
at 1920, and 16 steps across that is below notice. 8 levels starts to gamble on
thin strokes.
## Rejected as not worth it
- **Encoder effort tuning** — Pillow's `method=6` buys 3% and a dependency.
- **`alpha_quality=60`** — 24%, for less control than quantisation gives.
- **Grayscale WebP** — no such mode exists. It wouldn't help anyway: the RGB
channels are constant black and compress to nearly nothing, so alpha is the
entire payload.
@@ -0,0 +1,55 @@
# Detection proposes, the human disposes — there is no unattended mode
Every automatic result the slicer produces — skew angle, cut positions, source
type, staff height, ink bounds — is a **suggestion the user confirms or modifies**
before it is committed. There is no batch mode, no headless "slice this folder",
and no code path that writes a bundle without a human having looked at it.
This is a constraint on the tool's shape, not a UI preference, which is why it
gets an ADR: it deletes an entire phase of the original plan and it will look
like a missing feature to anyone who finds the detection code and wonders why it
isn't wired to a CLI.
## Why
The corpus is PDFs from a choir's distribution channel, and quality varies
wildly — clean vector engravings at one end, noisy scans with a previous owner's
pencil markings at the other. **Testing showed the detection algorithms produce
unusable slices on any source with speckles or otherwise poor quality.** Not
slightly-off slices: unusable ones.
But the same testing showed the suggestions land *close* on decent sources —
close enough that correcting them is faster than placing cuts from scratch. So
detection earns its place as an accelerator, and loses any claim to being
load-bearing.
## What this rejected
The original plan's **Phase A** was a deliberately non-interactive CLI:
rasterize, auto-deskew, auto-detect boundaries, write numbered slices, and fix
the misses by hand in GIMP. Its justification was "learn the failure modes before
designing the editor," which is a good idea.
It doesn't survive the premise. A CLI whose output can't be trusted has GIMP as
its repair path — routing work back into the manual process the project exists to
remove. A diagnostic variant (dump per-page PNGs with proposed cuts drawn in red)
was considered and also dropped: it only re-shows a failure already confirmed by
testing, and the editor shows the same thing live.
Release 1 is therefore the editor and detection together. There is no smaller
first release that is actually usable.
## Consequences
- **Manual placement is the primary interaction**, not a correction affordance.
The editor must be fully usable with detection producing nothing.
- **Despeckling targets the detector, not the output.** The known failure mode is
specks, so a median blur and a small-component filter clean the row-darkness
profile the detector reads; the shipped pixels come from the levels-adjusted
image.
- Cut placement is deliberately **forgiving** — anywhere in the whitespace gap
yields the same output, since trim crops to ink afterwards. Precision is not
asked of the human.
- The editor should surface **slice edges**, not just cut lines, so trim
anomalies (a speck anchoring the bounding box) are visible rather than
discovered later in the viewer.
@@ -0,0 +1,36 @@
# PyMuPDF for all PDF access, accepting AGPL
All PDF work — rasterizing at a chosen DPI, exporting SVG, and inspecting page
content to classify a source as bitmap or vector — goes through **PyMuPDF**. It
is a single wheel with MuPDF bundled, so the tool needs no system packages. Its
licence is **AGPL-3.0**, which we accept.
## Why not the permissive combination
The obvious permissive stack was `pypdfium2` (Apache/BSD) for rasterizing plus
`mutool` or `pdftocairo` shelled out for SVG. Both of those are **system
packages** — `mupdf-tools`, `poppler` — and a system package on the vector path
is precisely the failure the language choice was made to avoid: the tool is
supposed to install once and run from any directory on any machine.
The SVG step can't simply be skipped, either. Music glyphs come from a notation
font (Emmentaler, Bravura, or Sibelius/Finale's). An SVG that *references* a font
renders as garbage on a device that lacks it, so text must be converted to paths
at export. PyMuPDF does this **by default**`page.get_svg_image(text_as_path=1)`,
verified to emit `<path>` elements and zero `<text>` — so the font risk is closed
with no extra tooling.
Mixing the two (pypdfium2 for raster, PyMuPDF only for SVG) is the worst option:
two libraries with overlapping responsibilities, and AGPL linked in anyway.
## Consequences
- **The AGPL propagates only if the slicer is published.** For a local personal
tool it costs nothing. A future permissive release would need the rasterizer
swapped back to `pypdfium2` — a contained change, since PDF access sits behind
the renderer-agnostic geometry model.
- **Source-type detection comes free** from the same library: `get_images()` plus
a full-page-image area check distinguishes a scan from an engraving.
- The SVG export path is present and working even though the SVG *renderer* is
deferred — see
[ADR 0002](0002-raster-only-svg-renderer-deferred.md).
@@ -0,0 +1,67 @@
# Systems are found by vertical brackets, not by row-darkness gaps
System detection anchors on the **vertical bracket / barline** that spans a
system's staves, and uses the row-darkness profile only to expand each anchor to
its ink extent. The obvious approach — find gaps in the row-darkness profile and
cut in the middle of them — does not work on multi-voice choral scores, which is
most of the corpus.
## Why the obvious approach fails
A row-darkness profile cannot distinguish an **inter-staff** gap from an
**inter-system** gap. In a 6-voice closed score, one system is six staves joined
by a bracket, and the gaps between those six staves look exactly like the gap
between two systems — only smaller, and not reliably so.
Measured on *Ketun joululaulu*, a 12-page 6-voice arrangement and the hardest
score in the repertoire:
- On page 2's first system, staff gaps run ~47px against a ~211px system gap. A
merge threshold tuned there works.
- On the same page's second system the lyrics fill the inter-staff gaps, so the
ratios invert and the same threshold merges the wrong things.
Result across all 12 pages, row-profile-only versus bracket-anchored:
| | bracket-anchored | row-profile only |
|---|---|---|
| systems per page | 2, 2, 2, 2, 2, 2, 3, 2, 2, 2, 2, 1 | 10, 6, 7, 7, 7, 8, 8, 5, 4, 5, 8, 4 |
The bracket-anchored counts match the score. The row-profile counts are wrong on
every page, and wrong by a different amount each time — so no threshold fixes
them.
## The algorithm
1. **Deskew per page.** Projection-profile variance sweep over ±5°. Measured skew
on this song ranges 2.6° to +1.2° *between pages of the same PDF*, so per-page
is not optional.
2. **Find anchors.** Binarise, then morphological open with a tall thin kernel
(height ≈ 3% of the page) so only long vertical strokes survive. Take
connected components taller than 4% of the page; walk them tallest-first,
keeping each one whose y-extent doesn't overlap an already-kept anchor. Each
surviving stroke is one system.
3. **Expand to ink.** Compute the row-darkness profile on a despeckled copy, take
its ink runs, and assign each run to the nearest anchor by centre distance. A
system's extent is the union of its runs.
4. **Place cuts** at the midpoint between consecutive systems' ink extents.
Step 3 is what makes this work rather than the bracket alone: a bracket stops at
the last staff line, but the slice must include the **lyrics below it**. On page
2, system 1's bracket spans 177994 while its true ink extent is 1791071 — the
77px difference is the bottom voice's lyric line, which the bracket misses
entirely and the row profile finds.
## Consequences
- Detection needs both signals. Neither the column pass nor the row pass is
sufficient alone, so `detect.py` computes both.
- **Scores without brackets** — single-staff melodies, lead sheets — have no
anchors, and fall back to row-profile runs. That fallback is the *only* correct
behaviour there, since every ink run genuinely is its own system.
- Bar numbers printed above a system (this score uses 11, 16, …) sit in their own
ink run and get absorbed into the nearest system by step 3. That is right: they
belong to the system they label.
- A page number can be absorbed the same way if its darkness clears the profile
threshold, inflating the last system's extent. The content rectangle and the
bottom discard slice both prevent this; don't rely on the threshold.