Replace the handoff notes with a durable spec and ADRs
The handoff was written as a message to relay information; several decisions lived only there. Split into permanent homes: - docs/spec.md — scope, geometry model, pipeline order, detection, levels, editor, project file, markers, bundle format, noteman's changes, reference values - ADR 0002 — raster only in release 1; SVG slices measured at 40x WebP naive, 2.6x with a bounding-box cull, deferred on risk not size - ADR 0003 — lossless WebP with levels and 16-level alpha; every lossy option and alternative format measured larger for line art - ADR 0004 — detection proposes, the human disposes; no unattended mode - ADR 0005 — PyMuPDF for all PDF access, accepting AGPL - ADR 0006 — systems are found by vertical brackets, not row-darkness gaps, which miscount every page of a 6-voice score Also from testing against the hardest score in the repertoire: scanned PDFs carry their scan as an embedded image and must be extracted at native resolution rather than re-rendered at 600 DPI, and per-page deskew is mandatory (skew varies -2.6 to +1.2 within one PDF).
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# Raster only in release 1; the SVG renderer is deferred
|
||||
|
||||
Vector PDFs are most of the newer corpus, and keeping them vector all the way to
|
||||
the viewer was an early goal — sheet music is line art, and SVG stays crisp at any
|
||||
tablet zoom. We measured it before building it, and decided to **rasterize vector
|
||||
sources like everything else in release 1** and revisit the SVG renderer once
|
||||
real songs have been cut.
|
||||
|
||||
## The measurement
|
||||
|
||||
One real vector song, 6 pages, 65 systems, rendered both ways:
|
||||
|
||||
| Approach | Total | vs WebP |
|
||||
|---|---|---|
|
||||
| WebP slices (600 DPI → 1920, ink→alpha, lossless) | 1.19 MB | 1× |
|
||||
| SVG, naive `viewBox` + `clipPath` | 26.0 MB | 40× |
|
||||
| SVG, `set_cropbox` per band | 26.5 MB | 41× |
|
||||
| SVG, bounding-box cull + glyph subset | 3.09 MB | 2.6× |
|
||||
|
||||
- **The naive cut is unusable.** A `viewBox` + `clipPath` slice contains the
|
||||
entire page's geometry and merely hides eleven-twelfths of it.
|
||||
- **`set_cropbox` does not help.** MuPDF renders full page content regardless of
|
||||
the crop, so there is no free version of the cull.
|
||||
- **The cull works.** PyMuPDF emits a `<defs>` glyph table (111 KB of a 256 KB
|
||||
page) referenced by `<use transform="matrix(...)">`, plus body `<path>`
|
||||
elements. Filter both by y-extent, then keep only the glyphs the survivors
|
||||
reference. Roughly 50 lines, 15× improvement.
|
||||
|
||||
## Why defer, given the cull works
|
||||
|
||||
**Not size.** At 3.1 MB vs 1.2 MB per song — 225 MB vs 87 MB across a 73-song
|
||||
corpus — both are nothing on a homelab. The measurement killed the lazy
|
||||
implementation, not the idea.
|
||||
|
||||
What defers it is risk and missing evidence:
|
||||
|
||||
- The cull is **heuristic parsing**: glyph extents bounded at baseline ±14pt,
|
||||
path extents read from raw `d` coordinates. It is over-inclusive by design, so
|
||||
it fails safe — but "fails safe" still means a slice quietly carrying a
|
||||
neighbour's slur, or a hairline dropped because the y-window was wrong on some
|
||||
publisher's output. That needs eyeballing per song, a QA loop the raster path
|
||||
doesn't have.
|
||||
- Rendering 65 complex SVGs in a scrolling column may be slower than 65 WebPs.
|
||||
Unmeasured.
|
||||
- **The deciding question is unanswerable from here**: does 1920px WebP actually
|
||||
feel insufficient when pinch-zooming on a tablet? Cutting real songs answers
|
||||
it; more measurement doesn't.
|
||||
|
||||
Vector PDFs are also the *clean* case for the raster path — deskew is a no-op,
|
||||
detection works best, there are no scan artefacts — so rasterizing them is not a
|
||||
degraded fallback.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The geometry model stays **renderer-agnostic**, in normalised page coordinates,
|
||||
so adding the SVG renderer later is an output stage rather than a redesign.
|
||||
- **Re-export from the project file** regenerates every song's bundle without
|
||||
repeating human work, so songs cut before the SVG renderer exists are not
|
||||
stranded.
|
||||
- noteman needs no SVG support (`image/svg+xml`, `.svg` content type, CSP header
|
||||
on SVG responses) until the renderer ships.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Lossless WebP, with levels and alpha quantised to 16 levels
|
||||
|
||||
Slice images are encoded as **lossless WebP**, with the levels adjustment applied
|
||||
and the alpha channel quantised to 16 levels. About 7 KB per slice, ~450 KB for a
|
||||
65-system song. Every lossy option and every alternative format measured
|
||||
*larger* for this content, which is the opposite of the usual intuition — hence
|
||||
this record.
|
||||
|
||||
## The measurement
|
||||
|
||||
20 slices of one real song, levels applied throughout, relative to plain lossless
|
||||
WebP:
|
||||
|
||||
| | vs baseline | |
|
||||
|---|---|---|
|
||||
| **WebP lossless + alpha quantised to 16** | **68%** | chosen |
|
||||
| AVIF q60 | 90% | lossy, for 10% |
|
||||
| WebP lossless | 100% | baseline |
|
||||
| WebP lossy q85 (alpha) | 107% | |
|
||||
| AVIF q85 | 114% | |
|
||||
| JXL lossless | 130–133% | |
|
||||
| WebP lossy q85 (opaque ink-on-white) | 158% | |
|
||||
| PNG grayscale + alpha | 165% | |
|
||||
| AVIF lossless | 188% | |
|
||||
|
||||
Separately, before levels: applying levels alone takes 338 KB → 211 KB, a 38%
|
||||
reduction.
|
||||
|
||||
## Four results that contradict an instinct
|
||||
|
||||
- **Lossy is bigger than lossless here.** Not a quality problem — the measured
|
||||
difference between q85 and lossless is max 12/255, mean 0.33, i.e. invisible.
|
||||
Lossy VP8 simply spends more bits on sharp black/white edges than VP8L's
|
||||
palette and predictor transforms do, and notation is nothing but sharp edges.
|
||||
The "q85 looks fine" intuition comes from photographs and inverts here.
|
||||
- **AVIF and JXL both lose**, AVIF lossless by nearly 2×. Their lossless modes
|
||||
are afterthoughts on photo codecs. WebP's VP8L is close to purpose-built for
|
||||
flat two-tone line art — sheet music is the content type it is best at. JXL
|
||||
additionally has no path forward in Chrome.
|
||||
- **Alpha costs nothing.** Opaque ink-on-white and black-plus-alpha are within
|
||||
0.1% at lossless, so paper-tint removal and future non-rectangular slices are
|
||||
free.
|
||||
- **Levels is the single biggest lever** — 38%, as a side effect of a control
|
||||
that exists for quality reasons anyway. Pushing the white point below the
|
||||
paper's luminance sets vast regions to exactly `alpha = 0`, which costs almost
|
||||
nothing to encode.
|
||||
|
||||
Alpha quantisation to 16 levels is imperceptible: antialiased edges span 2–3 px
|
||||
at 1920, and 16 steps across that is below notice. 8 levels starts to gamble on
|
||||
thin strokes.
|
||||
|
||||
## Rejected as not worth it
|
||||
|
||||
- **Encoder effort tuning** — Pillow's `method=6` buys 3% and a dependency.
|
||||
- **`alpha_quality=60`** — 24%, for less control than quantisation gives.
|
||||
- **Grayscale WebP** — no such mode exists. It wouldn't help anyway: the RGB
|
||||
channels are constant black and compress to nearly nothing, so alpha is the
|
||||
entire payload.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Detection proposes, the human disposes — there is no unattended mode
|
||||
|
||||
Every automatic result the slicer produces — skew angle, cut positions, source
|
||||
type, staff height, ink bounds — is a **suggestion the user confirms or modifies**
|
||||
before it is committed. There is no batch mode, no headless "slice this folder",
|
||||
and no code path that writes a bundle without a human having looked at it.
|
||||
|
||||
This is a constraint on the tool's shape, not a UI preference, which is why it
|
||||
gets an ADR: it deletes an entire phase of the original plan and it will look
|
||||
like a missing feature to anyone who finds the detection code and wonders why it
|
||||
isn't wired to a CLI.
|
||||
|
||||
## Why
|
||||
|
||||
The corpus is PDFs from a choir's distribution channel, and quality varies
|
||||
wildly — clean vector engravings at one end, noisy scans with a previous owner's
|
||||
pencil markings at the other. **Testing showed the detection algorithms produce
|
||||
unusable slices on any source with speckles or otherwise poor quality.** Not
|
||||
slightly-off slices: unusable ones.
|
||||
|
||||
But the same testing showed the suggestions land *close* on decent sources —
|
||||
close enough that correcting them is faster than placing cuts from scratch. So
|
||||
detection earns its place as an accelerator, and loses any claim to being
|
||||
load-bearing.
|
||||
|
||||
## What this rejected
|
||||
|
||||
The original plan's **Phase A** was a deliberately non-interactive CLI:
|
||||
rasterize, auto-deskew, auto-detect boundaries, write numbered slices, and fix
|
||||
the misses by hand in GIMP. Its justification was "learn the failure modes before
|
||||
designing the editor," which is a good idea.
|
||||
|
||||
It doesn't survive the premise. A CLI whose output can't be trusted has GIMP as
|
||||
its repair path — routing work back into the manual process the project exists to
|
||||
remove. A diagnostic variant (dump per-page PNGs with proposed cuts drawn in red)
|
||||
was considered and also dropped: it only re-shows a failure already confirmed by
|
||||
testing, and the editor shows the same thing live.
|
||||
|
||||
Release 1 is therefore the editor and detection together. There is no smaller
|
||||
first release that is actually usable.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Manual placement is the primary interaction**, not a correction affordance.
|
||||
The editor must be fully usable with detection producing nothing.
|
||||
- **Despeckling targets the detector, not the output.** The known failure mode is
|
||||
specks, so a median blur and a small-component filter clean the row-darkness
|
||||
profile the detector reads; the shipped pixels come from the levels-adjusted
|
||||
image.
|
||||
- Cut placement is deliberately **forgiving** — anywhere in the whitespace gap
|
||||
yields the same output, since trim crops to ink afterwards. Precision is not
|
||||
asked of the human.
|
||||
- The editor should surface **slice edges**, not just cut lines, so trim
|
||||
anomalies (a speck anchoring the bounding box) are visible rather than
|
||||
discovered later in the viewer.
|
||||
@@ -0,0 +1,36 @@
|
||||
# PyMuPDF for all PDF access, accepting AGPL
|
||||
|
||||
All PDF work — rasterizing at a chosen DPI, exporting SVG, and inspecting page
|
||||
content to classify a source as bitmap or vector — goes through **PyMuPDF**. It
|
||||
is a single wheel with MuPDF bundled, so the tool needs no system packages. Its
|
||||
licence is **AGPL-3.0**, which we accept.
|
||||
|
||||
## Why not the permissive combination
|
||||
|
||||
The obvious permissive stack was `pypdfium2` (Apache/BSD) for rasterizing plus
|
||||
`mutool` or `pdftocairo` shelled out for SVG. Both of those are **system
|
||||
packages** — `mupdf-tools`, `poppler` — and a system package on the vector path
|
||||
is precisely the failure the language choice was made to avoid: the tool is
|
||||
supposed to install once and run from any directory on any machine.
|
||||
|
||||
The SVG step can't simply be skipped, either. Music glyphs come from a notation
|
||||
font (Emmentaler, Bravura, or Sibelius/Finale's). An SVG that *references* a font
|
||||
renders as garbage on a device that lacks it, so text must be converted to paths
|
||||
at export. PyMuPDF does this **by default** — `page.get_svg_image(text_as_path=1)`,
|
||||
verified to emit `<path>` elements and zero `<text>` — so the font risk is closed
|
||||
with no extra tooling.
|
||||
|
||||
Mixing the two (pypdfium2 for raster, PyMuPDF only for SVG) is the worst option:
|
||||
two libraries with overlapping responsibilities, and AGPL linked in anyway.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The AGPL propagates only if the slicer is published.** For a local personal
|
||||
tool it costs nothing. A future permissive release would need the rasterizer
|
||||
swapped back to `pypdfium2` — a contained change, since PDF access sits behind
|
||||
the renderer-agnostic geometry model.
|
||||
- **Source-type detection comes free** from the same library: `get_images()` plus
|
||||
a full-page-image area check distinguishes a scan from an engraving.
|
||||
- The SVG export path is present and working even though the SVG *renderer* is
|
||||
deferred — see
|
||||
[ADR 0002](0002-raster-only-svg-renderer-deferred.md).
|
||||
@@ -0,0 +1,67 @@
|
||||
# Systems are found by vertical brackets, not by row-darkness gaps
|
||||
|
||||
System detection anchors on the **vertical bracket / barline** that spans a
|
||||
system's staves, and uses the row-darkness profile only to expand each anchor to
|
||||
its ink extent. The obvious approach — find gaps in the row-darkness profile and
|
||||
cut in the middle of them — does not work on multi-voice choral scores, which is
|
||||
most of the corpus.
|
||||
|
||||
## Why the obvious approach fails
|
||||
|
||||
A row-darkness profile cannot distinguish an **inter-staff** gap from an
|
||||
**inter-system** gap. In a 6-voice closed score, one system is six staves joined
|
||||
by a bracket, and the gaps between those six staves look exactly like the gap
|
||||
between two systems — only smaller, and not reliably so.
|
||||
|
||||
Measured on *Ketun joululaulu*, a 12-page 6-voice arrangement and the hardest
|
||||
score in the repertoire:
|
||||
|
||||
- On page 2's first system, staff gaps run ~47px against a ~211px system gap. A
|
||||
merge threshold tuned there works.
|
||||
- On the same page's second system the lyrics fill the inter-staff gaps, so the
|
||||
ratios invert and the same threshold merges the wrong things.
|
||||
|
||||
Result across all 12 pages, row-profile-only versus bracket-anchored:
|
||||
|
||||
| | bracket-anchored | row-profile only |
|
||||
|---|---|---|
|
||||
| systems per page | 2, 2, 2, 2, 2, 2, 3, 2, 2, 2, 2, 1 | 10, 6, 7, 7, 7, 8, 8, 5, 4, 5, 8, 4 |
|
||||
|
||||
The bracket-anchored counts match the score. The row-profile counts are wrong on
|
||||
every page, and wrong by a different amount each time — so no threshold fixes
|
||||
them.
|
||||
|
||||
## The algorithm
|
||||
|
||||
1. **Deskew per page.** Projection-profile variance sweep over ±5°. Measured skew
|
||||
on this song ranges −2.6° to +1.2° *between pages of the same PDF*, so per-page
|
||||
is not optional.
|
||||
2. **Find anchors.** Binarise, then morphological open with a tall thin kernel
|
||||
(height ≈ 3% of the page) so only long vertical strokes survive. Take
|
||||
connected components taller than 4% of the page; walk them tallest-first,
|
||||
keeping each one whose y-extent doesn't overlap an already-kept anchor. Each
|
||||
surviving stroke is one system.
|
||||
3. **Expand to ink.** Compute the row-darkness profile on a despeckled copy, take
|
||||
its ink runs, and assign each run to the nearest anchor by centre distance. A
|
||||
system's extent is the union of its runs.
|
||||
4. **Place cuts** at the midpoint between consecutive systems' ink extents.
|
||||
|
||||
Step 3 is what makes this work rather than the bracket alone: a bracket stops at
|
||||
the last staff line, but the slice must include the **lyrics below it**. On page
|
||||
2, system 1's bracket spans 177–994 while its true ink extent is 179–1071 — the
|
||||
77px difference is the bottom voice's lyric line, which the bracket misses
|
||||
entirely and the row profile finds.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Detection needs both signals. Neither the column pass nor the row pass is
|
||||
sufficient alone, so `detect.py` computes both.
|
||||
- **Scores without brackets** — single-staff melodies, lead sheets — have no
|
||||
anchors, and fall back to row-profile runs. That fallback is the *only* correct
|
||||
behaviour there, since every ink run genuinely is its own system.
|
||||
- Bar numbers printed above a system (this score uses 11, 16, …) sit in their own
|
||||
ink run and get absorbed into the nearest system by step 3. That is right: they
|
||||
belong to the system they label.
|
||||
- A page number can be absorbed the same way if its darkness clears the profile
|
||||
threshold, inflating the last system's extent. The content rectangle and the
|
||||
bottom discard slice both prevent this; don't rely on the threshold.
|
||||
Reference in New Issue
Block a user