Replace the handoff notes with a durable spec and ADRs
The handoff was written as a message to relay information; several decisions lived only there. Split into permanent homes: - docs/spec.md — scope, geometry model, pipeline order, detection, levels, editor, project file, markers, bundle format, noteman's changes, reference values - ADR 0002 — raster only in release 1; SVG slices measured at 40x WebP naive, 2.6x with a bounding-box cull, deferred on risk not size - ADR 0003 — lossless WebP with levels and 16-level alpha; every lossy option and alternative format measured larger for line art - ADR 0004 — detection proposes, the human disposes; no unattended mode - ADR 0005 — PyMuPDF for all PDF access, accepting AGPL - ADR 0006 — systems are found by vertical brackets, not row-darkness gaps, which miscount every page of a 6-voice score Also from testing against the hardest score in the repertoire: scanned PDFs carry their scan as an embedded image and must be extracted at native resolution rather than re-rendered at 600 DPI, and per-page deskew is mandatory (skew varies -2.6 to +1.2 within one PDF).
This commit is contained in:
Esa Kataja
+377
@@ -0,0 +1,377 @@
|
||||
# noteman-slicer — specification
|
||||
|
||||
What the tool does and how it behaves. Vocabulary is in
|
||||
[`CONTEXT.md`](../CONTEXT.md); the reasoning behind the expensive decisions is in
|
||||
[`docs/adr/`](adr/).
|
||||
|
||||
## Scope
|
||||
|
||||
A local, single-user tool that turns a score PDF into the ordered slice images
|
||||
[noteman](../../noteman) consumes, plus the navigation markers that sit on them.
|
||||
It automates the mechanical part of noteman's ingestion boundary.
|
||||
|
||||
It is **not** a GIMP replacement. Erasing previous-owner pencil marks, chord
|
||||
letters and breath marks stays in GIMP — the irreducible manual part, which GIMP
|
||||
with a stylus already does well.
|
||||
|
||||
**One PDF → one song → one project → one bundle.** Never a many-to-one in any
|
||||
direction. A PDF is either bitmap or vector, never mixed.
|
||||
|
||||
### Why it's separate from noteman
|
||||
|
||||
Splitting it out removed the double-implementation constraint — in-app, every
|
||||
operation needs both a fast browser preview and a real server-side render, and
|
||||
that constraint is what priced dewarp and brush masking out entirely, not the
|
||||
algorithms. It also removed infrastructure noteman doesn't otherwise need (a
|
||||
scratch workspace for multi-MB rasters, an edit-list table, cleanup sweeps for
|
||||
orphaned temp files, poppler in the Docker image, an admin UI surface), and
|
||||
unlocked real image libraries.
|
||||
|
||||
It costs nothing: song creation is admin-only, done at home, once per song.
|
||||
|
||||
## Operating principle
|
||||
|
||||
**Detection proposes, the human disposes.** Every automatic result — skew angle,
|
||||
cut positions, source type, staff height, ink bounds — is a suggestion the user
|
||||
confirms or modifies before it is committed. There is no unattended mode. See
|
||||
[ADR 0004](adr/0004-detection-proposes-the-human-disposes.md).
|
||||
|
||||
## Geometry model
|
||||
|
||||
**One geometry model, two renderers.** Geometry is stored in **normalised page
|
||||
coordinates** (0–1 of page width and height), independent of DPI and of which
|
||||
renderer produces the output. Only the final stage differs.
|
||||
|
||||
| Concept | Raster | Vector |
|
||||
|---|---|---|
|
||||
| Cut | y in pixels | y in PDF user space |
|
||||
| Discard | drop the slice | drop the slice |
|
||||
| Content rectangle | crop before cutting | clip before cutting |
|
||||
| Trim | crop to ink bbox | crop `viewBox` to ink bbox |
|
||||
| Uniform width | transparent right pad | wider `viewBox`, same content |
|
||||
| Staff-height normalise | scale factor | scale factor |
|
||||
| Deskew, levels, ink→alpha, 1920 cap | yes | no |
|
||||
|
||||
Only the raster renderer ships in release 1 — see
|
||||
[ADR 0002](adr/0002-raster-only-svg-renderer-deferred.md). The editor is one
|
||||
editor regardless, since a vector PDF has to be rasterized just to display it on
|
||||
screen.
|
||||
|
||||
### Pipeline order
|
||||
|
||||
```
|
||||
load raster → deskew → levels → content rect → cut → discard
|
||||
→ trim → scale → pad → ink→alpha → encode
|
||||
```
|
||||
|
||||
**Load raster** differs by source type. A scanned PDF carries one full-page image
|
||||
per page, and that image *is* the scan — extract it at its native resolution
|
||||
(`extract_image`) rather than re-rendering the page. Re-rendering at a fixed
|
||||
600 DPI resamples a 200 DPI scan up by 3×, which triples the pixel count and adds
|
||||
no detail. A vector PDF has no embedded raster, so it is rendered — see the DPI
|
||||
note in *Reference values*.
|
||||
|
||||
The rest of the order is not arbitrary:
|
||||
|
||||
- **Levels before anything geometric**, so the trim bounding box is computed on
|
||||
the image that actually ships.
|
||||
- **Content rect before cutting**, so margin junk never enters a slice.
|
||||
- **Trim before scale**, since the scale factor derives from the widest *trimmed*
|
||||
slice.
|
||||
|
||||
### Slices, cuts and discard
|
||||
|
||||
A page starts as a single slice; each cut splits one slice into two. Slices
|
||||
therefore tile the page with no gaps and no overlap.
|
||||
|
||||
Headers, footers and blank regions leave the song via a **discard** flag, not via
|
||||
cuts at the page edges. Modelling a slice as "the region between two cuts" leaks:
|
||||
page 2 has no header, so it would need an invented top cut whose position depends
|
||||
on whether that page happens to have one.
|
||||
|
||||
Cut placement is forgiving — anywhere inside the whitespace gap yields the same
|
||||
output, because trim crops to ink afterwards.
|
||||
|
||||
### Content rectangle
|
||||
|
||||
The region of a page that holds music, set per PDF and adjustable per page,
|
||||
applied before cutting. Everything outside it is dropped.
|
||||
|
||||
This handles margin junk structurally rather than case-by-case, because margin
|
||||
junk is by definition outside the music: scan-edge bands, spine shadows, and page
|
||||
numbers printed in the side margin level with a system. That last one matters
|
||||
more than it looks — see the trim consequences below.
|
||||
|
||||
### Trim, scale, pad
|
||||
|
||||
**Trim** tight on all four sides, per slice. This normalises away the left-margin
|
||||
drift between scanned pages, and flattens the engraved indent of the first
|
||||
system — correct here, since noteman strips the printed header the indent made
|
||||
room for.
|
||||
|
||||
Two consequences:
|
||||
|
||||
- A stray speck at the far left anchors the trim, shifting that slice relative to
|
||||
its neighbours. Mitigate by ignoring connected components under a few hundred
|
||||
pixels (`cv2.connectedComponentsWithStats`) when computing the bounding box.
|
||||
- A page number in the side margin level with a system would set that slice's
|
||||
bounding box, which sets the song's widest slice, which scales the whole song
|
||||
down. One artefact, whole song smaller. Hence the content rectangle.
|
||||
|
||||
**Scale is normalised on staff height, not width.** Width-based scaling assumes
|
||||
every slice comes from the same scan at the same DPI. It breaks for a rescanned
|
||||
page, a PDF mixing scan generations, or a re-engraved replacement system — whose
|
||||
width depends on how much music is in it, not on matching its neighbours. Staff
|
||||
height is the invariant a reader perceives as "the notes are the same size", and
|
||||
it falls out of the same row-darkness profile detection already computes.
|
||||
|
||||
Two steps, both per song: normalise every slice to a common staff height, then
|
||||
scale the song uniformly so its widest slice lands at **1920px**. That is a
|
||||
ceiling, never a target — **never upscale**. A song that comes out narrower stays
|
||||
narrower; enlarging a 600 DPI scan past its real resolution buys softness and
|
||||
bytes and no detail.
|
||||
|
||||
**Pad** narrower slices with transparency on the right, so every slice in a song
|
||||
is the same width, flush left, notes the same size. A short system simply ends
|
||||
earlier.
|
||||
|
||||
### Encoding
|
||||
|
||||
**Lossless WebP, with levels applied and alpha quantised to 16 levels.** Roughly
|
||||
7 KB per slice, about 450 KB for a 65-system song. Lossy encodings and the
|
||||
alternative formats are all *larger* for this content — measured, with the
|
||||
figures, in
|
||||
[ADR 0003](adr/0003-lossless-webp-with-levels-and-alpha-quantisation.md).
|
||||
|
||||
Ink handling is luminance → alpha: ink forced to pure black,
|
||||
`alpha = 255 − luminance`. Not `pixel == white` thresholding — staff lines are
|
||||
antialiased, and binary removal leaves jagged edges.
|
||||
|
||||
## Detection
|
||||
|
||||
All of it is a suggestion, all of it overridable.
|
||||
|
||||
**Deskew** — per page, and not optionally so: measured skew varies from −2.6° to
|
||||
+1.2° *between pages of the same PDF*. Staff lines are by far the strongest
|
||||
horizontal signal in sheet music, so a projection-profile variance sweep over ±5°
|
||||
finds the angle reliably — sum row-darkness for each candidate angle, take the
|
||||
angle of maximum variance. Run on a downscaled copy. Pair with a manual slider.
|
||||
|
||||
**Systems** — anchored on the **vertical bracket** that spans a system's staves,
|
||||
not on gaps in the row-darkness profile. A row profile cannot distinguish an
|
||||
inter-staff gap from an inter-system gap on multi-voice choral scores, and gets
|
||||
the system count wrong on every page. See
|
||||
[ADR 0006](adr/0006-systems-are-found-by-brackets-not-row-gaps.md) for the
|
||||
measurement and the full algorithm. In outline:
|
||||
|
||||
1. Binarise; morphological open with a tall thin kernel so only long vertical
|
||||
strokes survive; keep non-overlapping components taller than 4% of the page.
|
||||
Each is one system.
|
||||
2. Take ink runs from the row-darkness profile and assign each to the nearest
|
||||
anchor. A system's extent is the union of its runs — this is what pulls in the
|
||||
lyrics printed *below* the last staff, which the bracket stops short of.
|
||||
3. Propose cuts at the midpoint between consecutive systems' ink extents, and
|
||||
pre-set the discard flag on a page's top and bottom slice when they contain no
|
||||
system.
|
||||
|
||||
Scores with no bracket — single-staff melodies, lead sheets — have no anchors and
|
||||
fall back to row-profile runs, which is correct there.
|
||||
|
||||
**Staff height** — peak-to-peak spacing in the row profile.
|
||||
|
||||
**Source type** — `get_images(full=True)` / `get_drawings()` proposes bitmap or
|
||||
vector per PDF; the tool asks the user to confirm before routing. (`full=True` is
|
||||
required, or `get_image_bbox` rejects the item.)
|
||||
|
||||
**Despeckle feeds detection only.** A median blur plus dropping tiny connected
|
||||
components denoises the *profile the detector reads*; the shipped pixels come
|
||||
from the levels-adjusted image. The known failure mode is specks, so the fix
|
||||
belongs on the signal, not the output.
|
||||
|
||||
## Levels
|
||||
|
||||
Two sliders per song (black point, white point) applied via `cv2.LUT`, with a
|
||||
per-page override.
|
||||
|
||||
In release 1, not deferred: with `alpha = 255 − luminance`, a scan's greyness
|
||||
*becomes* transparency, so a faint or yellowed source produces washed-out notes
|
||||
on a hazy background and **nothing downstream can rescue it**. Set the white
|
||||
point just under the paper's luminance and the paper vanishes completely; set the
|
||||
black point at the ink's darkest and notes go solid. It is also the single
|
||||
biggest lever on output size.
|
||||
|
||||
Adaptive methods (CLAHE, adaptive thresholding) are the trap — tuned for text,
|
||||
they eat the thin stuff on notation: hairpin tips, slur ends, ledger lines,
|
||||
tapered beams. A global LUT whose effect you can see beats a local algorithm you
|
||||
can't predict.
|
||||
|
||||
## Editor
|
||||
|
||||
**PySide6.** `QGraphicsView` provides the viewport — pan, zoom, screen↔image
|
||||
coordinate mapping, resampling, hit-testing — which would otherwise be ~150 lines
|
||||
of hand-rolled geometry. `cv2.imshow` was rejected: OpenCV's highgui is GTK/X11
|
||||
and lands on XWayland at best, and it has no text input at all.
|
||||
|
||||
What the editor does: pan and zoom the page, drag cut lines, toggle discard,
|
||||
adjust the content rectangle, move the levels sliders, place markers, fill in
|
||||
song metadata, export.
|
||||
|
||||
Marker placement needs a **slice picker** — a `QListView` in icon mode over the
|
||||
slice previews — since every jump source stores an explicit target. One widget
|
||||
serving all six jump types.
|
||||
|
||||
## Project file
|
||||
|
||||
Autosaved JSON beside the source PDF, holding the source path and hash, cuts,
|
||||
discards, content rectangle, skew angles, levels, staff-height overrides, markers
|
||||
and metadata. The bundle is *generated* from it, so export is a pure function of
|
||||
the project file plus the PDF.
|
||||
|
||||
It buys crash safety, resume across sessions (authoring is trickle-in), and
|
||||
**re-export** — change the 1920 cap, fix one cut, or add the SVG renderer later,
|
||||
and every song's bundle regenerates without repeating any human work.
|
||||
|
||||
The project file references the PDF and never contains it; the hash lets the
|
||||
editor warn if the PDF changed underneath.
|
||||
|
||||
## Markers
|
||||
|
||||
Placed here rather than in noteman: at cut time you are already reading the score
|
||||
page by page at full resolution, so the Segno, the Coda sign, the "to coda" text
|
||||
and the rehearsal letters are on screen. Deferring means reading the whole score a
|
||||
second time to find the same symbols.
|
||||
|
||||
noteman's vocabulary, carried verbatim — `rehearsal_letter`, `section_label`,
|
||||
`segno`, `coda`, `fine`, `repeat_start`, `repeat_end`, `volta`, `to_coda`,
|
||||
`ds_al_coda`, `ds_al_fine`, `dc_al_coda`, `dc_al_fine`, `generic_jump`. A small
|
||||
stable enum, but real coupling: adding a type means changing both repos.
|
||||
|
||||
Three shapes among them:
|
||||
|
||||
- **Bare tags:** `segno`, `coda`, `fine`, `repeat_start`, `repeat_end`.
|
||||
- **Tags with free text:** `rehearsal_letter` ("C"), `section_label` ("CHORUS"),
|
||||
`volta` ("1.").
|
||||
- **Jump sources:** `to_coda`, `ds_al_coda`, `ds_al_fine`, `dc_al_coda`,
|
||||
`dc_al_fine`, `generic_jump`.
|
||||
|
||||
**Every jump source stores its target slice explicitly.** noteman's viewer
|
||||
currently resolves by type — a `to_coda` finds the song's unique `coda` at tap
|
||||
time — but that puts an unwritten "exactly one Coda per song" invariant into a
|
||||
contract between two separately-maintained repos, enforced by neither. Authoring
|
||||
the target costs one click on a slice already on screen, and in exchange the
|
||||
bundle is self-describing and a score with two codas simply works.
|
||||
|
||||
## Bundle
|
||||
|
||||
The only channel to noteman. No API, no direct upload — see
|
||||
[ADR 0001](adr/0001-slicer-owns-image-processing-bundle-is-the-only-channel.md).
|
||||
|
||||
```
|
||||
song.zip
|
||||
song.json
|
||||
original.pdf
|
||||
001.webp 002.webp …
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"v": 1,
|
||||
"title": "…", "composer": "…", "arranger": "…",
|
||||
"slices": [
|
||||
{ "file": "001.webp" },
|
||||
{ "file": "002.webp", "markers": [{ "type": "rehearsal_letter", "label": "A" }] },
|
||||
{ "file": "003.webp", "markers": [{ "type": "to_coda", "destination": 7 }] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Array order **is** slice order — one ordering, not two. Markers nest inside the
|
||||
slice they sit on, so indices appear in exactly one place: a jump source's
|
||||
`destination`.
|
||||
|
||||
`"v": 1` is eight bytes of insurance. The bundle is the only channel, MIDI and
|
||||
MP3s are planned for a later phase, and bundles are archived artifacts that may be
|
||||
re-imported a year later.
|
||||
|
||||
Otherwise: plain zip, no manifest beyond this, no checksums, hand-fixable.
|
||||
Python's `zipfile` is stdlib; the import side needs one zero-dep library
|
||||
(`fflate`), since Bun has zlib but no zip reader.
|
||||
|
||||
**Contents:** slices, markers, the original PDF, and song-level text metadata
|
||||
(title, subtitle, composer, original artist, arranger, lyricist, translator,
|
||||
voice list). Metadata is included not because the slicer transforms it but
|
||||
because you have to read the title block anyway to mark the header slice
|
||||
discarded — typing eight fields while it's on screen beats reopening the PDF
|
||||
later.
|
||||
|
||||
Rehearsal MIDI and MP3s are deliberately out of the first bundle.
|
||||
|
||||
### One rule for the import side
|
||||
|
||||
**Import creates a new song only; never re-import onto an existing one.** Jump
|
||||
destinations reference slices by ID, so replacing a song's slices silently
|
||||
orphans every marker on it. Re-cutting happens *before* marker authoring in
|
||||
practice, so forbidding it costs nothing and prevents a genuinely nasty data-loss
|
||||
mode. Re-export from the project file is the supported path.
|
||||
|
||||
## Implementation
|
||||
|
||||
**Python**, chosen for OpenCV access and iteration speed. Installed as a package
|
||||
via `uv tool install --editable .`, which puts a `noteman-slicer` command on PATH
|
||||
that runs from any directory with no venv to activate. The one cwd trap: load
|
||||
bundled data via `Path(__file__).parent` or `importlib.resources`, never a
|
||||
relative path.
|
||||
|
||||
Dependencies: **PyMuPDF**, **PySide6**, **opencv-python-headless**, **numpy** —
|
||||
all wheels, no system packages. PyMuPDF covers every PDF need; see
|
||||
[ADR 0005](adr/0005-pymupdf-for-all-pdf-access.md).
|
||||
|
||||
Verified: `cv2` 5.0.0 writes 4-channel lossless WebP with alpha preserved
|
||||
byte-exact (`IMWRITE_WEBP_QUALITY, 101`).
|
||||
|
||||
Module boundaries: `pdf.py` (load, source-type detect, rasterize), `detect.py`
|
||||
(deskew, row-darkness profile, system runs, staff height), `bundle.py`,
|
||||
`editor.py`.
|
||||
|
||||
## Changes required in noteman
|
||||
|
||||
On noteman's timeline, not the slicer's — but release 1 produces artifacts
|
||||
nothing consumes until this lands.
|
||||
|
||||
1. **Delete the sharp normalisation pipeline.** The slicer's output is final.
|
||||
2. **Bundle import** — unzip → read `song.json` → create song → insert slices in
|
||||
array order → insert markers, mapping index → new slice UUID → store the PDF.
|
||||
3. **Jump sources carry explicit destinations** — `destinationSliceId` is already
|
||||
nullable on every marker type, so this is viewer logic, not schema.
|
||||
|
||||
SVG support on the noteman side (`image/svg+xml` in the upload path, `.svg` in
|
||||
`CONTENT_TYPES`, and a CSP header on SVG responses) is not needed until the SVG
|
||||
renderer ships.
|
||||
|
||||
## Phasing
|
||||
|
||||
**Release 1 — editor + detection + bundle export, raster only.** Vector PDFs are
|
||||
rasterized like everything else; they're the clean case, where deskew is a no-op
|
||||
and detection works best. Levels, content rectangle, discard, markers, project
|
||||
file.
|
||||
|
||||
Everything else is deferred and tracked as issues on the Gitea repo.
|
||||
|
||||
## Reference values
|
||||
|
||||
- Final slice width cap: **1920px**, matching the viewer sheet's max-width. A
|
||||
ceiling, not a target.
|
||||
- Output format: **lossless WebP**.
|
||||
- Working resolution:
|
||||
- **Scanned sources — the embedded image's native resolution.** Never
|
||||
re-render. Real scans in this corpus run ~200 DPI (1653×2332 for A4), which
|
||||
is *below* the 1920 cap, so those songs ship narrower than 1920 and are never
|
||||
upscaled.
|
||||
- **Vector sources — 600 DPI**, configurable. A4 @ 600 DPI is ~4960×7016 px;
|
||||
the ~2.6× downsample to 1920 is itself a quality win via antialiasing. 300
|
||||
DPI would suffice for the target, but 600 buys headroom for deskew
|
||||
resampling.
|
||||
- A slice = **one system** = one full line of music across all voices, typically
|
||||
4–12 bars, lyrics intact.
|
||||
- Upload/bundle sizes are not constrained by noteman's old 25 MB/file limits —
|
||||
the bundle bypasses that path entirely.
|
||||
Reference in New Issue
Block a user