noteman-slicer/docs/spec.md

noteman-slicer — specification

What the tool does and how it behaves. Vocabulary is in CONTEXT.md; the reasoning behind the expensive decisions is in docs/adr/.

Scope

A local, single-user tool that turns a score PDF into the ordered slice images noteman consumes, plus the navigation markers that sit on them. It automates the mechanical part of noteman's ingestion boundary.

It is not a GIMP replacement. Erasing previous-owner pencil marks, chord letters and breath marks stays in GIMP — the irreducible manual part, which GIMP with a stylus already does well.

One PDF → one song → one project → one bundle. Never a many-to-one in any direction. A PDF is either bitmap or vector, never mixed.

Why it's separate from noteman

Splitting it out removed the double-implementation constraint — in-app, every operation needs both a fast browser preview and a real server-side render, and that constraint is what priced dewarp and brush masking out entirely, not the algorithms. It also removed infrastructure noteman doesn't otherwise need (a scratch workspace for multi-MB rasters, an edit-list table, cleanup sweeps for orphaned temp files, poppler in the Docker image, an admin UI surface), and unlocked real image libraries.

It costs nothing: song creation is admin-only, done at home, once per song.

Operating principle

Detection proposes, the human disposes. Every automatic result — skew angle, cut positions, source type, staff height, ink bounds — is a suggestion the user confirms or modifies before it is committed. There is no unattended mode. See ADR 0004.

Geometry model

One geometry model, two renderers. Geometry is stored in normalised page coordinates (0–1 of page width and height), independent of DPI and of which renderer produces the output. Only the final stage differs.

Concept	Raster	Vector
Cut	y in pixels	y in PDF user space
Discard	drop the slice	drop the slice
Content rectangle	crop before cutting	clip before cutting
Trim	crop to ink bbox	crop viewBox to ink bbox
Uniform width	transparent right pad	wider viewBox, same content
Staff-height normalise	scale factor	scale factor
Deskew, levels, ink→alpha, 1920 cap	yes	no

Only the raster renderer ships in release 1 — see ADR 0002. The editor is one editor regardless, since a vector PDF has to be rasterized just to display it on screen.

Pipeline order

load raster → deskew → levels → content rect → cut → discard
            → trim → scale → pad → ink→alpha → encode

Load raster differs by source type. A scanned PDF carries one full-page image per page, and that image is the scan — extract it at its native resolution (extract_image) rather than re-rendering the page. Re-rendering at a fixed 600 DPI resamples a 200 DPI scan up by 3×, which triples the pixel count and adds no detail. A vector PDF has no embedded raster, so it is rendered — see the DPI note in Reference values.

The rest of the order is not arbitrary:

• Levels before anything geometric, so the trim bounding box is computed on the image that actually ships.
• Content rect before cutting, so margin junk never enters a slice.
• Trim before scale, since the scale factor derives from the widest trimmed slice.

Slices, cuts and discard

A page starts as a single slice; each cut splits one slice into two. Slices therefore tile the page with no gaps and no overlap.

Headers, footers and blank regions leave the song via a discard flag, not via cuts at the page edges. Modelling a slice as "the region between two cuts" leaks: page 2 has no header, so it would need an invented top cut whose position depends on whether that page happens to have one.

Cut placement is forgiving — anywhere inside the whitespace yields the same output, because trim crops to ink afterwards.

A cut is a polyline, not a line. Two points — a straight horizontal boundary — is the ordinary case and what detection proposes. Extra vertices exist because publishers routinely print a section label in the left margin at the same height as the previous system's lyrics. On page 1 of Engel (Bosse/Partitura edition), the boxed VERSE 1 label and the preceding system's bass lyric line occupy the same rows: ink is present on both sides of the page throughout that band, so no horizontal line separates them. VERSE 1 belongs to system 2, the lyrics to system 1. The cut has to step — above the label on the left, below the lyrics on the right.

A slice bounded by a non-straight cut is not rectangular. Its image is the bounding box of the region, with everything outside the region made transparent. That composites invisibly on the viewer's sheet, so nothing downstream needs to know. This is also why masking must paint transparency rather than white.

Content rectangle

The region of a page that holds music, set per PDF and adjustable per page, applied before cutting. Everything outside it is dropped.

This handles margin junk structurally rather than case-by-case, because margin junk is by definition outside the music: scan-edge bands, spine shadows, and page numbers printed in the side margin level with a system. That last one matters more than it looks — see the trim consequences below.

Trim, scale, pad

Trim tight on all four sides, per slice. This normalises away the left-margin drift between scanned pages, and flattens the engraved indent of the first system — correct here, since noteman strips the printed header the indent made room for.

Two consequences:

• A stray speck at the far left anchors the trim, shifting that slice relative to its neighbours. Mitigate by ignoring connected components under a few hundred pixels (cv2.connectedComponentsWithStats) when computing the bounding box.
• A page number in the side margin level with a system would set that slice's bounding box, which sets the song's widest slice, which scales the whole song down. One artefact, whole song smaller. Hence the content rectangle.

Scale is normalised on staff height, not width. Width-based scaling assumes every slice comes from the same scan at the same DPI. It breaks for a rescanned page, a PDF mixing scan generations, or a re-engraved replacement system — whose width depends on how much music is in it, not on matching its neighbours. Staff height is the invariant a reader perceives as "the notes are the same size", and it falls out of the same row-darkness profile detection already computes.

Two steps, both per song: normalise every slice to a common staff height, then scale the song uniformly so its widest slice lands at 1920px. That is a ceiling, never a target — never upscale. A song that comes out narrower stays narrower; enlarging a 600 DPI scan past its real resolution buys softness and bytes and no detail.

Pad narrower slices with transparency on the right, so every slice in a song is the same width, flush left, notes the same size. A short system simply ends earlier.

Encoding

Lossless WebP, with levels applied and alpha quantised to 16 levels. Roughly 7 KB per slice, about 450 KB for a 65-system song. Lossy encodings and the alternative formats are all larger for this content — measured, with the figures, in ADR 0003.

Ink handling is luminance → alpha: ink forced to pure black, alpha = 255 − luminance. Not pixel == white thresholding — staff lines are antialiased, and binary removal leaves jagged edges.

Detection

All of it is a suggestion, all of it overridable.

Deskew — per page, and not optionally so: measured skew varies from −2.6° to +1.2° between pages of the same PDF. Staff lines are by far the strongest horizontal signal in sheet music, so a projection-profile variance sweep over ±5° finds the angle reliably — sum row-darkness for each candidate angle, take the angle of maximum variance. Run on a downscaled copy. Pair with a manual slider.

Systems — anchored on the vertical bracket that spans a system's staves, not on gaps in the row-darkness profile. A row profile cannot distinguish an inter-staff gap from an inter-system gap on multi-voice choral scores, and gets the system count wrong on every page. See ADR 0006 for the measurement and the full algorithm. In outline:

1. Binarise; morphological open with a tall thin kernel so only long vertical strokes survive; keep non-overlapping components taller than 4% of the page. Each is one system.
2. Take ink runs from the row-darkness profile and assign each to the nearest anchor. A system's extent is the union of its runs — this is what pulls in the lyrics printed below the last staff, which the bracket stops short of.
3. Propose cuts at the midpoint between consecutive systems' ink extents, and pre-set the discard flag on a page's top and bottom slice when they contain no system.

Scores with no bracket — single-staff melodies, lead sheets — have no anchors and fall back to row-profile runs, which is correct there.

Staff height — peak-to-peak spacing in the row profile.

Content rectangle — proposed per page from the staff lines. Staff lines are long horizontal runs, while a scan-edge shadow, a spine darkening and the streak a dirty scanner glass leaves are all vertical, so opening with a wide flat kernel keeps the music and erases the artefacts. Three details make it work:

• Only rows inside detected systems are searched. Otherwise a horizontal scan artefact above or below the music is itself a long horizontal run, and it reaches the paper edge.
• The horizontal bounds come from a percentile of the staff-line extents, not their maximum. Where an artefact touches the end of a staff line the two merge into one component; a page has dozens of staff lines and only a few are contaminated.
• The left bound also considers the brackets, which sit left of every staff line. A bound taken from staff lines alone crops the bracket off, and a bracket is notation.

Only the horizontal bounds are proposed. Vertically the cuts and discard flags already isolate the header and footer, and cropping the top would risk clipping a tempo mark or a section label above the first staff.

Source type — get_images(full=True) / get_drawings() proposes bitmap or vector per PDF; the tool asks the user to confirm before routing. (full=True is required, or get_image_bbox rejects the item.)

Despeckle feeds detection only. A median blur plus dropping tiny connected components denoises the profile the detector reads; the shipped pixels come from the levels-adjusted image. The known failure mode is specks, so the fix belongs on the signal, not the output.

Levels

Two sliders per song (black point, white point) applied via cv2.LUT, with a per-page override.

In release 1, not deferred: with alpha = 255 − luminance, a scan's greyness becomes transparency, so a faint or yellowed source produces washed-out notes on a hazy background and nothing downstream can rescue it. Set the white point just under the paper's luminance and the paper vanishes completely; set the black point at the ink's darkest and notes go solid. It is also the single biggest lever on output size.

Detection proposes both, like it proposes cuts and skew, because the default 0–255 is the one setting whose harm is invisible until the bundle is on a tablet. Notation is two-tone, so Otsu's split between ink and paper is the measurement; the points sit halfway from it to each end of the range, leaving the ramp between them as the antialiasing. A page already scanned bilevel has no interior split — Otsu degenerates to 0 there — and is left at 0–255. The proposal is per page and the median becomes the song's, so a near-blank page cannot set it.

Adaptive methods (CLAHE, adaptive thresholding) are the trap — tuned for text, they eat the thin stuff on notation: hairpin tips, slur ends, ledger lines, tapered beams. A global LUT whose effect you can see beats a local algorithm you can't predict.

Editor

PySide6. QGraphicsView provides the viewport — pan, zoom, screen↔image coordinate mapping, resampling, hit-testing — which would otherwise be ~150 lines of hand-rolled geometry. cv2.imshow was rejected: OpenCV's highgui is GTK/X11 and lands on XWayland at best, and it has no text input at all.

What the editor does: pan and zoom the page, drag cut lines, toggle discard, adjust the content rectangle, move the levels sliders, place markers, fill in song metadata, export.

Marker placement needs a slice picker — a QListView in icon mode over the slice previews — since every jump source stores an explicit target. One widget serving all six jump types.

Project file

Autosaved JSON beside the source PDF, holding the source path and hash, cuts, discards, content rectangle, skew angles, levels, staff-height overrides, markers and metadata. The bundle is generated from it, so export is a pure function of the project file plus the PDF.

It buys crash safety and resume across sessions, since authoring is trickle-in: a session interrupted halfway through a 12-page scan picks up exactly where it stopped.

A project is spent once its song has been exported. Export records that in the file, and opening the PDF again starts a fresh session from detection rather than resuming. A re-cut therefore never inherits decisions that have already shipped. --resume overrides it on the edit, export and project commands when the old state really is wanted.

The cost is deliberate: re-export is no longer free. Changing the width cap or adding the SVG renderer later means re-cutting each song by hand rather than regenerating every bundle from its project file.

The project file references the PDF and never contains it; the hash lets the editor warn if the PDF changed underneath.

Markers

Placed here rather than in noteman: at cut time you are already reading the score page by page at full resolution, so the Segno, the Coda sign, the "to coda" text and the rehearsal letters are on screen. Deferring means reading the whole score a second time to find the same symbols.

noteman's vocabulary, carried verbatim — rehearsal_letter, section_label, segno, coda, fine, repeat_start, repeat_end, volta, to_coda, ds_al_coda, ds_al_fine, dc_al_coda, dc_al_fine, generic_jump. A small stable enum, but real coupling: adding a type means changing both repos.

Three shapes among them:

• Bare tags: segno, coda, fine, repeat_start, repeat_end.
• Tags with free text: rehearsal_letter ("C"), section_label ("CHORUS"), volta ("1.").
• Jump sources: to_coda, ds_al_coda, ds_al_fine, dc_al_coda, dc_al_fine, generic_jump.

Every jump source stores its target slice explicitly. noteman's viewer currently resolves by type — a to_coda finds the song's unique coda at tap time — but that puts an unwritten "exactly one Coda per song" invariant into a contract between two separately-maintained repos, enforced by neither. Authoring the target costs one click on a slice already on screen, and in exchange the bundle is self-describing and a score with two codas simply works.

Bundle

The only channel to noteman. No API, no direct upload — see ADR 0001.

song.zip
  song.json
  original.pdf
  001.webp 002.webp …

{
  "v": 1,
  "title": "…", "composer": "…", "arranger": "…",
  "slices": [
    { "file": "001.webp" },
    { "file": "002.webp", "markers": [{ "type": "rehearsal_letter", "label": "A" }] },
    { "file": "003.webp", "markers": [{ "type": "to_coda", "destination": 7 }] }
  ]
}

Array order is slice order — one ordering, not two. Markers nest inside the slice they sit on, so indices appear in exactly one place: a jump source's destination.

"v": 1 is eight bytes of insurance. The bundle is the only channel, MIDI and MP3s are planned for a later phase, and bundles are archived artifacts that may be re-imported a year later.

Otherwise: plain zip, no manifest beyond this, no checksums, hand-fixable. Python's zipfile is stdlib; the import side needs one zero-dep library (fflate), since Bun has zlib but no zip reader.

Contents: slices, markers, the original PDF, and song-level metadata (title, subtitle, composer, original artist, arranger, lyricist, translator, tempo, voice list). Metadata is included not because the slicer transforms it but because you have to read the title block anyway to mark the header slice discarded — typing the fields while it's on screen beats reopening the PDF later.

Title is required; everything else is optional and omitted when blank. Tempo is an integer, beats per minute — a number can drive a metronome and a starting-chord playback where Andante cannot, and two people will not agree what Andante means. noteman's column is currently free-form text and needs changing; see bundle-format.md.

Rehearsal MIDI and MP3s are deliberately out of the first bundle.

One rule for the import side

Import creates a new song only; never re-import onto an existing one. Jump destinations reference slices by ID, so replacing a song's slices silently orphans every marker on it. Re-cutting happens before marker authoring in practice, so forbidding it costs nothing and prevents a genuinely nasty data-loss mode. Re-export from the project file is the supported path.

Implementation

Python, chosen for OpenCV access and iteration speed. Installed as a package via uv tool install --editable ., which puts a noteman-slicer command on PATH that runs from any directory with no venv to activate. The one cwd trap: load bundled data via Path(__file__).parent or importlib.resources, never a relative path.

Dependencies: PyMuPDF, PySide6, opencv-python-headless, numpy — all wheels, no system packages. PyMuPDF covers every PDF need; see ADR 0005.

Verified: cv2 5.0.0 writes 4-channel lossless WebP with alpha preserved byte-exact (IMWRITE_WEBP_QUALITY, 101).

Module boundaries: pdf.py (load, source-type detect, rasterize), detect.py (deskew, row-darkness profile, system runs, staff height), bundle.py, editor.py.

Changes required in noteman

On noteman's timeline, not the slicer's — but release 1 produces artifacts nothing consumes until this lands.

1. Delete the sharp normalisation pipeline. The slicer's output is final.
2. Bundle import — unzip → read song.json → create song → insert slices in array order → insert markers, mapping index → new slice UUID → store the PDF.
3. Jump sources carry explicit destinations — destinationSliceId is already nullable on every marker type, so this is viewer logic, not schema.

SVG support on the noteman side (image/svg+xml in the upload path, .svg in CONTENT_TYPES, and a CSP header on SVG responses) is not needed until the SVG renderer ships.

Phasing

Release 1 — editor + detection + bundle export, raster only. Vector PDFs are rasterized like everything else; they're the clean case, where deskew is a no-op and detection works best. Levels, content rectangle, discard, markers, project file.

Everything else is deferred and tracked as issues on the Gitea repo.

Reference values

• Final slice width cap: 1920px, matching the viewer sheet's max-width. A ceiling, not a target.
• Output format: lossless WebP.
• Working resolution:
  • Scanned sources — the embedded image's native resolution. Never re-render. Real scans in this corpus run ~200 DPI (1653×2332 for A4), which is below the 1920 cap, so those songs ship narrower than 1920 and are never upscaled.
  • Vector sources — 600 DPI, configurable. A4 @ 600 DPI is ~4960×7016 px; the ~2.6× downsample to 1920 is itself a quality win via antialiasing. 300 DPI would suffice for the target, but 600 buys headroom for deskew resampling.
• A slice = one system = one full line of music across all voices, typically 4–12 bars, lyrics intact.
• Upload/bundle sizes are not constrained by noteman's old 25 MB/file limits — the bundle bypasses that path entirely.