noteman-slicer/docs/spec.md

# noteman-slicer — specification

What the tool does and how it behaves. Vocabulary is in
[`CONTEXT.md`](../CONTEXT.md); the reasoning behind the expensive decisions is in
[`docs/adr/`](adr/).

## Scope

A local, single-user tool that turns a score PDF into the ordered slice images
[noteman](../../noteman) consumes, plus the navigation markers that sit on them.
It automates the mechanical part of noteman's ingestion boundary.

It is **not** a GIMP replacement. Erasing previous-owner pencil marks, chord
letters and breath marks stays in GIMP — the irreducible manual part, which GIMP
with a stylus already does well.

**One PDF → one song → one project → one bundle.** Never a many-to-one in any
direction. A PDF is either bitmap or vector, never mixed.

### Why it's separate from noteman

Splitting it out removed the double-implementation constraint — in-app, every
operation needs both a fast browser preview and a real server-side render, and
that constraint is what priced dewarp and brush masking out entirely, not the
algorithms. It also removed infrastructure noteman doesn't otherwise need (a
scratch workspace for multi-MB rasters, an edit-list table, cleanup sweeps for
orphaned temp files, poppler in the Docker image, an admin UI surface), and
unlocked real image libraries.

It costs nothing: song creation is admin-only, done at home, once per song.

## Operating principle

**Detection proposes, the human disposes.** Every automatic result — skew angle,
cut positions, source type, staff height, ink bounds — is a suggestion the user
confirms or modifies before it is committed. There is no unattended mode. See
[ADR 0004](adr/0004-detection-proposes-the-human-disposes.md).

## Geometry model

**One geometry model, two renderers.** Geometry is stored in **normalised page
coordinates** (0–1 of page width and height), independent of DPI and of which
renderer produces the output. Only the final stage differs.

| Concept | Raster | Vector |
|---|---|---|
| Cut | y in pixels | y in PDF user space |
| Discard | drop the slice | drop the slice |
| Content rectangle | crop before cutting | clip before cutting |
| Trim | crop to ink bbox | crop `viewBox` to ink bbox |
| Uniform width | transparent right pad | wider `viewBox`, same content |
| Staff-height normalise | scale factor | scale factor |
| Deskew, levels, ink→alpha, 1920 cap | yes | no |

Only the raster renderer ships in release 1 — see
[ADR 0002](adr/0002-raster-only-svg-renderer-deferred.md). The editor is one
editor regardless, since a vector PDF has to be rasterized just to display it on
screen.

### Pipeline order

```
load raster → deskew → levels → content rect → cut → discard
            → trim → scale → pad → ink→alpha → encode
```

**Load raster** differs by source type. A scanned PDF carries one full-page image
per page, and that image *is* the scan — extract it at its native resolution
(`extract_image`) rather than re-rendering the page. Re-rendering at a fixed
600 DPI resamples a 200 DPI scan up by 3×, which triples the pixel count and adds
no detail. A vector PDF has no embedded raster, so it is rendered — see the DPI
note in *Reference values*.

The rest of the order is not arbitrary:

- **Levels before anything geometric**, so the trim bounding box is computed on
  the image that actually ships.
- **Content rect before cutting**, so margin junk never enters a slice.
- **Trim before scale**, since the scale factor derives from the widest *trimmed*
  slice.

### Slices, cuts and discard

A page starts as a single slice; each cut splits one slice into two. Slices
therefore tile the page with no gaps and no overlap.

Headers, footers and blank regions leave the song via a **discard** flag, not via
cuts at the page edges. Modelling a slice as "the region between two cuts" leaks:
page 2 has no header, so it would need an invented top cut whose position depends
on whether that page happens to have one.

Cut placement is forgiving — anywhere inside the whitespace yields the same
output, because trim crops to ink afterwards.

**A cut is a polyline, not a line.** Two points — a straight horizontal
boundary — is the ordinary case and what detection proposes. Extra vertices exist
because publishers routinely print a section label in the left margin at the same
height as the *previous* system's lyrics. On page 1 of *Engel* (Bosse/Partitura
edition), the boxed `VERSE 1` label and the preceding system's bass lyric line
occupy the same rows: ink is present on both sides of the page throughout that
band, so no horizontal line separates them. `VERSE 1` belongs to system 2, the
lyrics to system 1. The cut has to step — above the label on the left, below the
lyrics on the right.

A slice bounded by a non-straight cut is **not rectangular**. Its image is the
bounding box of the region, with everything outside the region made transparent.
That composites invisibly on the viewer's sheet, so nothing downstream needs to
know. This is also why masking must paint transparency rather than white.

### Content rectangle

The region of a page that holds music, set per PDF and adjustable per page,
applied before cutting. Everything outside it is dropped.

This handles margin junk structurally rather than case-by-case, because margin
junk is by definition outside the music: scan-edge bands, spine shadows, and page
numbers printed in the side margin level with a system. That last one matters
more than it looks — see the trim consequences below.

### Trim, scale, pad

**Trim** tight on all four sides, per slice. This normalises away the left-margin
drift between scanned pages, and flattens the engraved indent of the first
system — correct here, since noteman strips the printed header the indent made
room for.

Two consequences:

- A stray speck at the far left anchors the trim, shifting that slice relative to
  its neighbours. Mitigate by ignoring connected components under a few hundred
  pixels (`cv2.connectedComponentsWithStats`) when computing the bounding box.
- A page number in the side margin level with a system would set that slice's
  bounding box, which sets the song's widest slice, which scales the whole song
  down. One artefact, whole song smaller. Hence the content rectangle.

**Scale is normalised on staff height, not width.** Width-based scaling assumes
every slice comes from the same scan at the same DPI. It breaks for a rescanned
page, a PDF mixing scan generations, or a re-engraved replacement system — whose
width depends on how much music is in it, not on matching its neighbours. Staff
height is the invariant a reader perceives as "the notes are the same size", and
it falls out of the same row-darkness profile detection already computes.

Two steps, both per song: normalise every slice to a common staff height, then
scale the song uniformly so its widest slice lands at **1920px**. That is a
ceiling, never a target — **never upscale**. A song that comes out narrower stays
narrower; enlarging a 600 DPI scan past its real resolution buys softness and
bytes and no detail.

**Pad** narrower slices with transparency on the right, so every slice in a song
is the same width, flush left, notes the same size. A short system simply ends
earlier.

### Encoding

**Lossless WebP, with levels applied and alpha quantised to 16 levels.** Roughly
7 KB per slice, about 450 KB for a 65-system song. Lossy encodings and the
alternative formats are all *larger* for this content — measured, with the
figures, in
[ADR 0003](adr/0003-lossless-webp-with-levels-and-alpha-quantisation.md).

Ink handling is luminance → alpha: ink forced to pure black,
`alpha = 255 − luminance`. Not `pixel == white` thresholding — staff lines are
antialiased, and binary removal leaves jagged edges.

## Detection

All of it is a suggestion, all of it overridable.

**Deskew** — per page, and not optionally so: measured skew varies from −2.6° to
+1.2° *between pages of the same PDF*. Staff lines are by far the strongest
horizontal signal in sheet music, so a projection-profile variance sweep over ±5°
finds the angle reliably — sum row-darkness for each candidate angle, take the
angle of maximum variance. Run on a downscaled copy. Pair with a manual slider.

**Systems** — anchored on the **vertical bracket** that spans a system's staves,
not on gaps in the row-darkness profile. A row profile cannot distinguish an
inter-staff gap from an inter-system gap on multi-voice choral scores, and gets
the system count wrong on every page. See
[ADR 0006](adr/0006-systems-are-found-by-brackets-not-row-gaps.md) for the
measurement and the full algorithm. In outline:

1. Binarise; morphological open with a tall thin kernel so only long vertical
   strokes survive; keep non-overlapping components taller than 4% of the page.
   Each is one system.
2. Take ink runs from the row-darkness profile and assign each to the nearest
   anchor. A system's extent is the union of its runs — this is what pulls in the
   lyrics printed *below* the last staff, which the bracket stops short of.
3. Propose cuts at the midpoint between consecutive systems' ink extents, and
   pre-set the discard flag on a page's top and bottom slice when they contain no
   system.

Scores with no bracket — single-staff melodies, lead sheets — have no anchors and
fall back to row-profile runs, which is correct there.

**Staff height** — peak-to-peak spacing in the row profile.

**Content rectangle** — proposed per page from the staff lines. Staff lines are
long *horizontal* runs, while a scan-edge shadow, a spine darkening and the
streak a dirty scanner glass leaves are all *vertical*, so opening with a wide
flat kernel keeps the music and erases the artefacts. Three details make it
work:

- Only rows inside detected systems are searched. Otherwise a horizontal scan
  artefact above or below the music is itself a long horizontal run, and it
  reaches the paper edge.
- The horizontal bounds come from a *percentile* of the staff-line extents, not
  their maximum. Where an artefact touches the end of a staff line the two merge
  into one component; a page has dozens of staff lines and only a few are
  contaminated.
- The left bound also considers the **brackets**, which sit left of every staff
  line. A bound taken from staff lines alone crops the bracket off, and a
  bracket is notation.

Only the horizontal bounds are proposed. Vertically the cuts and discard flags
already isolate the header and footer, and cropping the top would risk clipping
a tempo mark or a section label above the first staff.

**Source type** — `get_images(full=True)` / `get_drawings()` proposes bitmap or
vector per PDF; the tool asks the user to confirm before routing. (`full=True` is
required, or `get_image_bbox` rejects the item.)

**Despeckle feeds detection only.** A median blur plus dropping tiny connected
components denoises the *profile the detector reads*; the shipped pixels come
from the levels-adjusted image. The known failure mode is specks, so the fix
belongs on the signal, not the output.

## Levels

Two sliders per song (black point, white point) applied via `cv2.LUT`, with a
per-page override.

In release 1, not deferred: with `alpha = 255 − luminance`, a scan's greyness
*becomes* transparency, so a faint or yellowed source produces washed-out notes
on a hazy background and **nothing downstream can rescue it**. Set the white
point just under the paper's luminance and the paper vanishes completely; set the
black point at the ink's darkest and notes go solid. It is also the single
biggest lever on output size.

Adaptive methods (CLAHE, adaptive thresholding) are the trap — tuned for text,
they eat the thin stuff on notation: hairpin tips, slur ends, ledger lines,
tapered beams. A global LUT whose effect you can see beats a local algorithm you
can't predict.

## Editor

**PySide6.** `QGraphicsView` provides the viewport — pan, zoom, screen↔image
coordinate mapping, resampling, hit-testing — which would otherwise be ~150 lines
of hand-rolled geometry. `cv2.imshow` was rejected: OpenCV's highgui is GTK/X11
and lands on XWayland at best, and it has no text input at all.

What the editor does: pan and zoom the page, drag cut lines, toggle discard,
adjust the content rectangle, move the levels sliders, place markers, fill in
song metadata, export.

Marker placement needs a **slice picker** — a `QListView` in icon mode over the
slice previews — since every jump source stores an explicit target. One widget
serving all six jump types.

## Project file

Autosaved JSON beside the source PDF, holding the source path and hash, cuts,
discards, content rectangle, skew angles, levels, staff-height overrides, markers
and metadata. The bundle is *generated* from it, so export is a pure function of
the project file plus the PDF.

It buys crash safety and resume across sessions, since authoring is trickle-in:
a session interrupted halfway through a 12-page scan picks up exactly where it
stopped.

**A project is spent once its song has been exported.** Export records that in
the file, and opening the PDF again starts a *fresh session from detection*
rather than resuming. A re-cut therefore never inherits decisions that have
already shipped. `--resume` overrides it on the `edit`, `export` and `project`
commands when the old state really is wanted.

The cost is deliberate: re-export is no longer free. Changing the width cap or
adding the SVG renderer later means re-cutting each song by hand rather than
regenerating every bundle from its project file.

The project file references the PDF and never contains it; the hash lets the
editor warn if the PDF changed underneath.

## Markers

Placed here rather than in noteman: at cut time you are already reading the score
page by page at full resolution, so the Segno, the Coda sign, the "to coda" text
and the rehearsal letters are on screen. Deferring means reading the whole score a
second time to find the same symbols.

noteman's vocabulary, carried verbatim — `rehearsal_letter`, `section_label`,
`segno`, `coda`, `fine`, `repeat_start`, `repeat_end`, `volta`, `to_coda`,
`ds_al_coda`, `ds_al_fine`, `dc_al_coda`, `dc_al_fine`, `generic_jump`. A small
stable enum, but real coupling: adding a type means changing both repos.

Three shapes among them:

- **Bare tags:** `segno`, `coda`, `fine`, `repeat_start`, `repeat_end`.
- **Tags with free text:** `rehearsal_letter` ("C"), `section_label` ("CHORUS"),
  `volta` ("1.").
- **Jump sources:** `to_coda`, `ds_al_coda`, `ds_al_fine`, `dc_al_coda`,
  `dc_al_fine`, `generic_jump`.

**Every jump source stores its target slice explicitly.** noteman's viewer
currently resolves by type — a `to_coda` finds the song's unique `coda` at tap
time — but that puts an unwritten "exactly one Coda per song" invariant into a
contract between two separately-maintained repos, enforced by neither. Authoring
the target costs one click on a slice already on screen, and in exchange the
bundle is self-describing and a score with two codas simply works.

## Bundle

The only channel to noteman. No API, no direct upload — see
[ADR 0001](adr/0001-slicer-owns-image-processing-bundle-is-the-only-channel.md).

```
song.zip
  song.json
  original.pdf
  001.webp 002.webp …
```

```json
{
  "v": 1,
  "title": "…", "composer": "…", "arranger": "…",
  "slices": [
    { "file": "001.webp" },
    { "file": "002.webp", "markers": [{ "type": "rehearsal_letter", "label": "A" }] },
    { "file": "003.webp", "markers": [{ "type": "to_coda", "destination": 7 }] }
  ]
}
```

Array order **is** slice order — one ordering, not two. Markers nest inside the
slice they sit on, so indices appear in exactly one place: a jump source's
`destination`.

`"v": 1` is eight bytes of insurance. The bundle is the only channel, MIDI and
MP3s are planned for a later phase, and bundles are archived artifacts that may be
re-imported a year later.

Otherwise: plain zip, no manifest beyond this, no checksums, hand-fixable.
Python's `zipfile` is stdlib; the import side needs one zero-dep library
(`fflate`), since Bun has zlib but no zip reader.

**Contents:** slices, markers, the original PDF, and song-level metadata (title,
subtitle, composer, original artist, arranger, lyricist, translator, tempo,
voice list). Metadata is included not because the slicer transforms it but
because you have to read the title block anyway to mark the header slice
discarded — typing the fields while it's on screen beats reopening the PDF
later.

**Title is required**; everything else is optional and omitted when blank.
**Tempo is an integer**, beats per minute — a number can drive a metronome and
a starting-chord playback where *Andante* cannot, and two people will not agree
what *Andante* means. noteman's column is currently free-form text and needs
changing; see [`bundle-format.md`](bundle-format.md).

Rehearsal MIDI and MP3s are deliberately out of the first bundle.

### One rule for the import side

**Import creates a new song only; never re-import onto an existing one.** Jump
destinations reference slices by ID, so replacing a song's slices silently
orphans every marker on it. Re-cutting happens *before* marker authoring in
practice, so forbidding it costs nothing and prevents a genuinely nasty data-loss
mode. Re-export from the project file is the supported path.

## Implementation

**Python**, chosen for OpenCV access and iteration speed. Installed as a package
via `uv tool install --editable .`, which puts a `noteman-slicer` command on PATH
that runs from any directory with no venv to activate. The one cwd trap: load
bundled data via `Path(__file__).parent` or `importlib.resources`, never a
relative path.

Dependencies: **PyMuPDF**, **PySide6**, **opencv-python-headless**, **numpy** —
all wheels, no system packages. PyMuPDF covers every PDF need; see
[ADR 0005](adr/0005-pymupdf-for-all-pdf-access.md).

Verified: `cv2` 5.0.0 writes 4-channel lossless WebP with alpha preserved
byte-exact (`IMWRITE_WEBP_QUALITY, 101`).

Module boundaries: `pdf.py` (load, source-type detect, rasterize), `detect.py`
(deskew, row-darkness profile, system runs, staff height), `bundle.py`,
`editor.py`.

## Changes required in noteman

On noteman's timeline, not the slicer's — but release 1 produces artifacts
nothing consumes until this lands.

1. **Delete the sharp normalisation pipeline.** The slicer's output is final.
2. **Bundle import** — unzip → read `song.json` → create song → insert slices in
   array order → insert markers, mapping index → new slice UUID → store the PDF.
3. **Jump sources carry explicit destinations** — `destinationSliceId` is already
   nullable on every marker type, so this is viewer logic, not schema.

SVG support on the noteman side (`image/svg+xml` in the upload path, `.svg` in
`CONTENT_TYPES`, and a CSP header on SVG responses) is not needed until the SVG
renderer ships.

## Phasing

**Release 1 — editor + detection + bundle export, raster only.** Vector PDFs are
rasterized like everything else; they're the clean case, where deskew is a no-op
and detection works best. Levels, content rectangle, discard, markers, project
file.

Everything else is deferred and tracked as issues on the Gitea repo.

## Reference values

- Final slice width cap: **1920px**, matching the viewer sheet's max-width. A
  ceiling, not a target.
- Output format: **lossless WebP**.
- Working resolution:
  - **Scanned sources — the embedded image's native resolution.** Never
    re-render. Real scans in this corpus run ~200 DPI (1653×2332 for A4), which
    is *below* the 1920 cap, so those songs ship narrower than 1920 and are never
    upscaled.
  - **Vector sources — 600 DPI**, configurable. A4 @ 600 DPI is ~4960×7016 px;
    the ~2.6× downsample to 1920 is itself a quality win via antialiasing. 300
    DPI would suffice for the target, but 600 buys headroom for deskew
    resampling.
- A slice = **one system** = one full line of music across all voices, typically
  4–12 bars, lyrics intact.
- Upload/bundle sizes are not constrained by noteman's old 25 MB/file limits —
  the bundle bypasses that path entirely.