Merge dev: rotated scans, detection and levels fixes, editor tweaks

docs/guide.md

@@ -44,10 +44,11 @@ off.
    one back.
 4. **Set the content rectangle.** Drag the blue edges so they hold the music and
    nothing else. This is the horizontal crop for every slice on the page.
-5. **Set black and white points.** Pull the *White point* down until the paper
-   goes pure white and its texture disappears; pull *Black point* up until the
-   notes are solid black rather than grey mush. Scans need this; clean digital
-   PDFs usually don't.
+5. **Check black and white points.** These arrive proposed from the scan, like
+   the cuts do. If the paper still shows texture, pull *White point* down; if the
+   notes look grey rather than solid, pull *Black point* up. Getting this wrong
+   is the one mistake you cannot see until the bundle is on the tablet — grey ink
+   becomes half-transparent ink, and nothing downstream can rescue it.
 
 Page Up / Page Down move between pages. Levels carry over from the previous
 page, so a consistent scan only needs setting once.
@@ -77,7 +78,8 @@ and "Andante" cannot.
 
 ## Export
 
-**Export bundle…**, choose where the `.zip` goes, done. Inside are the slice
+**Export bundle…**, choose where the `.zip` goes, done. It is named after the
+song's title — *Bicycle Race* becomes `Bicycle-Race.zip`. Inside are the slice
 images in order, their markers, the song metadata, and the original PDF as the
 archive copy. That zip is the whole interface to noteman; hand it over and open
 it there.

docs/spec.md

@@ -236,6 +236,14 @@ point just under the paper's luminance and the paper vanishes completely; set th
 black point at the ink's darkest and notes go solid. It is also the single
 biggest lever on output size.
 
+**Detection proposes both**, like it proposes cuts and skew, because the default
+0–255 is the one setting whose harm is invisible until the bundle is on a tablet.
+Notation is two-tone, so Otsu's split between ink and paper is the measurement;
+the points sit halfway from it to each end of the range, leaving the ramp between
+them as the antialiasing. A page already scanned bilevel has no interior split —
+Otsu degenerates to 0 there — and is left at 0–255. The proposal is per page and
+the median becomes the song's, so a near-blank page cannot set it.
+
 Adaptive methods (CLAHE, adaptive thresholding) are the trap — tuned for text,
 they eat the thin stuff on notation: hairpin tips, slur ends, ledger lines,
 tapered beams. A global LUT whose effect you can see beats a local algorithm you

noteman_slicer/bundle.py

@@ -13,6 +13,7 @@ a jump source's `destination`.
 from __future__ import annotations
 
 import json
+import re
 import zipfile
 from pathlib import Path
 
@@ -39,6 +40,19 @@ METADATA_FIELDS = (
 NUMERIC_FIELDS = frozenset({"tempo"})
 
 
+def filename(project: Project) -> str:
+    """The bundle's name, from the song's title.
+
+    Spaces become dashes and anything that is not a letter, digit, dash, dot or
+    underscore goes. Letters keep their accents — ä and ö are not a filesystem's
+    problem — but a leading dot would make the bundle invisible.
+    """
+    # Drop the unsafe characters before collapsing whitespace, not after, or
+    # "Sävel & Ääni" keeps the dash the ampersand left behind.
+    title = re.sub(r"[^\w\s.-]", "", (project.metadata.get("title") or ""))
+    return f"{re.sub(r'\s+', '-', title.strip()).lstrip('.-') or 'song'}.zip"
+
+
 def _engraving(project: Project, page: int, slot: int) -> dict | None:
     """The notation behind a re-engraved slice, or None for a scanned one.
 

noteman_slicer/cli.py

@@ -87,7 +87,7 @@ def _export(args: argparse.Namespace) -> int:
     elif project.source_changed():
         print("WARNING: the PDF has changed since these cuts were made")
 
-    out = Path(args.out) if args.out else source.path.with_suffix(".zip")
+    out = Path(args.out) if args.out else source.path.with_name(bundle.filename(project))
     try:
         bundle.write(project, source, out)
     except ValueError as error:

noteman_slicer/detect.py

@@ -26,6 +26,7 @@ _SKEW_WORK_SCALE = 0.25
 _INK = 128  # below this is ink, above is paper
 _ANCHOR_KERNEL = 0.03  # vertical open kernel, as a fraction of page height
 _ANCHOR_MIN = 0.04  # a bracket is at least this tall, as a fraction of page
+_ANCHOR_MAX_RATIO = 2.0  # a stroke this much taller than the typical one is an artefact
 _PROFILE_FLOOR = 0.02  # ink-run threshold, as a fraction of the profile peak
 _EXPAND_REACH = 1.5  # how far past the bracket a system's ink reaches, in staff heights
 _STAFF_KERNEL = 0.05  # horizontal open kernel, as a fraction of page width
@@ -64,6 +65,7 @@ class PageDetection:
     systems: list[System] = field(default_factory=list)
     cuts: list[int] = field(default_factory=list)
     content: tuple[float, float, float, float] = (0.0, 0.0, 1.0, 1.0)
+    levels: tuple[int, int] = (0, 255)
 
     @property
     def bracketless(self) -> bool:
@@ -123,6 +125,17 @@ def system_anchors(gray: np.ndarray) -> list[Anchor]:
         if stats[i, cv2.CC_STAT_HEIGHT] > h * _ANCHOR_MIN
     ]
 
+    # A scanner leaves a dark line down the sheet edge — the binder shadow, the
+    # glass, the page next to it — and it runs the whole height of the scan.
+    # Being the tallest stroke on the page it wins every overlap below and
+    # swallows every system into one. A page's brackets and barlines are all
+    # about one system tall, so anything wildly taller than the typical stroke
+    # is not notation. Relative, not an absolute fraction of the page: a page
+    # holding one big system is legitimate and must survive.
+    if len(tall) > 1:
+        limit = float(np.median([a.bottom - a.top for a in tall])) * _ANCHOR_MAX_RATIO
+        tall = [a for a in tall if a.bottom - a.top <= limit] or tall
+
     # Tallest first, keeping only strokes that don't overlap one already kept:
     # a system's barlines all overlap its bracket, so each system yields one.
     # The kept stroke is the tallest, which is the bracket rather than a barline.
@@ -280,6 +293,26 @@ def staff_height(gray: np.ndarray, top: int, bottom: int) -> float | None:
     return float(np.median(intra) * 4)  # 5 lines, 4 spaces
 
 
+def ink_levels(gray: np.ndarray) -> tuple[int, int]:
+    """Black and white points that put the ink on black and the paper on white.
+
+    Left at 0–255 a slice ships whatever grey the scanner produced, and the
+    downscale to the song's width then blends every stroke edge further, so a
+    fine engraving arrives on the tablet as a wash. Notation is two-tone by
+    nature — ink and paper, nothing in between — so Otsu's split is exactly the
+    measurement wanted, and the points sit halfway to each end of the range from
+    it. Halfway rather than at the split itself: the ramp between them is the
+    antialiasing, and collapsing it would leave the notes jagged.
+    """
+    split = float(cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[0])
+    if split < 1:
+        # A page already scanned bilevel has no interior split to find, and
+        # Otsu degenerates to 0. There is nothing between ink and paper to
+        # stretch, so leave the sliders where they are.
+        return 0, 255
+    return int(split / 2), int(split + (255 - split) / 2)
+
+
 def _gap(run: tuple[int, int], anchor: Anchor) -> int:
     """Vertical distance between an ink run and a bracket; 0 if they overlap."""
     start, end = run
@@ -375,5 +408,9 @@ def detect_page(gray: np.ndarray, skew: float | None = None) -> PageDetection:
     # would risk clipping a tempo mark or a section label above the first staff.
     left, right = content_columns(straight, anchors)
     return PageDetection(
-        skew=angle, systems=systems, cuts=cuts, content=(left, 0.0, right, 1.0)
+        skew=angle,
+        systems=systems,
+        cuts=cuts,
+        content=(left, 0.0, right, 1.0),
+        levels=ink_levels(straight),
     )

noteman_slicer/editor.py

@@ -75,6 +75,8 @@ _CUT = QColor(220, 40, 40)
 _CUT_ACTIVE = QColor(255, 120, 0)
 _VERTEX = QColor(255, 200, 0)
 _DISCARD = QColor(120, 120, 140, 90)
+_SELECT = QColor(0, 170, 0)
+_SELECT_WASH = QColor(0, 200, 60, 40)
 _RECT = QColor(40, 140, 220)
 _MARKER = QColor(150, 60, 190)
 _ENGRAVED = QColor(200, 120, 0)
@@ -104,6 +106,7 @@ class PageView(QGraphicsView):
         self.selected_slice = 0
         self.picking = False
         self._drag: tuple[str, int, int] | None = None
+        self._fitted = False
 
     # -- state ------------------------------------------------------------
 
@@ -137,10 +140,15 @@ class PageView(QGraphicsView):
                     self._slice_polygon(slot, w, h), QPen(Qt.NoPen), QBrush(_DISCARD)
                 )
 
-        # The selected slice, outlined so trim anomalies are visible.
-        pen = QPen(QColor(0, 170, 0), 2)
+        # The selected slice. The outline alone is nearly invisible: its top and
+        # bottom edges run under the cut lines drawn over them, leaving two thin
+        # verticals at the page margins. A wash says which slice is selected at
+        # a glance; the outline stays, because it is what shows trim anomalies.
+        selected = self._slice_polygon(self.selected_slice, w, h)
+        scene.addPolygon(selected, QPen(Qt.NoPen), QBrush(_SELECT_WASH))
+        pen = QPen(_SELECT, 3)
         pen.setCosmetic(True)
-        scene.addPolygon(self._slice_polygon(self.selected_slice, w, h), pen)
+        scene.addPolygon(selected, pen)
 
         x0, y0, x1, y1 = self.project.page_content_rect(self.page_index)
         pen = QPen(_RECT, 2, Qt.DashLine)
@@ -361,6 +369,15 @@ class PageView(QGraphicsView):
         self.redraw()
         self.changed.emit()
 
+    def resizeEvent(self, event) -> None:
+        super().resizeEvent(event)
+        # The fit in show_page runs before the window has been laid out, when
+        # the viewport is still its default size, so the first page opens at
+        # some arbitrary zoom. Redo it once, when the real size arrives.
+        if not self._fitted and self.pixmap is not None:
+            self._fitted = True
+            self.fitInView(self.scene().sceneRect(), Qt.KeepAspectRatio)
+
     def wheelEvent(self, event) -> None:
         factor = 1.15 if event.angleDelta().y() > 0 else 1 / 1.15
         self.scale(factor, factor)
@@ -844,7 +861,10 @@ class Editor(QMainWindow):
             self.metadata["title"].setFocus()
             return
         target, _ = QFileDialog.getSaveFileName(
-            self, "Export bundle", str(self.source.path.with_suffix(".zip")), "Bundle (*.zip)"
+            self,
+            "Export bundle",
+            str(self.source.path.with_name(bundle.filename(self.project))),
+            "Bundle (*.zip)",
         )
         if not target:
             return

noteman_slicer/pdf.py

@@ -90,13 +90,28 @@ def page_raster(source: Source, index: int) -> np.ndarray:
             # Pixmap(doc, xref) rather than decoding extract_image() bytes:
             # MuPDF handles JBIG2 and CCITT, which no image library will.
             pix = pymupdf.Pixmap(source.doc, xref)
-            return _to_gray(pix)
+            # The embedded image is in its own orientation, not the page's: a
+            # scanner that fed the sheet sideways stores it landscape and the
+            # PDF sets /Rotate so viewers turn it upright. Extracting by xref
+            # bypasses that, so apply it here — otherwise every system runs
+            # down the page and detection finds nothing.
+            return _rotate(_to_gray(pix), page.rotation)
         # A scanned PDF whose page has no embedded image (a blank, or a
         # cover typeset in vector). Rendering is the only option left.
 
     return _to_gray(page.get_pixmap(dpi=source.render_dpi, colorspace=pymupdf.csGRAY))
 
 
+def _rotate(gray: np.ndarray, degrees: int) -> np.ndarray:
+    """Turn a page raster clockwise by a multiple of 90°, as /Rotate means it.
+
+    ponytail: quarter turns only. A page rotated by anything else would need
+    resampling, and no scanner produces one.
+    """
+    turns = round(degrees / 90) % 4
+    return np.ascontiguousarray(np.rot90(gray, -turns)) if turns else gray
+
+
 def _to_gray(pix: pymupdf.Pixmap) -> np.ndarray:
     if pix.alpha or pix.colorspace is None or pix.colorspace.n != 1:
         pix = pymupdf.Pixmap(pymupdf.csGRAY, pix)

noteman_slicer/project.py

@@ -15,6 +15,7 @@ import hashlib
 import json
 from dataclasses import dataclass, field
 from pathlib import Path
+from statistics import median
 
 from .detect import PageDetection
 
@@ -296,7 +297,18 @@ class Project:
                     content_rect=detection.content,
                 )
             )
-        return cls(source=source, source_hash=hash_file(source), pages=pages)
+        # Levels per song, not per page: a scanner's contrast does not change
+        # between sheets, and one pair of sliders for the whole song is what a
+        # user actually wants to nudge. The median keeps a near-blank page —
+        # where the ink/paper split is guesswork — from setting them.
+        proposals = [d.levels for d in detections] or [(0, 255)]
+        levels = (
+            int(median(b for b, _ in proposals)),
+            int(median(w for _, w in proposals)),
+        )
+        return cls(
+            source=source, source_hash=hash_file(source), pages=pages, levels=levels
+        )
 
     def save(self, path: Path | None = None) -> Path:
         """Atomic write, so a crash mid-save cannot destroy the previous state."""

noteman_slicer/render.py

@@ -26,7 +26,10 @@ from .project import Cut, Project
 
 MAX_WIDTH = 1920
 ALPHA_LEVELS = 16  # quantising alpha costs nothing visible and ~32% of the bytes
-_SPECK_AREA = 300  # ink blobs smaller than this don't anchor a trim
+# A row or column carrying less ink than this is a fleck, not content: at least
+# this many pixels, and at least this share of the slice's own size.
+_SPECK_INK = 8
+_SPECK_SHARE = 0.005
 
 
 @dataclass
@@ -105,26 +108,24 @@ def _ink_bbox(gray: np.ndarray) -> tuple[int, int, int, int] | None:
 
     One scan fleck at the far left would otherwise anchor the trim and shift
     that slice relative to every other one.
+
+    Measured per row and per column rather than per blob. Judging each blob on
+    its own area throws away a whole line of lyrics — every letter is its own
+    small component, and no single one is big enough to keep — which is how a
+    slice loses its bottom voice's words. A row carrying a line of text carries
+    plenty of ink *in total*, and a fleck's row carries almost none.
     """
-    ink = (gray < 200).astype(np.uint8)
-    count, _, stats, _ = cv2.connectedComponentsWithStats(ink, 8)
-    boxes = [
-        (
-            stats[i, cv2.CC_STAT_LEFT],
-            stats[i, cv2.CC_STAT_TOP],
-            stats[i, cv2.CC_STAT_LEFT] + stats[i, cv2.CC_STAT_WIDTH],
-            stats[i, cv2.CC_STAT_TOP] + stats[i, cv2.CC_STAT_HEIGHT],
-        )
-        for i in range(1, count)
-        if stats[i, cv2.CC_STAT_AREA] >= _SPECK_AREA
-    ]
-    if not boxes:
+    ink = gray < 200
+    rows, cols = ink.sum(axis=1), ink.sum(axis=0)
+    kept_rows = np.where(rows >= max(_SPECK_INK, ink.shape[1] * _SPECK_SHARE))[0]
+    kept_cols = np.where(cols >= max(_SPECK_INK, ink.shape[0] * _SPECK_SHARE))[0]
+    if not kept_rows.size or not kept_cols.size:
         return None
     return (
-        min(b[0] for b in boxes),
-        min(b[1] for b in boxes),
-        max(b[2] for b in boxes),
-        max(b[3] for b in boxes),
+        int(kept_cols[0]),
+        int(kept_rows[0]),
+        int(kept_cols[-1]) + 1,
+        int(kept_rows[-1]) + 1,
     )
 
 

tests/test_detect.py

@@ -14,7 +14,12 @@ import numpy as np
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 
-from noteman_slicer.detect import deskew, deskew_angle, detect_page  # noqa: E402
+from noteman_slicer.detect import (  # noqa: E402
+    deskew,
+    deskew_angle,
+    detect_page,
+    ink_levels,
+)
 
 W, H = 1000, 1400
 STAFF_GAP = 15  # → staff height 60, so expansion reaches 90px past a bracket
@@ -71,6 +76,23 @@ def main() -> int:
         found = deskew_angle(deskew(page, angle))
         assert abs(found + angle) <= 0.15, f"skew {angle}: got {found}"
 
+    # Levels are proposed too. A grey scan left at 0–255 ships its wash to the
+    # tablet, and the downscale to the song's width only blends it further.
+    grey = np.full((H, W), 210, np.uint8)  # paper, not white
+    grey[200:400, 100:900] = 70  # ink, not black
+    black, white = ink_levels(grey)
+    assert black < 70 < white < 210, (black, white)
+    # A page already bilevel has nothing between ink and paper to stretch.
+    assert ink_levels(_page()) == (0, 255)
+
+    # A scanner's edge line runs the whole height of the sheet. Being taller
+    # than every bracket it used to win each overlap and swallow the page into
+    # one system — Olukainen juomukainen, where five pages of six came out as a
+    # single slice each.
+    scanned = _page()
+    scanned[10 : H - 10, W - 8 : W - 4] = 0
+    assert len(detect_page(scanned).systems) == 2, "an edge artefact is not a bracket"
+
     # No brackets: every ink run is its own system.
     bare = np.full((H, W), 255, np.uint8)
     for y in (200, 500, 800):

tests/test_editor.py

@@ -19,6 +19,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 
 from PySide6.QtWidgets import QApplication  # noqa: E402
 
+from noteman_slicer import bundle  # noqa: E402
 from noteman_slicer.detect import detect_page  # noqa: E402
 from noteman_slicer.editor import Editor  # noqa: E402
 from noteman_slicer.pdf import open_source, page_raster  # noqa: E402
@@ -105,6 +106,28 @@ def main() -> int:
     assert reloaded.pages[0].levels == (40, 210)
     assert [c.points for c in reloaded.pages[0].cuts] == [c.points for c in page.cuts]
 
+    # The page fits the viewport once the window has a real size. show_page's
+    # own fit runs before layout, when the viewport is still its default.
+    editor.resize(900, 700)
+    editor.show()
+    app.processEvents()
+    scene = editor.view.sceneRect()
+    scale = editor.view.transform().m11()
+    viewport = editor.view.viewport()
+    fill = max(
+        scale * scene.width() / viewport.width(),
+        scale * scene.height() / viewport.height(),
+    )
+    # Fit means nearly touching one edge — Qt leaves a small margin of its own.
+    # A "≤ 1" check alone would pass a page zoomed down to a dot.
+    assert 0.9 <= fill <= 1.02, f"page is not fitted to the window: {fill:.3f}"
+
+    # The bundle is named after the song, not the PDF.
+    assert bundle.filename(project) == "Ketun-joululaulu.zip"
+    project.metadata["title"] = "AC/DC: T.N.T. (live)"
+    assert bundle.filename(project) == "ACDC-T.N.T.-live.zip"
+    project.metadata["title"] = "Ketun joululaulu"
+
     editor.close()
     source.close()
     for f in (pdf, saved):

tests/test_pdf.py

@@ -28,15 +28,21 @@ def _vector_pdf(path: Path, pages: int = 2) -> None:
     doc.save(path)
 
 
-def _scan_pdf(path: Path, pages: int = 2, w: int = 1653, h: int = 2332) -> None:
-    """Each page is one full-page grayscale image — what a real scan looks like."""
+def _scan_pdf(path: Path, pages: int = 2, w: int = 1653, h: int = 2332, rotation: int = 0) -> None:
+    """Each page is one full-page grayscale image — what a real scan looks like.
+
+    `rotation` reproduces a sheet fed sideways: the image is stored in its own
+    orientation and /Rotate turns it upright for a viewer.
+    """
     art = np.full((h, w), 255, np.uint8)
     art[500:505, 100 : w - 100] = 0  # a staff line, so it isn't uniform
+    art[:60, :60] = 0  # a corner mark, so orientation is checkable
     pix = pymupdf.Pixmap(pymupdf.csGRAY, w, h, bytearray(art.tobytes()), False)
     doc = pymupdf.open()
     for _ in range(pages):
-        page = doc.new_page(width=A4.width, height=A4.height)
+        page = doc.new_page(width=A4.width, height=A4.width * h / w)
         page.insert_image(page.rect, pixmap=pix)
+        page.set_rotation(rotation)
     doc.save(path)
 
 
@@ -64,6 +70,22 @@ def main() -> int:
     assert page.min() == 0 and page.max() == 255, (page.min(), page.max())
     src.close()
 
+    # The corner mark sits top-left in an upright scan.
+    assert page[:60, :60].max() == 0 and page[:60, -60:].min() == 255
+
+    # A sideways scan comes back upright: the page's /Rotate applies to the
+    # image extracted by xref, which bypasses it. Okular gets this right and
+    # the slicer used to not.
+    sideways = tmp / "sideways.pdf"
+    _scan_pdf(sideways, pages=1, w=2332, h=1653, rotation=90)
+    src = open_source(sideways)
+    assert src.type is SourceType.RASTER
+    turned = page_raster(src, 0)
+    assert turned.shape == (2332, 1653), turned.shape
+    # Turned clockwise, so the mark that was top-left is now top-right.
+    assert turned[:60, -60:].max() == 0 and turned[:60, :60].min() == 255
+    src.close()
+
     # An override must win over detection, and say so.
     src = open_source(scan, SourceType.VECTOR)
     assert src.type is SourceType.VECTOR and src.detected is SourceType.RASTER
@@ -71,7 +93,7 @@ def main() -> int:
     assert page_raster(src, 0).shape[1] > 4000, "override must force a render"
     src.close()
 
-    for f in (vec, scan):
+    for f in (vec, scan, sideways):
         f.unlink()
     tmp.rmdir()
     print("ok")

tests/test_render.py

@@ -19,6 +19,7 @@ from noteman_slicer.pdf import open_source, page_raster  # noqa: E402
 from noteman_slicer.project import Cut, Project, default_path  # noqa: E402
 from noteman_slicer.render import (  # noqa: E402
     ALPHA_LEVELS,
+    _ink_bbox,
     apply_levels,
     encode,
     pad_right,
@@ -87,6 +88,19 @@ def main() -> int:
     assert rgba[:, :, 3].min() == 0, "paper must be fully transparent"
     assert len(np.unique(rgba[:, :, 3])) <= ALPHA_LEVELS
 
+    # Trim keeps a line of lyrics and drops a fleck. Each letter is its own
+    # small blob, so judging blobs by area threw the whole line away and the
+    # bottom voice lost its words; a fleck's row carries almost no ink at all.
+    art = np.full((300, 800), 255, np.uint8)
+    art[100:150, 50:750] = 0  # a staff
+    for x in range(60, 700, 30):  # lyrics: many small glyphs, one row
+        art[200:220, x : x + 14] = 0
+    art[5:9, 10:14] = 0  # a fleck in the far corner
+    x0, y0, x1, y1 = _ink_bbox(art)
+    assert (y0, y1) == (100, 220), f"lyrics kept, fleck dropped: {(y0, y1)}"
+    assert (x0, x1) == (50, 750), (x0, x1)
+    assert _ink_bbox(np.full((50, 50), 255, np.uint8)) is None, "blank slice has no box"
+
     # Levels: a white point below the paper value wipes the paper out entirely.
     faint = np.full((10, 10), 200, np.uint8)
     assert apply_levels(faint, 0, 180).max() == 255