# photosift photosift turns a chaotic dump of phone backups, WhatsApp exports and camera cards into a clean, date-organized library. It finds exact and near-duplicate photos (resized, re-compressed, slightly edited) and never moves anything until you have reviewed a plan. Every apply is hash-verified, journaled and undoable. ## Features - Parallel recursive scanner with a SQLite cache keyed by `(path, size, mtime)`; re-scans only touch changed files. - Capture date from EXIF `DateTimeOriginal` (plus sub-seconds and offset), MP4/MOV `mvhd` atom (parsed by hand, no ffprobe), filename patterns (`IMG_20230105_101500`, `IMG-20230105-WA0001`, `Screenshot_2023-01-05-10-15-00`, `PXL_...`), then file mtime. The source used is recorded. - Camera make/model, GPS and orientation extraction. Images are orientation-normalized before hashing. - SHA-256 for exact duplicates (size and first-64KB pre-filter, full hash only on collisions) and aHash, dHash and pHash (32x32 DCT) implemented on Pillow and numpy. - Near-duplicate grouping with a BK-tree and union-find, a keeper policy with human-readable reasons, and burst detection that keeps burst frames apart. - `plan` writes a reviewable JSON of every placement with a reason; `apply` copies or moves with verification; `undo` reverts from the journal. - `stats`, `fix-dates` (writes EXIF with piexif) and `map` (Leaflet map of GPS clusters). - Safe by default: destructive commands are dry runs until `--execute`, nothing is ever unlinked without verification, and "delete" is a recoverable move to a trash folder. ## Install Python 3.12+. ```bash uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]' source .venv/bin/activate ``` ## Quickstart ```bash python examples/make_sample_library.py /tmp/sample # ~200 messy synthetic files photosift scan /tmp/sample photosift dupes --html dupes.html # open in a browser photosift plan --dest /tmp/Library --out plan.json photosift apply plan.json --move # dry run photosift apply plan.json --move --execute --journal run.journal.jsonl photosift undo run.journal.jsonl --execute # put everything back ``` The cache lives in `~/.cache/photosift/cache.db`; override with `--db PATH` or `PHOTOSIFT_DB`. In the transcripts below the sample library path is shortened to `/demo`. ## Usage ### scan ``` $ photosift scan /demo/lib found 204 files: 204 analyzed, 0 cached, 0 stale removed, 24 fully hashed (size+prefix collisions) $ photosift scan /demo/lib found 204 files: 0 analyzed, 204 cached, 0 stale removed, 0 fully hashed (size+prefix collisions) ``` `--workers N` sets the process pool size. Vanished files under the scanned roots are dropped from the cache. Corrupt images are reported as warnings and kept (hashed bytes, dated by filename or mtime, no perceptual hash). ### dupes `--near` (default) groups visually similar images, including byte-identical ones; `--exact` groups only identical files. `--threshold` is the maximum pHash Hamming distance (default 8). `--prefer` sets the path-substring ranking used as a keeper tie-breaker (default `DCIM`, `Camera`). `--include-bursts` treats burst frames as duplicates. ``` $ photosift dupes g0001 [exact] 2 files, 82.9 KB reclaimable KEEP /demo/lib/phone_backup/DCIM/Camera/IMG_20210519_214322.jpg (preferred path) dist 0 /demo/lib/camera_card/IMG_20210519_214322_copy.jpg (less preferred path) ... g0013 [near] 2 files, 18.7 KB reclaimable KEEP /demo/lib/phone_backup/DCIM/Camera/IMG_20210831_084639.jpg (highest resolution) dist 0 /demo/lib/downloads/Screenshot_2022-07-01-14-30-00.jpg (lower resolution (640x480 vs 1200x900)) ... 61 groups, 2.7 MB reclaimable, 1 bursts kept separate ``` `--html report.html` writes one self-contained page: base64 thumbnails side by side, the keeper outlined in green with a KEEP badge, reasons, pHash distances, and a separate section for bursts. ### plan ``` $ photosift plan --dest /demo/library --out plan.json plan: plan.json 143 place, 61 quarantine ``` Options: `--template`, `--dupes quarantine|skip|delete`, `--exact-only`, `--threshold`, `--prefer`, `--include-bursts`, `--under PATH`. The template is a Python format string. Fields: `year month day hour minute second date time camera seq ext stem kind source`. `{month:02d}` works because numbers are ints. The default is `{year}/{year}-{month:02d}/{date}_{time}_{camera}{seq}.{ext}`, which renders for example `2023/2023-08/2023-08-13_144031_Apple-iPhone-13.jpg`. Collisions get `-2`, `-3`, ... in `{seq}` (inserted before the extension if the template has no `{seq}`). A destination that already holds identical bytes becomes a `skip`; one that holds different bytes is never overwritten. Templates that escape the destination are rejected. Duplicate losers follow `--dupes`: `quarantine` places them under `/_duplicates//`, `skip` leaves them where they are, `delete` removes them from the source (as a move into `/.photosift-trash/`, only with `--move`). A plan entry: ```json { "src": "/demo/lib/whatsapp/Media/WhatsApp Images/IMG-20210313-WA1438.jpg", "action": "quarantine", "reason": "near duplicate of IMG_20210313_233140.jpg (g0041): lower resolution (800x600 vs 1200x900)", "dest": "/demo/library/_duplicates/g0041/IMG-20210313-WA1438.jpg", "sha256": "477038d3...", "size": 19481, "group": "g0041" } ``` ### apply and undo ``` $ photosift apply plan.json --move move /demo/lib/old_phone/future.jpg -> /demo/library/2028/2028-12/2028-12-24_192423_Apple-iPhone-13.jpg ... dry run: 204 done, 0 skipped, 0 failed re-run with --execute to apply $ photosift apply plan.json --move --execute --journal run.journal.jsonl ... applied: 204 done, 0 skipped, 0 failed $ photosift undo run.journal.jsonl # dry run dry run: 204 done, 0 skipped, 0 failed $ photosift undo run.journal.jsonl --execute undone: 204 done, 0 skipped, 0 failed ``` `--copy` (default) keeps sources and ignores `delete` entries; `--move` removes each source only after its copy is verified. Exit status is 1 if any file failed. See [docs/safety-model.md](docs/safety-model.md). ### stats ``` $ photosift stats 204 files, 13.2 MB (201 image, 3 video) By year 1904 1 1970 1 2021 57 ... By camera unknown 46 Google Pixel 7 42 ... By date source exif 158 filename 42 mp4 3 mtime 1 Duplicates exact: 13 groups, 952.2 KB reclaimable near: 61 groups, 2.7 MB reclaimable (includes exact) Date problems (4 files) /demo/lib/old_phone/epoch.jpg - missing: no EXIF/container or filename date, using file mtime - suspicious: 1970 is before 1990 /demo/lib/old_phone/future.jpg - suspicious: date is in the future /demo/lib/old_phone/IMG_20180101_100000.jpg - suspicious: EXIF/container and filename dates differ by 1251 days ``` ### fix-dates ``` $ photosift fix-dates --from-filename would write 2023:06:15 12:00:00 /demo/lib/whatsapp/Media/WhatsApp Images/IMG-20230615-WA2236.jpg ... 42 files need updating (dry run, add --execute) ``` Only JPEGs whose date came from the filename are touched. Date-only names (WhatsApp) get 12:00:00. Each write goes to a temp copy, is read back, then swapped in; the file's mtime is preserved. ### map ``` $ photosift map --html map.html 101 geotagged photos in 3 clusters -> map.html ``` Points are bucketed into a grid (`--radius-km`, default 5) and touching cells merge into one cluster. The page loads Leaflet from cdnjs and tiles from OpenStreetMap, so it needs a network connection to display. ## Configuration | Setting | Where | Default | | --- | --- | --- | | Cache database | `--db`, `PHOTOSIFT_DB` | `~/.cache/photosift/cache.db` | | pHash threshold | `--threshold` | 8 | | Keeper path priority | `--prefer` (repeatable) | `DCIM`, `Camera` | | Destination template | `plan --template` | see above | Commands that read the cache accept `--under PATH` to limit them to one directory. ## Architecture | Module | Role | | --- | --- | | `cli.py` | click commands, thin glue | | `scanner.py` | walk, process pool, cache reuse, collision-only full hashing | | `metadata.py` | per-file analysis (EXIF, GPS, dates, hashes), runs in workers | | `dates.py`, `mp4.py` | filename/EXIF date parsing, `mvhd` atom parsing | | `hashing.py` | SHA-256, aHash, dHash, pHash, Hamming distance | | `bktree.py` | BK-tree radius search | | `grouping.py` | union-find grouping, keeper policy, burst detection | | `planner.py` | template rendering, collision handling, plan JSON | | `applier.py` | verified copy/move, journal, undo | | `stats.py`, `fixdates.py`, `mapper.py`, `report.py` | the matching commands | | `db.py`, `models.py` | SQLite cache and the `FileInfo` record | Deeper docs: [perceptual hashing](docs/perceptual-hashing.md) and [safety model](docs/safety-model.md). ## Tests ```bash .venv/bin/pytest ``` 72 tests cover date priority and patterns, MP4 parsing, hash invariance, BK-tree against brute force, keeper policy, bursts, template rendering and collisions, apply/undo round trips with hash verification, tamper and overwrite refusal, stats, fix-dates and the CLI end to end. ## Limitations - HEIC/HEIF files are indexed and hashed byte-wise but not decoded (Pillow has no HEIC support by default), so they get no perceptual hash. - Videos are only matched when byte-identical; duration and resolution are not read. - MP4 `mvhd` times are UTC; EXIF times are camera-local. They are not reconciled. - Near-duplicate groups are connected components: a chain A~B~C joins A and C even if they are far apart. Lower `--threshold` if groups look too big. Strong crops or edits can exceed the default threshold of 8. - The keeper is chosen by resolution first, so an upscaled re-export can beat the original. Review the report, or adjust the plan JSON, before applying. - Burst frames from a camera that records no sub-second time and stored in the same second are indistinguishable from copies. - Map clustering uses a plain lat/lon grid, so cells are narrower in longitude at high latitudes.