photosift

photosift turns a chaotic dump of phone backups, WhatsApp exports and camera cards into a clean, date-organized library. It finds exact and near-duplicate photos (resized, re-compressed, slightly edited) and never moves anything until you have reviewed a plan. Every apply is hash-verified, journaled and undoable.

Features

• Parallel recursive scanner with a SQLite cache keyed by (path, size, mtime); re-scans only touch changed files.
• Capture date from EXIF DateTimeOriginal (plus sub-seconds and offset), MP4/MOV mvhd atom (parsed by hand, no ffprobe), filename patterns (IMG_20230105_101500, IMG-20230105-WA0001, Screenshot_2023-01-05-10-15-00, PXL_...), then file mtime. The source used is recorded.
• Camera make/model, GPS and orientation extraction. Images are orientation-normalized before hashing.
• SHA-256 for exact duplicates (size and first-64KB pre-filter, full hash only on collisions) and aHash, dHash and pHash (32x32 DCT) implemented on Pillow and numpy.
• Near-duplicate grouping with a BK-tree and union-find, a keeper policy with human-readable reasons, and burst detection that keeps burst frames apart.
• plan writes a reviewable JSON of every placement with a reason; apply copies or moves with verification; undo reverts from the journal.
• stats, fix-dates (writes EXIF with piexif) and map (Leaflet map of GPS clusters).
• Safe by default: destructive commands are dry runs until --execute, nothing is ever unlinked without verification, and "delete" is a recoverable move to a trash folder.

Install

Python 3.12+.

uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]'
source .venv/bin/activate

Quickstart

python examples/make_sample_library.py /tmp/sample          # ~200 messy synthetic files
photosift scan /tmp/sample
photosift dupes --html dupes.html                           # open in a browser
photosift plan --dest /tmp/Library --out plan.json
photosift apply plan.json --move                            # dry run
photosift apply plan.json --move --execute --journal run.journal.jsonl
photosift undo run.journal.jsonl --execute                  # put everything back

The cache lives in ~/.cache/photosift/cache.db; override with --db PATH or PHOTOSIFT_DB. In the transcripts below the sample library path is shortened to /demo.

Usage

scan

$ photosift scan /demo/lib
found 204 files: 204 analyzed, 0 cached, 0 stale removed, 24 fully hashed (size+prefix collisions)
$ photosift scan /demo/lib
found 204 files: 0 analyzed, 204 cached, 0 stale removed, 0 fully hashed (size+prefix collisions)

--workers N sets the process pool size. Vanished files under the scanned roots are dropped from the cache. Corrupt images are reported as warnings and kept (hashed bytes, dated by filename or mtime, no perceptual hash).

dupes

--near (default) groups visually similar images, including byte-identical ones; --exact groups only identical files. --threshold is the maximum pHash Hamming distance (default 8). --prefer sets the path-substring ranking used as a keeper tie-breaker (default DCIM, Camera). --include-bursts treats burst frames as duplicates.

$ photosift dupes
g0001 [exact] 2 files, 82.9 KB reclaimable
  KEEP    /demo/lib/phone_backup/DCIM/Camera/IMG_20210519_214322.jpg  (preferred path)
  dist 0  /demo/lib/camera_card/IMG_20210519_214322_copy.jpg  (less preferred path)
...
g0013 [near] 2 files, 18.7 KB reclaimable
  KEEP    /demo/lib/phone_backup/DCIM/Camera/IMG_20210831_084639.jpg  (highest resolution)
  dist 0  /demo/lib/downloads/Screenshot_2022-07-01-14-30-00.jpg  (lower resolution (640x480 vs 1200x900))
...
61 groups, 2.7 MB reclaimable, 1 bursts kept separate

--html report.html writes one self-contained page: base64 thumbnails side by side, the keeper outlined in green with a KEEP badge, reasons, pHash distances, and a separate section for bursts.

plan

$ photosift plan --dest /demo/library --out plan.json
plan: plan.json 143 place, 61 quarantine

Options: --template, --dupes quarantine|skip|delete, --exact-only, --threshold, --prefer, --include-bursts, --under PATH.

The template is a Python format string. Fields: year month day hour minute second date time camera seq ext stem kind source. {month:02d} works because numbers are ints. The default is {year}/{year}-{month:02d}/{date}_{time}_{camera}{seq}.{ext}, which renders for example 2023/2023-08/2023-08-13_144031_Apple-iPhone-13.jpg. Collisions get -2, -3, ... in {seq} (inserted before the extension if the template has no {seq}). A destination that already holds identical bytes becomes a skip; one that holds different bytes is never overwritten. Templates that escape the destination are rejected.

Duplicate losers follow --dupes: quarantine places them under <dest>/_duplicates/<group>/, skip leaves them where they are, delete removes them from the source (as a move into <dest>/.photosift-trash/, only with --move).

A plan entry:

{
 "src": "/demo/lib/whatsapp/Media/WhatsApp Images/IMG-20210313-WA1438.jpg",
 "action": "quarantine",
 "reason": "near duplicate of IMG_20210313_233140.jpg (g0041): lower resolution (800x600 vs 1200x900)",
 "dest": "/demo/library/_duplicates/g0041/IMG-20210313-WA1438.jpg",
 "sha256": "477038d3...",
 "size": 19481,
 "group": "g0041"
}

apply and undo

$ photosift apply plan.json --move
move   /demo/lib/old_phone/future.jpg -> /demo/library/2028/2028-12/2028-12-24_192423_Apple-iPhone-13.jpg
...
dry run: 204 done, 0 skipped, 0 failed
re-run with --execute to apply
$ photosift apply plan.json --move --execute --journal run.journal.jsonl
...
applied: 204 done, 0 skipped, 0 failed
$ photosift undo run.journal.jsonl            # dry run
dry run: 204 done, 0 skipped, 0 failed
$ photosift undo run.journal.jsonl --execute
undone: 204 done, 0 skipped, 0 failed

--copy (default) keeps sources and ignores delete entries; --move removes each source only after its copy is verified. Exit status is 1 if any file failed. See docs/safety-model.md.

stats

$ photosift stats
204 files, 13.2 MB  (201 image, 3 video)

By year
  1904                         1
  1970                         1
  2021                         57
  ...
By camera
  unknown                      46
  Google Pixel 7               42
  ...
By date source
  exif                         158
  filename                     42
  mp4                          3
  mtime                        1

Duplicates
  exact: 13 groups, 952.2 KB reclaimable
  near:  61 groups, 2.7 MB reclaimable (includes exact)

Date problems (4 files)
  /demo/lib/old_phone/epoch.jpg
    - missing: no EXIF/container or filename date, using file mtime
    - suspicious: 1970 is before 1990
  /demo/lib/old_phone/future.jpg
    - suspicious: date is in the future
  /demo/lib/old_phone/IMG_20180101_100000.jpg
    - suspicious: EXIF/container and filename dates differ by 1251 days

fix-dates

$ photosift fix-dates --from-filename
would write 2023:06:15 12:00:00  /demo/lib/whatsapp/Media/WhatsApp Images/IMG-20230615-WA2236.jpg
...
42 files need updating (dry run, add --execute)

Only JPEGs whose date came from the filename are touched. Date-only names (WhatsApp) get 12:00:00. Each write goes to a temp copy, is read back, then swapped in; the file's mtime is preserved.

map

$ photosift map --html map.html
101 geotagged photos in 3 clusters -> map.html

Points are bucketed into a grid (--radius-km, default 5) and touching cells merge into one cluster. The page loads Leaflet from cdnjs and tiles from OpenStreetMap, so it needs a network connection to display.

Configuration

Commands that read the cache accept --under PATH to limit them to one directory.

Architecture

Deeper docs: perceptual hashing and safety model.

Tests

.venv/bin/pytest

72 tests cover date priority and patterns, MP4 parsing, hash invariance, BK-tree against brute force, keeper policy, bursts, template rendering and collisions, apply/undo round trips with hash verification, tamper and overwrite refusal, stats, fix-dates and the CLI end to end.

Limitations

• HEIC/HEIF files are indexed and hashed byte-wise but not decoded (Pillow has no HEIC support by default), so they get no perceptual hash.
• Videos are only matched when byte-identical; duration and resolution are not read.
• MP4 mvhd times are UTC; EXIF times are camera-local. They are not reconciled.
• Near-duplicate groups are connected components: a chain A~B~C joins A and C even if they are far apart. Lower --threshold if groups look too big. Strong crops or edits can exceed the default threshold of 8.
• The keeper is chosen by resolution first, so an upscaled re-export can beat the original. Review the report, or adjust the plan JSON, before applying.
• Burst frames from a camera that records no sub-second time and stored in the same second are indistinguishable from copies.
• Map clustering uses a plain lat/lon grid, so cells are narrower in longitude at high latitudes.