Find duplicate images in Amazon S3.
Twinsift scans the images in your S3 buckets, groups every copy, rendition, crop and edit with its original, and shows how much storage they take.
- 99.9%of duplicates found
- 96%+of crops traced to their source
- 0files deleted without your approval
Accuracy measured on a test library of 250,995 files.
Why S3 can't find them on its own
S3 stores whatever it is given. Image pipelines, CMS exports and backups write the same photo again under new keys, in new sizes and formats, and nothing in S3 connects them.
Comparing ETags only catches byte-identical uploads, and not even those when a file was sent with multipart upload. A JPEG made from a TIFF, or a 1080 px web version, looks like an unrelated object.
What Twinsift finds
- Duplicates: the same picture saved again in another format, size or quality.
- Renditions: smaller web and thumbnail versions.
- Crops and banners: parts of the photo, found even after colour grading.
- Edits: black and white versions and versions with added text.
Each file is grouped with its original. The original is chosen by what is in the pixels: true resolution first, then format, bit depth, compression and sharpness.
How it connects
You create an IAM role that Twinsift assumes with an external ID, limited to the buckets and prefixes you choose. The scan needs these read permissions:
s3:ListBucketands3:ListBucketVersionss3:GetObjectands3:GetObjectVersions3:GetBucketVersioning
Objects in Glacier and Glacier Deep Archive are skipped and listed in the report, so the scan never triggers a restore.
Deletes you can undo
Nothing changes until you approve a cleanup plan, and only duplicates and renditions can be deleted, never the original. Deletes need s3:DeleteObject and are refused on a bucket without versioning. On a versioned bucket a delete adds a delete marker, and the earlier version stays restorable.
Does your DAM keep its files in S3? Read about duplicate assets in a DAM.
Questions
Does Twinsift change our objects or bucket settings?
No. The scan only reads. Twinsift adds no tags or metadata, and deletes happen only after you approve a plan.
Can we limit it to one prefix?
Yes. You choose the buckets and prefixes, and the IAM role allows only those.
Does the scan add to our AWS bill?
Reading objects out of S3 is billed by AWS as data transfer, at your usual rates. We tell you the expected amount before the scan starts.
See how many copies your buckets hold.
The free scan counts the duplicates, renditions, crops, banners and edits in your library and how much space they take. Read-only access, and nothing in your library changes.
Request a free scan