Soft82.com » Windows » Design & Photo » Image Editing » ImageVerifier 1.3.01 Beta

ImageVerifier Description

ImageVerifier (IV for short) traverses a hierarchy of folders looking for image files to verify

It can verify TIFFs, JPEGs. PSDs, DNGs, and non-DNG raws (e.g., NEF, CR2).

IV is designed to process large numbers of images. Folder hierarchies with 100,000 images or more should be no problem. In one test run, IV ran for 14 hours.

There are two kinds of verification that IV performs: Structure checking and hash checking. They are described in detail below.

Basic Operation
All structure verification other than for non-DNG raws is built-in; for DNGs IV uses Adobe's DNG SDK directly. JPEGs and TIFFs are verified using built-in libraries as well. PSDs are validated using a method designed specifically for ImageVerifier that follows Adobe documentation for the PSD format. Non-DNG raws (e.g., NEFs) are verified by running them through Adobe DNG Converter.

For all image files, structure checking is performed by reading the actual image data, decompressing as necessary. This can find many errors, but not all, as some errors are indistinguishable for image data. See below for more information.

The list of extensions for raw files is the same as ImageIngester's. (If your raw files end in TIF you can set an option for that on the main window.)

The real work is done by subprocesses, so IV can take advantage of multiple CPU (or multiple core) computers. If you have 4 CPUs, it should be capable of fully loading all 4 at once.

For each verification run, called a job, you can choose the folders, whether to process subfolders or just the top level, what kinds of images to process (TIFF, JPEG, PSD, DNG, and/or non-DNG raw), the maximum number of errors to report, and whether to store the results in a built-in database.

You can save the settings in a named job, which acts something like ImageIngester's presets, except saving is automatic.

There's a built-in scheduler on OS X (not on Windows) that allows you to schedule jobs to be run once at a specified time; daily at a specified time on specified days (e.g., Tuesdays and Saturdays at 2am); and monthly on a specified day and time (e.g., the 3rd of every month at 5am).

The scheduler uses the "cron" facility built into OS X. IV doesn't have to be running for a scheduled job to run, nor does it keep its own daemon process running.
More About Detection of Invalid Files With Structure Checking

On one test of 100 defective images produced by "punching" holes of 40,000 zero bytes at random points, ImageVerifier reported 88% of them as invalid, which is exactly the same result produced by DNG Converter running standalone.

For DNGs, IV uses Adobe's DNG SDK (software development kit). It produced exactly the same results on a test of 100 defective DNGs as did DNG Converter.

Detection of invalid JPEGs was even better (98% or something like that). The very high compression of JPEGs makes even small errors detectable.

Detection of invalid TIFFs with an 800,000 byte hole punched in them was poor: Only 13%. However, 6MB TIFFs with 100,000, 200,000, 400,000, and 800,000 bytes lost from the end (i.e., truncated files) were all reported as invalid. Perhaps truncated files and files with their headers clobbered are the most commonly found bad files, and those are all detected.

(For some additional insight into the problem of detecting invalid files, imagine a file format that consists of a 4-byte width, a 4-byte length, and then width-times-height 3-byte RGB pixels. In this arrangement, as long as the width and length accurately describe the file, since any RGB combination is valid, no holes punched into the file would be detectable unless they affect the first 8 bytes. That's the worst case, and it's close to the TIFF case if no compression is used.)
Hash Checking
Structure checking is verifying the image file by reading through its various structures and decompressing any compressed image data, looking for errors. This can be effective in finding damage if the damage is large and/or the image is compressed. For highly compressed images like JPEGs, damage detection is very good. It's not so good for uncompressed raws, such as the DNGs that come straight from a Leica M8. It's better for compressed DNGs, but not as good as it is for JPEGs.

Another approach entirely is hash checking, which is maintaining for each image known to be good a fixed-length hash computed from all the bytes in the file so that it's unlikely that two different files will produce the same hash. (Not impossible, since the hash is of fixed length and the number of possible image files is infinite.) If the two files are the good one and a copy (or even the original) that's been damaged, then comparing hashes of the two files will show that the files are not the same.

Comparing the actual files is even better, but in the case of a single file that's been damaged you don't have two files. All you have is the damaged file and the hash from when it used to be good. Also, reading one file to compute its hash takes half as long as reading two files.

The nice thing about structure checking is that no bookkeeping is involved—each file stands on its own. Hash checking, however, does create complications because you need to put the hash somewhere, and you need a way of associating the image with its hash. This is easy for a DAM system that controls all the assets, but much harder with a passive utility like ImageVerifier. Putting the hash inside the file is one approach, but this has two problems: It's safe only for certain formats for which it's allowed, such as DNG, and it requires IV to write into the file, which I don't want to do because it raises the possibility of damage to the file during verification and because many photographers don't want to use any utilities that write into their files.

So, here's the scheme that IV uses: For each file, a key is generated that's rich enough so that two different images won't have the same key. The key is the concatenation of the filename (not the path, just the last component), the size, the modification date/time of the file, the EXIF DateTimeDigitized, the EXIF SubSecTimeDigitized, and the EXIF DateTimeOriginal (also called plain DateTime). It's still possible for two different images to have identical keys, but the worse that will happen in that case is that IV will erroneously say that they are different, and then later you can determine that they are not.

After the key is generated, a 512-bit hash is computed from the entire file, and both are stored in the IV database. Note that the location of the file--what folder it's in--is stored for reference purposes, but plays no role at all in associating keys (and therefore files) and their hashes.

Here's an example key:

DSC_0003.JPG - 2591469 - 2007-02-05 22:56:43 - 2004:06:23 18:59:27.70 - 2007:02:03 14:00:20

The corresponding example hash is a 128-character string of hex digits.

If the file is copied to a backup, or moved to another folder, IV will still find its hash, as long as none of the components of the key have changed. (Copying normally doesn't change the modification date, and you need to be careful not to rename your files when you back them up. Once a file is named, ideally during ingestion, it ought to keep that name forever.)

A key and hash take up around 150 bytes in the database, depending on the length of the key. (Recall that the hash represents all the bytes in the file, which for most of us is around 10MB or more per image.) In addition, there's space needed for the parent folder of each image, but these are shared by all the images in that folder, so if you have, say, 50 images in a folder, storing the parent path adds only around 1 or 2 bytes per image. Doing the math, the space to store the keys and hashes for 200,000 images is only about 30MB, which is about the space of 3 images. A lot of overhead space is also need for indexing, so the 30MB maybe should be tripled. Still, the space cost is reasonable.

When you run IV, you have the option of doing a hash check, a structure check, or both. To run a hash check, you must have previously generated the hashes for the files you want to check. If during a hash check no stored hash can be found for an image (that is, no match for the key), you get a failure message that's different from the message you get when the hashes fail to match.

The hash check is pretty fast: Less than a half-second per image, compared to 2 - 4 seconds for a structure check.

There's a Keep Existing Hashes option that restricts verification and storing of hashes to those files that don't already have a hash stored. This is a quick way of processing just files that were added (or which failed verification) since the last time hashes were stored on that folder hierarchy.

A Store Hashes for Invalid TIFFs option causes a hash to be stored even if a TIFF file failed the validity check, since many TIFFs deviate from the standard and therefore can't be checked by ImageVerifier. Don't use this option unless you know the images to be good. (Hashes are also stored for valid TIFFs and for any other types you're also processing.)

In practice, the workflow is something like this: You take a folder of images known to be good and run IV on them with the Store Hashes checkbox checked. This automatically checks the Check Structure checkbox, because a hash is stored in the database only if the file passed the structure check. Then, after the hashes are stored, you can run a hash check on the same files now, or a year from now, or 5 years from now.

Here's a simple test I ran: I stored hashes from a folder of JPEGs, and then took one of them, copied it to another folder, and wrote a single "x" byte into it, 123 bytes from the end, being careful to reset the modification time back to its original value. The resulting damaged JPEG passed the structure check, but it failed the hash check.

In practice, IV seems to magically just remember all your images and what their bytes are supposed to be, and then dips into its memory whenever it sees a repeat image. All this works no matter what folder structure you use, how many backups you have, or how often you rearrange things. (Just don't change the file names.)

A Manage Hashes menu choice opens a panel where you can see the folders whose images were hashed, along with a list of the keys for those images. You can purge a single folder, although you don't have to, since storing new hashes with the same keys will overwrite the old hashes.

ImageVerifier Download Button For Your Site

If you want to place a Download Button for ImageVerifier on your website just insert the following code:

If you want a Link to this page you can use one of the logos bellow:

ImageVerifier User Reviews

New Image Editing Software

Crowd3 May 25, 2012 Create images of crowd scenes using the people in your photo collection				Image Editor and Converter May 25, 2012 Create, edit, and compose digital images
Magic Pixel May 24, 2012 Get the color code of the Pixel where the cursor is at				Snapseed May 23, 2012 Make any photo extraordinary
Ticno Screenshoter May 23, 2012 Make a picture of your screen, windows or custom area and save it to your hard drive				Vectorder May 22, 2012 Arrange Lock Unlock made easy and simple
Inspyder Photo Lab May 22, 2012 The Inspyder Photo Lab enables you to modify images in bulk				LineTracer May 22, 2012 A graphical application for vectorizing and tidying up scanned line-art sketches
Xara Photo & Graphic Designer MX May 21, 2012 Great value graphics software for all your drawing, photo and page design needs				Raylectron May 21, 2012 A photorealistic rendering engine for Google Sketchup
Image Watermarks May 21, 2012 Program for batch image watermarks creation				PicPick May 21, 2012 An all-in-one software for software developers, graphic designers and home user
Beauty Pilot May 18, 2012 Bring out the beauty in women's portraits taken with a digital camera				GIMP Portable May 18, 2012 The popular the GIMP for Windows image editor packaged as a portable app
Light Image Resizer May 17, 2012 Image resizer to resize and convert images				PicPick Portable May 17, 2012 An all-in-one software for software developers, graphic designers and home user
AKVIS ArtSuite May 17, 2012 An impressing collection of versatile effects for decoration of photos				AKVIS Sketch May 17, 2012 AKVIS Sketch - Conversion of photos to pencil and watercolor drawings, B&W and color sketches
Smart Photo Import May 17, 2012 Sort and organize all your images				HyperSnap DX May 16, 2012 HyperSnap-DX is a powerful screen capture

Last 7 Days Most Downloaded Image Editing Products

Adobe Photoshop 12,206 downloads Adobe Photoshop creates powerful images with the professional standard				Macromedia Fireworks 1,269 downloads Macromedia created Fireworks as a complete graphics applications aimed towards Web...
PhotoShine 1,190 downloads Make photo collage by adding your photo into different beautiful templates				Photoscape 736 downloads Photo editing software that enables you to fix and enhance photos
Adobe Illustrator CS6 509 downloads Create distinctive vector artwork for any project with Adobe Illustrator CS5 graphic design and				ArcSoft PhotoStudio 421 downloads ArcSoft PhotoStudio is a powerful photo editing application featuring an array of advanced tools
Ulead PhotoImpact 403 downloads Ulead PhotoImpact ia a complete image editing suite				Adobe Photoshop Lightroom 388 downloads Adobe Photoshop Lightroom is the efficient new way for professional photographers
QuarkXPress 364 downloads Creative design and page layout				Sweet Home 3D 160 downloads Sweet Home 3D is a free interior design application

All Time Most Downloaded Image Editing Products

Adobe Photoshop 5,744,051 downloads Adobe Photoshop creates powerful images with the professional standard				Macromedia Fireworks 909,109 downloads Macromedia created Fireworks as a complete graphics applications aimed towards Web...
PhotoShine 129,319 downloads Make photo collage by adding your photo into different beautiful templates				Photoscape 235,057 downloads Photo editing software that enables you to fix and enhance photos
Adobe Illustrator CS6 151,071 downloads Create distinctive vector artwork for any project with Adobe Illustrator CS5 graphic design and				ArcSoft PhotoStudio 168,699 downloads ArcSoft PhotoStudio is a powerful photo editing application featuring an array of advanced tools
Ulead PhotoImpact 294,720 downloads Ulead PhotoImpact ia a complete image editing suite				Adobe Photoshop Lightroom 79,587 downloads Adobe Photoshop Lightroom is the efficient new way for professional photographers
QuarkXPress 32,714 downloads Creative design and page layout				Sweet Home 3D 40,996 downloads Sweet Home 3D is a free interior design application