February 3, 2005
Digital Image Management
Over the years, I have graduated from a 2 MP Olympus D490Z digital Point and Shoot camera to a 4 MP Canon S410 Point and Shoot and then on to a full fledged 8 MP Canon EOS 20D. Accompanying this gradual rising in the capabilities of the camera has been the explosion of images that I have had to organize, save, backup, view etc. At last count I had over 7000 images in my hard disk all neatly filed away under 'My Pictures' folder. And these are only my casual photos, taken over sun filled vacations, of cuddly cute kids and smiling (an not so smiling) relatives and friends. I have another 800 more from my serious photography collection taken over the course of 1 year. I shudder to think of how much it will increase over the next few years. So all this means I really need a solid, easy to use image management system.
Here are some issues at stake.
1) People are lazy
Face this reality, for any scheme to work, it has to be painless and easy to follow. C’mon, we aren't trying to get ISO 9000 certification here. So there is no way someone will take the time to neatly categorize each image, add keywords etc. Even if one does take the path of going through the pain for a few months, trust me, it will soon be abandoned. I used to use IMatch for a while but then gave up keeping it up. Such systems are error prone, incredibly boring and produce low low returns. Nothing to motivate someone who would be elsewhere doing something else.
2) Must be universally portable
This is where most cataloging software such as Picasa, IMatch, ACDSee and any others you can think of break down. I do not doubt their intentions nor their interface nor the way they work. They all do that very well. But they are not portable. When I talk about portable, I mean that they won't survive a move from the venerable Windows XP or the Microsoftian stable to Linux, MacOS or BeOS if required. Unless there are native implementations of these programs on other OS'es all the hard work put into these programs evaporate. Any solution that you has to be portable, now and in the future. Also, the million dollar question is how many programs will survive 10 years from now? Well, people can claim that you can still make it portable by embedding the cataloging information etc into the image itself, by using IPTC, EXIF, XMP or whatever. But my point is that, this is not anyway easy to do.
3) Must be Backup friendly
You don't do backups? Well, my friend, your days are numbered. The Hard disk crash is just a few heartbeats away. Anyway, to those who do backup, the method must be easy to maintain and should not add complexity. Mainly it must be easy to figure out which sections are backed up to a external media and which ones are not. It must also allow incremental backups.
4) Must be easy to use
Face it, there is no point if you have a top of the line image management system if it intimidates you, your wife/husband or whoever from using it.
5) Images must be easy to find
This is important, but you do need a way to find out which image is in which folder easily and quickly without searching through thousands of files.
So given these 5 criteria, without any further ado, let me introduce you to my image organization system. Note that this applies only to my serious photography stuff. I don't follow the concept for my casual photos, mostly because images are small and I hardly ever look at them frequently (who can and who does?)
So in my system I have a folders which are organized as follows:
Photography
+-------------->Archives
+-------------->Processed
+-------------->Misc
+-------------->Prints
+-------------->To File
Taking each folder:
To File: Everything I shoot gets downloaded in here. This is the clearing house before images enter my digital warehouse. Sometimes it takes weeks before I get to clearing stuff away. But this is better than downloading images to some arbitrary folder and then forgetting about it.
Prints: Folder to store processed images that I want to send for printing. These are usually finished TIFF files usually weighing in about 30 MB each.
Misc: This is a catch all folder, containing various odds and ends. I store lots of Photoshop types docs. Nothing that important.
Archives: This is the primary folder. This is organized as follows:
+-->Archives
+-------> 2002
+-----------> contacts
+-----------> 01-Jan 14, 2002
+------------------> 2002_01_14_45.jpg
+------------------> raw
+----------------------> 2002_01_14_1.cr2
+----------------------> 2002_01_14_2.cr2
+------------------> dng
+----------------------> 2002_01_14_1.dng
+----------------------> 2002_01_14_2.dng
+-----------> 09-Sep 01, 2002
+... More of the same
+-------> 2003
Key points of the structure above, every year(2002, 2003, 2004) is a top level folder under Archives. Each year has a subfolder for each day I actually downloaded images into the PC. The folder names are created using the following format '01-Jan 01, 2002' so that they appear in Windows Explorer etc properly sorted and in chronological order.
Every download date folder (e.g. '01-Jan 01, 2002') may contain up to 3 different types of images. If the images are only .jpg or .tiff they get dumped directly into the folder (see '2002_01_14_45.jpg'). If the images are other formats, like Canon EOS 20D's CR2 RAW format they are dumped into the 'raw' subdirectory. After images are dumped, they are all renamed to standard convention of YEAR_MONTH_DAY_IMGNUMBER.EXTENSION (see 2002_01_14_1.cr2) . One free renamer is 1-4a-rename. (Highly recommended) Also, since I don't believe in the longevity of the CR2 image formats, I hedge my bets by creating .DNG files under sub folders. DNG files are in the DIGITAL NEGATIVE format introduced by Adobe. This is meant to replace all the thousands of RAW image formats used by each different camera manufacturer. (for eg, .CR2, .CRW, .NEF) Note all images are only originals.
There is also a contacts folder that contains contact sheet .jpgs for all originals in that year. [A contact sheet is nothing but a image that contains thumbnails of other images.] You can break it into multiple parts if you have too many images. But the idea is to have a few jpgs that have small thumbnails of the entire year. [ You can create contact sheets many ways, the most easy way is by using Adobe Photoshop] Here is a sample contact sheet:
Processed: This is the secondary folder. This houses all the images processed.
+--Processed
+-------> 2002
+-----------> 01-Jan 14, 2002
+----------------------> 2002_01_14_2_color.psd
+----------------------> 2002_01_14_2_bw.psd
+-----------> 09-Sep 01, 2002
...
+-------> 2003
The reason there is a processed top level folder is that most images from a digital camera are unfit for consumption directly, especially if you are shooting RAW format. They need to be tweaked, modified or Photoshopped before they are deemed acceptable. So how do we deal with this? We can take any file from the archives, do the tweaks and save the results in the day we did the tweak under the Processed directory. The folder structure is very similar to the Archive section. *IMPORTANT* the files are saved under the same name as the file they originally originated from. Trailing information can be appended to add some more data about what the change was. (Eg, _color, _bw, _art etc)
Ok, now that you know about the system, let me show you how it solves the issues highlighted in the beginning of the article.
1) People are lazy
Yes, this is meant for really lazy people. All you need to do is save your images to the right folder and you are done. You might need to run the DNG converter and the batch renamer but these take a few minutes.
2) Must be universally portable
No proprietary databases etc. Everything is in the picture, folder and name itself. All this should be very portable. If they cannot port a binary folder/file all hope is lost. Well there is no cataloging you say. Well cataloging is for the dogs, I say. All the time spent cataloging is much better spent learning a new nifty trick in Photoshop or capturing more nifty images.
3) Must be backup friendly
Very easy to do incremental backups. Lets say you have 2 years under archive. 2002 & 2003. If you added the contact sheet for those years and saved them. You are ready to back them up to any media, even to external media. As long as you have the contact sheets, you will know which CD has the pics you want. Also this allows incremental backup as you will only adding new folders to the Archive/Processed folder, never go back to an old folder.
4) Must be easy to use
No complicated app to learn. Just browse your Hard drive.
5) Images must be easy to find
All images are named using the same syntax. So every image is unique. It is possible to figure out what the relationship is between images etc just by looking at their names. It is also easy to locate specific images by browsing the contact sheets.
That’s it, you have my low cost digital image management system. I hope you found it useful. Drop me a line if you have better solutions than these. I would love to hear from you.
Here are some issues at stake.
1) People are lazy
Face this reality, for any scheme to work, it has to be painless and easy to follow. C’mon, we aren't trying to get ISO 9000 certification here. So there is no way someone will take the time to neatly categorize each image, add keywords etc. Even if one does take the path of going through the pain for a few months, trust me, it will soon be abandoned. I used to use IMatch for a while but then gave up keeping it up. Such systems are error prone, incredibly boring and produce low low returns. Nothing to motivate someone who would be elsewhere doing something else.
2) Must be universally portable
This is where most cataloging software such as Picasa, IMatch, ACDSee and any others you can think of break down. I do not doubt their intentions nor their interface nor the way they work. They all do that very well. But they are not portable. When I talk about portable, I mean that they won't survive a move from the venerable Windows XP or the Microsoftian stable to Linux, MacOS or BeOS if required. Unless there are native implementations of these programs on other OS'es all the hard work put into these programs evaporate. Any solution that you has to be portable, now and in the future. Also, the million dollar question is how many programs will survive 10 years from now? Well, people can claim that you can still make it portable by embedding the cataloging information etc into the image itself, by using IPTC, EXIF, XMP or whatever. But my point is that, this is not anyway easy to do.
3) Must be Backup friendly
You don't do backups? Well, my friend, your days are numbered. The Hard disk crash is just a few heartbeats away. Anyway, to those who do backup, the method must be easy to maintain and should not add complexity. Mainly it must be easy to figure out which sections are backed up to a external media and which ones are not. It must also allow incremental backups.
4) Must be easy to use
Face it, there is no point if you have a top of the line image management system if it intimidates you, your wife/husband or whoever from using it.
5) Images must be easy to find
This is important, but you do need a way to find out which image is in which folder easily and quickly without searching through thousands of files.
So given these 5 criteria, without any further ado, let me introduce you to my image organization system. Note that this applies only to my serious photography stuff. I don't follow the concept for my casual photos, mostly because images are small and I hardly ever look at them frequently (who can and who does?)
So in my system I have a folders which are organized as follows:
Photography
+-------------->Archives
+-------------->Processed
+-------------->Misc
+-------------->Prints
+-------------->To File
Taking each folder:
To File: Everything I shoot gets downloaded in here. This is the clearing house before images enter my digital warehouse. Sometimes it takes weeks before I get to clearing stuff away. But this is better than downloading images to some arbitrary folder and then forgetting about it.
Prints: Folder to store processed images that I want to send for printing. These are usually finished TIFF files usually weighing in about 30 MB each.
Misc: This is a catch all folder, containing various odds and ends. I store lots of Photoshop types docs. Nothing that important.
Archives: This is the primary folder. This is organized as follows:
+-->Archives
+-------> 2002
+-----------> contacts
+-----------> 01-Jan 14, 2002
+------------------> 2002_01_14_45.jpg
+------------------> raw
+----------------------> 2002_01_14_1.cr2
+----------------------> 2002_01_14_2.cr2
+------------------> dng
+----------------------> 2002_01_14_1.dng
+----------------------> 2002_01_14_2.dng
+-----------> 09-Sep 01, 2002
+... More of the same
+-------> 2003
Key points of the structure above, every year(2002, 2003, 2004) is a top level folder under Archives. Each year has a subfolder for each day I actually downloaded images into the PC. The folder names are created using the following format '01-Jan 01, 2002' so that they appear in Windows Explorer etc properly sorted and in chronological order.
Every download date folder (e.g. '01-Jan 01, 2002') may contain up to 3 different types of images. If the images are only .jpg or .tiff they get dumped directly into the folder (see '2002_01_14_45.jpg'). If the images are other formats, like Canon EOS 20D's CR2 RAW format they are dumped into the 'raw' subdirectory. After images are dumped, they are all renamed to standard convention of YEAR_MONTH_DAY_IMGNUMBER.EXTENSION (see 2002_01_14_1.cr2) . One free renamer is 1-4a-rename. (Highly recommended) Also, since I don't believe in the longevity of the CR2 image formats, I hedge my bets by creating .DNG files under sub folders. DNG files are in the DIGITAL NEGATIVE format introduced by Adobe. This is meant to replace all the thousands of RAW image formats used by each different camera manufacturer. (for eg, .CR2, .CRW, .NEF) Note all images are only originals.
There is also a contacts folder that contains contact sheet .jpgs for all originals in that year. [A contact sheet is nothing but a image that contains thumbnails of other images.] You can break it into multiple parts if you have too many images. But the idea is to have a few jpgs that have small thumbnails of the entire year. [ You can create contact sheets many ways, the most easy way is by using Adobe Photoshop] Here is a sample contact sheet:
Processed: This is the secondary folder. This houses all the images processed.
+--Processed
+-------> 2002
+-----------> 01-Jan 14, 2002
+----------------------> 2002_01_14_2_color.psd
+----------------------> 2002_01_14_2_bw.psd
+-----------> 09-Sep 01, 2002
...
+-------> 2003
The reason there is a processed top level folder is that most images from a digital camera are unfit for consumption directly, especially if you are shooting RAW format. They need to be tweaked, modified or Photoshopped before they are deemed acceptable. So how do we deal with this? We can take any file from the archives, do the tweaks and save the results in the day we did the tweak under the Processed directory. The folder structure is very similar to the Archive section. *IMPORTANT* the files are saved under the same name as the file they originally originated from. Trailing information can be appended to add some more data about what the change was. (Eg, _color, _bw, _art etc)
Ok, now that you know about the system, let me show you how it solves the issues highlighted in the beginning of the article.
1) People are lazy
Yes, this is meant for really lazy people. All you need to do is save your images to the right folder and you are done. You might need to run the DNG converter and the batch renamer but these take a few minutes.
2) Must be universally portable
No proprietary databases etc. Everything is in the picture, folder and name itself. All this should be very portable. If they cannot port a binary folder/file all hope is lost. Well there is no cataloging you say. Well cataloging is for the dogs, I say. All the time spent cataloging is much better spent learning a new nifty trick in Photoshop or capturing more nifty images.
3) Must be backup friendly
Very easy to do incremental backups. Lets say you have 2 years under archive. 2002 & 2003. If you added the contact sheet for those years and saved them. You are ready to back them up to any media, even to external media. As long as you have the contact sheets, you will know which CD has the pics you want. Also this allows incremental backup as you will only adding new folders to the Archive/Processed folder, never go back to an old folder.
4) Must be easy to use
No complicated app to learn. Just browse your Hard drive.
5) Images must be easy to find
All images are named using the same syntax. So every image is unique. It is possible to figure out what the relationship is between images etc just by looking at their names. It is also easy to locate specific images by browsing the contact sheets.
That’s it, you have my low cost digital image management system. I hope you found it useful. Drop me a line if you have better solutions than these. I would love to hear from you.
