I'll probably have to write this myself ....

Tannin

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
4,448
Location
Huon Valley, Tasmania
Website
www.redhill.net.au
No, not the post, the program.

I need something to do a specific, quite well-defined task. Easiest to explain the "why" first, then I'll try the "what".

WHY
I take a lot of photographs, sometimes several thousand in a week. Some are taken in both raw and JPG, some JPG only. They all end up with a filename in the form:

010203-123456-XXXXX.XXX This represents:

2001
February
3rd
-
timestamp
-
(0 or more extra letters which differentiate between files with identical timestamps)
.
jpg or cr2 (= Canon raw file)

So it's a pretty straightforward naming scheme. There is a naming scheme for folders too, but that's not relevant here. For simplicity, I'll just pretend that the top-level folder is called HOME.

OK, so I start with a lot of files. First thing I do is sort out the raw files and put them in their own sub-folder, HOME/RAW. Then I flick through the JPGs, sorting the good from the bad. I never delete anything (well, practicallly never). I find it more time-efficient to move stuff. This is a multiple-pass process. Let's say I start with 1000 JPGs in HOME/. (There will be maybe 500 matching raw files in the HOME/RAW folder as well. We can ignore them for now.)

I look at every image, moving those that are clearly the worst ones into HOME/X1. When I've got rid of ~50% of the files, I rename HOME/X1 to HOME/X9 - these are the files I will probably never want to access again - but I keep them because (a) storage is cheap, (b) sometimes I make mistakes and I can work faster if I know that a mistake can always be reversed if need be, and (c) because sometimes (not often) I want some information contained in an otherwise worthless photograph - for example, a road sign so I can find a particular place again, a differerent view of a bird or plant that I can't identify from the more artistically worthwhile shots, stiff like that.

I usually take a break at this point, and come back to this folder a few days or weeks later, when I can see the images afresh and make better decisions. Then I trawl all the way through again, once more moving the worst ~50% into HOME/X1. When I'm done, this is renamed to HOME/x8.

Lather, rinse and repeat. From a starting poiint with 1000 files in HOME, I end up with something like this the file structure below:

HOME - 54 files
HOME/X5 - 20 files
HOME/X6 - 42 files
HOME/X7 - 104 files
HOME/X8 - 273 files
HOME/x9 - 507 files
HOME/RAW - 500 files

Does this sound complicated and cumbersome? Not really. It's pretty-much second-nature, and I can do it in my sleep - except for the human-being-level-decisions about which images to keep and which to get rid of.

In reality, it's a bit more complicated than it appears above (though still second nature) because the file structure above is what ends up on my file server, while the sorting actually takes place on my Thinkpad, so there is some copying and moving and deleting between machines involved. (Essentially, it is file synchronization, only backwards - I need to make the many-files folder the same as the few-files folder, where file synchronisation software is designed to do the reverse. I do it by hand. No big deal, the sequence is:

1: Count the files in the top level folder on the server
2: Count the number of files in both levels on the Thinkpad (e.g., HOME and HOME/X6).
3: If the two counts are different, there is an error. Sort it out. Otherwise:
4: Delete all the files in the HOME folder on the server (leaving the sub-folders intact)
5: Move HOME/X6 to the server
6: Copy the files in HOME/ to the server
7: Count the files again to be sure there is no error.

Takes longer to write down like this than it does to actually do it. This part isn't my problem (though an automated way to do this would be worth considering, if anyone knows of one). And, in reality, I rather enjoy the few minutes it takes to do this bit. There is a real satisfaction in watching my laptop drive get a little bit emptier after several hours hard work sorting images.

Now all this sorting takes place with the JPGs. The raw files are all archived off to the server right away, because I don't have the space to store them on the laptop. Mostly, when I want to do something with an image like post-process it and prepare it for further use, I work from the JPG. I only resort to the raw file when I get that rare combination of an image which is not good enough to use direct from JPG, but is good enough to use if I do the raw conversion myself and tweak it. Bad white balance is the most common reason, bad exposure less common - if it's too far out to use the JPG, the raw is unlikely to be presentation-quality anyway. But I do want particular raw files from time to time, so I keep them.

Now we get to the real problem. I would like to have the raw files sorted in exactly the same way as the JPGs. Remember, every raw file has an equivalent JPG, but not every JPG has a matching RAW. The program logic should be simple enough:

* Read filename from the HOME/RAW folder
* Look in the HOME/X9 folder for a *.JPG match. If found, move the raw file to HOME/RAW/X9
* Look in the HOME/X8 folder for *.JPG. If found ....
* etc
* repeat

Is there an off-the-shelf program that can do this for me?

OR, can I figure out a way to write it myself? Bear in mind that the only programming language I'm current in is PHP. You can run PHP on Win XP (as opposed to on a web server) but I'd have to do a fair bit of messing about with the install/config/security setup, and it's using a screwdriver to hammer nails.

I do not want to have to learn any new skills! If I'm going to write it in any language other than PHP, it has to be a language I can pick up and be productive in right away. I simply don't have time to learn something completely new. I used to be reasonably fluent in BASIC, Fortran, Pascal, Z-80 assembler, and Modula-2, but all those were a long, long time ago. C or any of the C-like languages are definately out of the question! Has to be something I can pick up in a few hours.

So ..... any ideas?

* One exception: if the best answer turn out to be something essentially involving some fairly serious regular expressions and not too much else, I'll regard developing my (small) skills in Pearl or POSIX regexes as a useful thing, and worth spending time on.
 

timwhit

Hairy Aussie
Joined
Jan 23, 2002
Messages
5,278
Location
Chicago, IL
I think you would be best served by writing some VBScript. I think you could hack something together from online examples in a couple hours pretty easily.
 

Fushigi

Storage Is My Life
Joined
Jan 23, 2002
Messages
2,890
Location
Illinois, USA
Here's a site with lots of batch & scripting info & samples. Using the description of NT's FOR command I came up with this batch file:
CD /HOME/X9
FOR %F IN (*.JPG) DO MOVE /HOME/RAW/%~nF.RAW /HOME/RAW/X9/
CD ../X8
FOR %F IN (*.JPG) DO MOVE /HOME/RAW/%~nF.RAW /HOME/RAW/X8/
etc. to count your way down to just X.

The ~n strips the extension from the file name which should leave you free to append the .RAW.

When there's no matching RAW file it should just toss an error & hopefully continue. If it doesn't continue, put the MOVE in a separate batch file and add an IF ERRORLEVEL check.
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,705
Location
Horsens, Denmark
I don't know nearly enough to help you, but it is an interesting insight into your workflow. It's actually not that different (though more complicated) from mine.
 

Howell

Storage? I am Storage!
Joined
Feb 24, 2003
Messages
4,740
Location
Chattanooga, TN
In reality, it's a bit more complicated than it appears above (though still second nature) because the file structure above is what ends up on my file server, while the sorting actually takes place on my Thinkpad, so there is some copying and moving and deleting between machines involved. (Essentially, it is file synchronization, only backwards - I need to make the many-files folder the same as the few-files folder, where file synchronisation software is designed to do the reverse.

I don't understand what you mean in this paragraph but "Beyond Compare" is a pretty sophisticated file comparison utility.
 

udaman

Wannabe Storage Freak
Joined
Sep 20, 2006
Messages
1,209
I have no solution, but what Tannin is describing sounds like a common digital asset management situation.

Being common, would not any number of professionals have worked with this before, such that there is a way to do this with some DAM software? Like IView Pro or Cumulus?

http://www.canto.com/

http://www.iview-multimedia.com/mediapro/iview-for-graphic-artists.php

I would ask the question on Fred Miranda's forums of some other photography forum. Might already be some shareware programs that those forum members can direct you to, which accomplish those goals?
 

Tea

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
3,749
Location
27a No Fixed Address, Oz.
Website
www.redhill.net.au
We have a small domestic problem just at the moment which seems to be preventing the normal routine. Nevertheless, thankyou all for your help and I'm investigating a couple of possibilities.

Beyond Compare doesn't seem to do what I need, though I need to check a bit more carefully to be sure. (I remember Time recommending this program a year or two ago.)

The batch file suggestion from Fushugi has some legs, I think. Thanks guys, I'll keep you posted on progress.
 

Will Rickards

Storage Is My Life
Joined
Jan 23, 2002
Messages
2,011
Location
Here
Website
willrickards.net
I tried to read your requirements but I couldn't and skipped ahead. Damn internet attention span. Maybe I can revist this thread later and have some useful ideas. But a succinct problem description might help too.
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,705
Location
Horsens, Denmark
I tried to read your requirements but I couldn't and skipped ahead. Damn internet attention span. Maybe I can revist this thread later and have some useful ideas. But a succinct problem description might help too.

I just got home from splitting 2 pitchers of margaritas with 2 women, attention span indeed...
 

Splash

Learning Storage Performance
Joined
Apr 2, 2002
Messages
235
Location
Seaworld
Just as a side note to this conversation, one of the best -- if not THE BEST -- file renaming utility software is BulkRenameUtility, which is currently at version 2.7.0.2. It is to be found here (including a 64-bit version):


http://www.bulkrenameutility.co.uk/
 

Tannin

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
4,448
Location
Huon Valley, Tasmania
Website
www.redhill.net.au
And you couldn't figure it out? I'm not surprised. Microsoft batch file syntax is truly terrible. Lacking in any sort of logic that I can fathom, and as for documentation .. what documentation? It is simply amazing how little batch file documentation there is on the web. I spent some time on that problem yesterday and after a couple of hours, Tea. I gave it up as a bad job. And I used to be a pretty fair hand at batch files back in DOS days .... trouble is I can't remember any of it now.

Don't feel bad about not being able to figure it out. After all, I couldn't do it either.
 

Tea

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
3,749
Location
27a No Fixed Address, Oz.
Website
www.redhill.net.au
Didn't find any instructions.

Lots of pages - pages and pages and pages of stuff that was useless and mostly harmful to the brain cellz of small apes, but nowhere anywhere is the there a simple bloody syntax guide. Sometimez there are two % signs, sometimes there is only one, sometimes you have to have them on both sides of the variable, sometimes you have to only have them in front of the variable, but nowhere does anybody say why! Or when.

I feel sick. My brain hurtz!
 

Tea

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
3,749
Location
27a No Fixed Address, Oz.
Website
www.redhill.net.au
I started with Fushigi's code:

CD /HOME/X9
FOR %F IN (*.JPG) DO MOVE /HOME/RAW/%~nF.RAW /HOME/RAW/X9/
CD ../X8
(and etc)

The key line is:
FOR %F IN (*.JPG) DO MOVE /HOME/RAW/%~nF.RAW /HOME/RAW/X9/

Turns out that, for no good reason at all, you have to write:

FOR %%Z in (*.jpg) DO MOVE "%HOME%\raw\%%~nZ.cr2" "%HOME%\raw\x9"

The whole thing looks like this:


SET HOME=%CD%
IF NOT EXIST x9 GOTO finished
IF NOT EXIST raw\x9 md raw\x9
cd x9
FOR %%Z in (*.jpg) DO MOVE "%HOME%\raw\%%~nZ.cr2" "%HOME%\raw\x9"
cd %HOME%

(Repeat as many times as required. I could probably figure out how to make it loop recursively down and do the X8, X7, X6 and so on, but it was easier just to copy and paste it nine times.)
 

Tannin

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
4,448
Location
Huon Valley, Tasmania
Website
www.redhill.net.au
Wow! You did that by trial and error? No wonder your head hurts! But you'd never have managed it without Fushigi giving you something to get you started. You had better thank him. No, wait, you are in no condition to do anything. I'll thank him.

Thankyou Fushigi!

Oh, and thankyou Tea!
 

Fushigi

Storage Is My Life
Joined
Jan 23, 2002
Messages
2,890
Location
Illinois, USA
You're welcome. It was interesting to try & figure out how it could be done without having to resort to outside utilities.
 

Tannin

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
4,448
Location
Huon Valley, Tasmania
Website
www.redhill.net.au
It's actually rather odd that there are no pre-written utilities to do this sort of thing. Not this exact task, of course, but the general scheme of it, what you might call reverse synchronisation.

There is a zillion apps out there that find and offer to delete duplicate files, and there is another zillion that are designed expressly to PUT IN to folder B what is already in folder A ..... but apparently nothing that can TAKE OUT of folder B what is NOT present in folder A.

Seems like an obvious enough task to want to do. Go figure.
 

Fushigi

Storage Is My Life
Joined
Jan 23, 2002
Messages
2,890
Location
Illinois, USA
Sync Toy is the tool from MS that should have this capability. I haven't used it in some time so I'm not sure if it could handle your requirements, but I somehow doubt it.
 

Tannin

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
4,448
Location
Huon Valley, Tasmania
Website
www.redhill.net.au
Getting on for ten years since this thread was live, and those same batch files have performed flawlessly all that time, and look set to continue doing so well into the future. Tea extended them a bit shortly after I last visited the thread, so that the streamlined workflow is as follows:

Standard preparation remains:
1: Copy files off the camera cards. The downloading software auto-creates subfolders for me.
2: Manually move the raw files into HOME/RAW, checking at the same time that the number of raw and JPG files is equal. (I always shoot raw + JPG these days, ever since camera buffers got a lot better.) Unequal number = trouble of some kind.
3: Backup onto the archive machine.


The actual post-sorting process is:
1: (Optional, only if I still have the raws on the laptop and want to sort them too, usually because I want to delete the local copies of all but those I might want to post-process.) Right-click on the folder to run a variation of the batch file above, which sorts the raw files to match the JPGs. (A simple registry mod gives me access to the batch process by right clicking on any desired folder.)
2: Copy only the X9, X8, X7 JPG folders to the archive machine. No need to delete, move or rename anything, nor to copy anything else. Just copy the folder(s) of rejected JPGs.
3: Right-click on the destination folder in the archive machine and run the batch file. It's a different batch file on the same theme. It sorts the raw files as before, but it also removes all the reject JPGs from the HOME folder, placing them in a "delete me" folder instead. (Paranoia there: the batch file could just delete them automatically, but I prefer to do that myself.)

Easy as pie!

Fushigi, your helpful suggestion - together with Tea's hard work - has saved me countless hours of tedious manual labour over the years. Hundreds of hours at least. Thankyou again!

Splash, I don't recall if I started using Bulk Rename Utility as a result of your suggestion or found it some other way, but it is on my priceless-must-have-utilities list. Remove Empty Directories (RED) is also on that list.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,894
Location
USA
Sounds like you built a nice reliable script to get your work done and it's pretty interesting that it's so relevant after so many years. Your workflow must be spot-on that you can continue to do the same process and it's scalable. I don't know why this thread is unfamiliar to me, I must have been hibernating at the time when you wrote it. Depending on how you want to extend the script and process, the act of removing empty directories could be relatively easy work to code in Python but that doesn't quite mesh with your batch scripts (unless if they are used to call a Python script).
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
17,339
Location
USA
Sounds like you built a nice reliable script to get your work done and it's pretty interesting that it's so relevant after so many years. Your workflow must be spot-on that you can continue to do the same process and it's scalable. I don't know why this thread is unfamiliar to me, I must have been hibernating at the time when you wrote it. Depending on how you want to extend the script and process, the act of removing empty directories could be relatively easy work to code in Python but that doesn't quite mesh with your batch scripts (unless if they are used to call a Python script).

I don't remember it either. I'm still using Breeze Downloader Pro to download three simultaneous copies to a laptop and an external 2.5" drive (SSD lately). I usually pull the drive from the enclosure, mainline it into a drive bay, and use the usual filesync programs to put a copy on the working drivespace and backup (NAS). Then I use BreezeBrowser to correct the timestamps to ±1 sec. if necessary and rename/number images from all cameras into a logical alphanumerical sequence. I make a spanned container with a checksum and send at least one copy offsite right away.
 

Tannin

Storage? I am Storage!
Joined
Jan 15, 2002
Messages
4,448
Location
Huon Valley, Tasmania
Website
www.redhill.net.au
Cheers Doug. I was a little unclear with what I wrote just before. Remove Empty Directories plays no direct part in this workflow, I just use it for various unrelated tasks. In general, there is no need to delete directories as part of my photo workflow, other than the (non-empty) one on the archive machine which contains some redundant copies. I probably should just have the batch file delete that one automatically but, being paranoid at heart, I prefer to eyeball it for myself and do it manually. But RED is a fantastic little tool! Just this week, for example, I was sorting out a great big mess of movies and stuff, collected over many years and often with multiple not-quite-identical copies, all in maybe a couple of hundred nested folders. With RED, it's a simple matter to trawl through gradually moving, renaming, and deleting things more or less at random, leaving those I'm unsure of to deal with later, and running RED every now and then to save endlessly clicking into empty folders to make sure that each particular one is complete. And dozens of other tasks.

Lunar, that's an interesting insight into your workflow, and it sounds very practical. I too use Breeze Downloader Pro. My version is pretty ancient now. I haven't troubled to upgrade lately because it seems to do everything I want. I don't get it to make multiple copies because, on a field trip, I just want to get a single copy onto the laptop ASAP so I can put the cameras away. That done, I have two copies (counting the flash cards in the cameras), and before I go to sleep I make a third copy onto external drives. At the same time, I do an extra copy of the previous day's work. First thing the following morning I format the flash cards.

So I have at least two copies at all times from the moment I've finished reading the flash cards (laptop plus flash cards and/or external #1), and three copies as of 24 hours later (external #2 as well). Sometimes I use extra external drives, especially on longer trips.

Actually, I should slip over to Breeze to see if Downloader Pro has added any features I'd find useful in the last five years or so since I last updated it. Thanks for the reminder.

EDiT: Nope. Lots of new things, mostly minor, but none relevant to me.
 
Top