Catalog News - A Huge Disappointment

[Andrew]

Yesterday I set up a new Lightroom catalogue and imported 35,000 nefs into it.  Most of these had fairly limited added metadata such as keywords (I have a lot of key wording still to do!).  The total time taken by LR4 to catalogue them was about 80 minutes, faster than I had expected.  On that basis perhaps your 300,000 images would take say 13 to 14 hours.

[NeilR]

Hi Andrew,

Your numbers look suspiciously good 

That works out to about 7 images per second.  If these are 12 mpx nefs, and figuring 15MB average per NEF, then LR4 is reading NEFs at the rate of 105MB per second.  A high end machine can only copy NEFs at that rate, much less render down the images for thumbnails, prep the metadata and write the DB records, plus whatever else it needs to do.

If the app is smart then it is not reading the entire NEF file, but just the embedded JPG.  If that is the case then out of camera embedded JPGs are only around 1.5MB, but if they were edited by Capture NX then they are around 5MB.  So there could be a big range there depending on how your process your NEFs and what percentage, etc.  But in even in this case a rate of 435 NEF's per minute sounds rather fast, simply based on what I've seen with various apps I've tried.  I have not demo'd LR4.

From a slightly different perspective, those 35,000 NEFs are probably very close to a half terabyte of data.   I'd like to have  amachine that could just duplicate that amount of data in 80 minutes, without doing any other processing 

I don't like to disagree with someone when I have no experience with what they are talking about- just trying to say that the number raises some alarms.  Is it possible that LR4 is only partially processing the images on the initial ingest and then finishes them in a background process that is not obvious?

[Richard Costin]

Hi Neil.

fyi Lightroom treats importing and thumbnail generation as two separate tasks.
So it could import (i.e. be made aware) of a large number of images and the preview generation/reading would lag behind (and it does).

Also importing doesn't necessarily mean copying the images, they can remain in place if set to. To import the image into lightroom (assuming you are simply pointing it at a folder on a hard drive, not copying/moving them to a new location)  is very fast. You just have to wait a little after the import process to see all the previews/thumbnails etc..

Whether or not you include the preview generation in the import timings is another matter though!

Cheers,
Richard.

[NeilR]

Hi Richard,

I fully understand that LR (or any other cataloger) is not actually copying the image files.  I used the copy operation as a sanity test to suggest the bandwidth isn't there to actually completely ingest and process images at the rate claimed.  And simply copying files is far less machine intensive than what has to go on to render thumbnails, off line cache images (if that is supported) and deal with all the metadata and the DB updates.

I had a fuzzy recollection that LR does the vast majority of the work in the background, and I recall that idimager does too (just based on old evals I did).

The effect of trying to ingest 300K images would be that LR might "come back to life" after 8 hours or so, and then spend the next 200 or (possibly far) more hours finishing the process.  That might work well or not, depending on a lot of things, and how well the app deals with things like being shut down in the middle and then picking up again, and things I wouldn't even want to try to predict ahead of time.  It's a really tough process to get through.  That I learned from iMatch, and I think I only had 50K or so images when I got started with that cataloger.

It will be interesting to see how CameraBits deals with all this.  I would assume they will also do a background process, which is something I wish iMatch would do.  I have a lot of confidence that this will be done right, just based on all the smart work that was done with PM.  It's obvious they think about those details.  But regardless of how well it is done, it is really an ugly process to get through :-).

[Andrew]

I don't know what LR was doing on import (I am not a technical expert), but I confirm it was not moving files, only pointing to them.  I doubt it was generating previews.  It certainly was much faster than I was expecting.

For ingesting, PM is much faster than LR, but the process I described was not ingesting.

[Pavel]

I very much want a cataloguing program from camerabits myself but as someone who has tried every, and I mean EVERY cataloguing program out there to date, wasting money and countless hours with never working junk,  I wait patiently and with fingers crossed.
 I have faith, born of prior experience with CameraBits, that the product will be well thought out and well executed. I for one would never rather have it NOW rather than when it is done to Camerabits standards.  Kirk; if it means that your upcoming program takes time - please take the time, and thank you for that philosophy!  Yeah I'm impatient, like a five year old on Christmas eve, sweaty palms and all, but I will just grin and bear it a while longer and remain grateful that your company does not play with us with half baked products - like the rest seem happy to.

[Luiz Muzzi]

Hi, Pavel
I totally agree with you.
If it means to wait to get a good product with PM standards, then we shall wait...
Rds,

-Luiz Muzzi

[edcatlett]

I completely agree.  I wish I had just been waiting 3 years.  In this thread:  http://forums.camerabits.com/index.php?topic=4379.30
3 years ago I had already been waiting 2 years

This is partially because Photo Mechanic is so good but in all reality if every other catalog program I have used didn't completely suck I wouldn't be considering catalog at all. 

This is one of the worst examples I have ever heard of regarding poor project planning, even worse communications, horrific management of user expectations a horrible job of listening to user requests, etc. etc. etc.

[JAS Photo]

Any updates on the catalog software?

[Bill Kelly]

This is one of the worst examples I have ever heard of regarding poor project planning, even worse communications, horrific management of user expectations a horrible job of listening to user requests, etc. etc. etc.

Unfortunately I can't really argue with this. 

Any updates on the catalog software?

We're still working on improvements and fixes to Photo Mechanic v5, with another build due out this week.  After that we have one further PMv5 build slated for release about two to three weeks from now.

At that point, PMv5 should be relatively stable, at which point we intend to shift our resources fully toward the usable beta of the catalog software outlined in my earlier posts.

(Admittedly, last time I posted I'd anticipated this transition in development occurring closer to mid-August, than, as it is turning out, mid-October.  Nevertheless, we are indeed closing in on having a stable PMv5, and we're truly eager to get cracking on the catalog beta.)


Regards,

Bill

[ostonica]

Thanks, very much,  for this update. Bill.

Jim

[vasha]

Thanks for the update Bill! I am looking forward to Catalog and have faith Camerbits will deliver a hit!

[crcowan]

Some of us have been on Idimager V5 for cataloging. This has just been discontinued. It represents a major opportunity for Camera Bits to embrace all of those customers. I haven't seen Photo Mechanic mentioned in the Idimager forums. Somebody might want to let people know that the PM catalog is coming.

All of a sudden I'm much more interested in the catalog module

[Woodie]

Some of us have been on Idimager V5 for cataloging. This has just been discontinued. It represents a major opportunity for Camera Bits to embrace all of those customers. I haven't seen Photo Mechanic mentioned in the Idimager forums. Somebody might want to let people know that the PM catalog is coming.

All of a sudden I'm much more interested in the catalog module

As a long time Idimage user (started with V4), I'm currently debating whether or not I upgrade to Photo Supreme or continue (what is it, four years now) the wait for CameraBits catalog app.  The latter SHOULD integrate easier with PM but it's still vapor ware.  Great features and integration don't help me much when they are only promises (hey, this sounds like the campaign promises we're hearing in the U.S. right now). 

[Willow]

Take your time, do it well and do it once.

I prefer a delay to occur and the outcome to be solid and stable software than something in Beta with problems.

Best,   WilloW