XMP sidecar read+write support for ANY file type?

[polarbeer]

Hi,

Sorry if this is in the wrong forum section.

I'm a hobbyist trying to find a DAM software to manage my photos and videos. Files are mainly from my Panasonic GH2 camera, so they are mostly JPG+RW2 photos or MTS/M2TS videos (AVCHD).

I would like to add necessary metadata to my files by using XMP sidecar files ONLY - leaving parent files intact. Ie. even for files that support embedding metadata inside the main file, I would like to use XMP sidecars instead. It would be nice, if the DAM software would allow XMP sidecars for any file type user wants to - even though the file type wasn't directly supported by the DAM software (office file, music file etc.). Ie. although the DAM software couldn't create thumbnail for the file, it would allow creating sidecars for it. So you wouldn't have to wait until your selected DAM software starts to support some specific file format - you could always attach metadata to ANY file type by using XMP sidecars.

None of the DAM tools I have tested so far have managed to work like this. Closest match I have found so far is DigiKam, that allows you to create XMP sidecars for all image file types - leaving original files intact. For some reason it doesn't create XMP sidecars for video files yet. Apparently metadata is stored only in DigiKam's database, which I don't trust for long term storage.

So is this approach I describe above possible in Photo Mechanic 2012 released in start of this year? Does new version support also AVCHD (MTS/M2Ts) video thumbnails - or even better - direct playback inside PM?

It would be nice to know, why aren't sidecars more commonly used and supported by different DAM tools. It seems like an ideal solution to store file specific metadata. Of course sidecars have their shortcomings also - but for me, they seem the best way to store all the additional data. Best way would be to allow user to choose where metadata is stored for every file type. A) Embedded in file (+ database) B) Stored in sidecar (+database) C) Embedded in file + sidecar (+database) D) Database only. This would offer the much needed flexibility, because every one has their own workflow.

(I haven't installed current Photo Mechanic trial, because I'm currently so busy, that 30 day trial would end before I have enough time to get a grip of it. However by just reading PM v4.6.3 PDF-manual it seems, that at least the current version doesn't support XMP sidecars for any file type user wants to.)

Thanks for your help!

-pb

[Kirk Baker]

Sorry if this is in the wrong forum section.

I'm a hobbyist trying to find a DAM software to manage my photos and videos. Files are mainly from my Panasonic GH2 camera, so they are mostly JPG+RW2 photos or MTS/M2TS videos (AVCHD).

I would like to add necessary metadata to my files by using XMP sidecar files ONLY - leaving parent files intact. Ie. even for files that support embedding metadata inside the main file, I would like to use XMP sidecars instead. It would be nice, if the DAM software would allow XMP sidecars for any file type user wants to - even though the file type wasn't directly supported by the DAM software (office file, music file etc.). Ie. although the DAM software couldn't create thumbnail for the file, it would allow creating sidecars for it. So you wouldn't have to wait until your selected DAM software starts to support some specific file format - you could always attach metadata to ANY file type by using XMP sidecars.

None of the DAM tools I have tested so far have managed to work like this. Closest match I have found so far is DigiKam, that allows you to create XMP sidecars for all image file types - leaving original files intact. For some reason it doesn't create XMP sidecars for video files yet. Apparently metadata is stored only in DigiKam's database, which I don't trust for long term storage.

So is this approach I describe above possible in Photo Mechanic 2012 released in start of this year? Does new version support also AVCHD (MTS/M2Ts) video thumbnails - or even better - direct playback inside PM?

It would be nice to know, why aren't sidecars more commonly used and supported by different DAM tools. It seems like an ideal solution to store file specific metadata. Of course sidecars have their shortcomings also - but for me, they seem the best way to store all the additional data. Best way would be to allow user to choose where metadata is stored for every file type. A) Embedded in file (+ database) B) Stored in sidecar (+database) C) Embedded in file + sidecar (+database) D) Database only. This would offer the much needed flexibility, because every one has their own workflow.

(I haven't installed current Photo Mechanic trial, because I'm currently so busy, that 30 day trial would end before I have enough time to get a grip of it. However by just reading PM v4.6.3 PDF-manual it seems, that at least the current version doesn't support XMP sidecars for any file type user wants to.)

PM only supports XMP sidecar files on RAW files.  They're the only file type (well really there are dozens of RAW formats like NEF, CR2, RW2, etc.) that other apps expect to see XMP sidecar files for.  I know of no other apps (DAM or otherwise) that generate XMP sidecar files for non-image file formats.

We will not be making an accommodation for this in PM5 (at least not for the initial release since the feature set is not going to be added onto before release.) because there just isn't enough time left to implement any other features.

-Kirk

[polarbeer]

PM only supports XMP sidecar files on RAW files.  They're the only file type (well really there are dozens of RAW formats like NEF, CR2, RW2, etc.) that other apps expect to see XMP sidecar files for.  I know of no other apps (DAM or otherwise) that generate XMP sidecar files for non-image file formats.

We will not be making an accommodation for this in PM5 (at least not for the initial release since the feature set is not going to be added onto before release.) because there just isn't enough time left to implement any other features.

-Kirk

Thanks for your quick response and info you provided Kirk!

How about AVCHD support? Can you give any details about that in PM5?

-pb

[Kirk Baker]

PM only supports XMP sidecar files on RAW files.  They're the only file type (well really there are dozens of RAW formats like NEF, CR2, RW2, etc.) that other apps expect to see XMP sidecar files for.  I know of no other apps (DAM or otherwise) that generate XMP sidecar files for non-image file formats.

We will not be making an accommodation for this in PM5 (at least not for the initial release since the feature set is not going to be added onto before release.) because there just isn't enough time left to implement any other features.

Thanks for your quick response and info you provided Kirk!

How about AVCHD support? Can you give any details about that in PM5?

Don't know yet if that format will be supported.  What OS are you running?

-Kirk

[polarbeer]

Hi!

I'm running Windows 7 Home Premium (64 bit).

By the way, although hardly any DAM tool supports XMP sidecars for video or other non-image files, why couldn't Photo Mechanic be the first?

As video asset management tools are really scarce or non-existent, I think this is really functionality needed by more and more users! It is hard to point any new camera model that wouldn't be marketed as HD video capable. I don't understand why video features aren't requested more among DAM tools! (Lightroom 4 beta seems to make some improments in this area...)

Isn't XMP marketed as format that can be used for ANY file type? By using it via sidecars instead of embedding inside "parent" file has some clear advantages for a regular home user.
- Fast backups to external hard drives, when only changed sidecar files have to be copied instead of huge video files.
- User can check XMP sidecars with simple text editor like Notepad to be sure, that data he has entered really is there. No need to understand database structures or cross check with other metadata capable software to see that embedded metadata is actually inside file. After all, it is user's data, not software vendor's data. 

Disadvantage is that sidecars (usually) have to be moved, copied and deleted with their parent files. If a user isn't aware about this, problems arise. However if a user really is using a DAM tool, he can easily absorb philosophy of using them. This could be also advantage when sending your files away. By sending only the media file you could be sure that no personal metadata is sent inside it - no need to use software to "clean" media file from metadata.

As I don't know anything about software engineering, I don't know how much work it is to implement sidecar support. But if a software already supports read+write of XMP sidecars for some file type (like RAW images), it feels like it shouldn't be a major feature to implement for other file types too? Of course I could be totally wrong about this! 


-pb

[Kirk Baker]

I'm running Windows 7 Home Premium (64 bit).

Does Windows Media Player play your AVCHD videos?

-Kirk

[polarbeer]

I'm running Windows 7 Home Premium (64 bit).

Does Windows Media Player play your AVCHD videos?

-Kirk

Yes it does. I think Windows 7 supports AVCHD "natively" - what ever it means. 

-pb

[sgbotsford]

Even though sidecar files are not supported by many other programs, I can see advantages to the OP's request, especially with movies.

Consider the following workflow:

I'm currently trying to organize many thousand slides from an archive.

1.  Scan to tiff.  (45 MB each)
2.  Convert to png (30 MB each)
3.  First metadata pass:  Add keywords for frame number, year/month stamp, mount type/markings (E.g. kodachrome, GAF, snapmount, plain cardboard)  (This, I hope will allow me to collect sets of images about the same event.)
4.  Second metadata pass:  Add keywords for any handwritten labels.

5.  Re-organize by event. 

6.  Third meta-data pass.  Add event info to all images of an event.

7.  Downsize to facebook size, water mark, post to facebook alumni group.

8.  Fourth metadata pass: Add tag info from facebook for people.

So over the course of a few weeks, each file will be rewritten a minimum of 4 times.  This is going to play havoc with Apple's time machine backup.

On the other hand, writing as sidecar files only means that the changed files are tiny.  Much easier to keep backed up.  The other advantage would be speed.  If you are writing one field at a time to a bunch of pictures, it's far quicker to write to sidecar files than to the original images.

It would be an easy feature to implement:  PM already can write sidecar files.  So it's just a matter of adding the choice for other formats.  It would also need one more menu option, "Write sidecar file data to images where possible."

Of course there is the alternative to just keep the metadata in a database.  That seems fragile.  I've seen database corruption too many times.  Belt & suspenders.









Post by event to facebook for alumni comment.

[ramblinwreck]

I would like to see this also. I asked about this before:
http://forums.camerabits.com/index.php?topic=5894.msg27032#msg27032

[polarbeer]

If you want to use XMP sidecar files for ANY file type, you might want to check JPhotoTagger.

www.jphototagger.org

It's freeware or even opensource, if I have understood it right. I don't remember how I found it, but it's a very interesting piece of software made mostly by one man. The feature set might not be on the level of commercial alternatives yet, but you can manage your videos with it too. You can choose to use long sidecar file naming (photo.jpg.xmp instead of photo.xmp) if you want to. Thumbnails for most video file types can be created, if you install a couple of other free software. There's a detailed pdf-tutorial on how to achieve this in Windows.

-pb