Need advice re workflow, tag management in ASP and iMatch

[coolrat]

I have several questions regarding workflow, and especially regarding using keywords to tag images.

I'm doing research on folk organizations in rural China and have thousands of photos from dozens of villages on a wide range of topics- many of which deal with religious groups.

My old work-flow was slow and inefficient. I shot everything in RAW +JPEG:
1. I would place all the photos taken on a certain day in folders named by Year, Month, Day ie) "2013_01_Jan_15 Visit to Chen village"
2. I would examine RAW +JPEG in BreezeBrowser (shown as one thumbnail), delete all the bad shots, and use BreezeBrowser's batch-rename tool to rename all the file-names to something like "2013_01_Jan_15 - Chen village wedding ritual".
I didn't realize the power of making non-destructive edits to RAW files back then, and strictly used JPEG for viewing, editing and posting to my webpage.

After learning about the advantages of using RAW, and converting adjusted RAW files to JPEG, I purchased Aftershot Pro because of Bibble's good reputation, and the fact was cross-platform and would run on Linux, and the great JPEGs it could produce. I stopped using BreezeBrowser, stopped shooting JPEGS. I began shooting in RAW only, and began using AftershotPro (AFP) for adjusting exposures and editing RAW, and making JPEGS and TIFFS. I also began to use the keyword tagging feature of ASP to tag thousands of photos.

After investing many days tagging photos in AfterShot Pro, I found a number of things that left me concerned.
Firstly, ASP's keyword manager seems to be very limited. The keyword manager has 2 panes as shown below:

[Diagram #1: Keyword Manager. I created a keyword list on the left-side. When keywords are dragged to the right side, AfterShot Pro arranges them in a 2-deep tree structure. ]

The screenshot above shows the Keyword Manager in AfterShot Pro. The user creates keyword tree-structure categories on the left side, and then drags the keywords we want to use into the pane on the right side. But I found that if I recategorize keywords on the left-hand side pane and drag them over to the right-hand side pane, the tree-structure categories are no longer shown: all keywords in the right-side pane are just jumbled into groups like "Default", "Rural Research", "Friends". This is quite confusing- the tree-structure categories are gone and we're left with long lists of keywords that are often jumbled up.

Complicating this, I soon found that the tree-structure categories that I had devised were faulty. As shown in the screenshot of ASP's Keyword Manger above, I had originally listed friends from villages listed under the name of the village where they lived. I used a list like the one in Diagram #1

The problem with that is that if I tag photos of these friends when they come to Beijing or Canada to visit, then photos in Canada will be tagged with the friend's name, as well as the name of his village, county, province, county. This will undoubtedly lead to a big mess of unrelated keywords all glommed together.

To find a solution, I went online to see how others (like http://www.keyword-catalog.com) use keywords lists in distinct categories like this:
- Animals
- Archaeology
- Attire
- Beauty and Well-being
- Birds
- Buildings and Architecture
- Business and Office
- Climate and Weather
- Communication
- Composition
- Computers and Computing
- Concepts and Broad Subjects

Then, I built my own keyword lists in Aftershot Pro's keyword manager. My lists must provide keywords related to sociology-related categories like this:

Sociology and Anthropology
.....Religion
..........Religious Rituals
..........China
...............Mainstream Religion
......................Daosism
......................Buddhism
......................Confucian
......................Islam
......................Christian
............................Catholic
............................Protestant
...............Folk Sectarian Religions
......................Group B
......................Group B
...............Folk Non-Sectarian Religions
.....Health and Illness
.....Family Life
..........Weddings
..........Funerals
.....Inequality and Social Class
.....Poverty
.....Racial issues
.....Social Change

However, after I had re-arranged keywords and created separate categories for geographical places, people, transportation, etc, and added many new keywords, the keywords that I had previously dragged over to the right-hand side pane of the Keyword Manager lost their relative connections (they were disconnected from the original tree-structured groups).

This was frustrating. I had tagged many hundreds of photos before I revised my keyword tree-structure categories. It seems that all of the photos that I had tagged using my old keyword tree-structure categories are broken. Now, I am really wondering what to do next, and this has led me to a number of questions about Aftershot Pro:

Questions regarding Aftershot Pro
1. Is that work all lost now? Will the keywords in sidecar XMP files that describe each NEF file still be readable/searchable even now that the original keyword tree-structure categories no longer exist?

2. Do I now have to re-create all of the JPEGS from the original NEF RAW files to get the keywords, and star-rankings re-embedded in the JPEGS?

3. Is Aftershot Pro's keyword manager, and overall DAM capabilities capable of maintaining elaborate keyword tree-structure categories for tagging and categorizing my photo collection?

4. Why is the Keyword Manager designed this way? Why am I unable to use the keyword tree-structure categories that I created in the left-hand side of the manager to tag my photos? Why must I drag them over to the right-side, create new 2-deep categories? There appears to be no way to alphabetize the list - this makes it quite hard to find keywords.


Questions regarding iMatch
Many professionals seem to like using iMatch as their DAM to categorize and tag photos. I have read descriptions, but hope to receive advice from users familiar with Aftershot Pro /Bibble and iMatch.

1. Will iMatch be able to handle all of this better?

2. As I understand, iMatch, if I create a detailed list of keywords and tag my RAW files (Olympus E1 .ORF), Pentax K-7 .PEF and Nikon D800 .NEF) with keywords, they keywords assigned to each photo is stored in a sidecar XMP metadata file.
If in the future I do not want to use, or cannot use iMatch, will any other program be able to read these sidecar XMP metadata files?
I don't want to invest hundreds of hours sorting my photos into a proprietary system that might disappear, or become unusable if I switch to Linux.

3. Is there a way to embed the sidecar XMP metadata files into RAW files as part of the IPTC metadata?

4. I had read that some photographers convert all of their RAW files by different camera manufacturers into Adobe DNG format because a) keyword metadata can be embedded into the DNG file; b) non-destructive edits made to the DNG file in Lightroom can are also embedded in the DNG file. Can Aftershot embed non-destructive edits made to the DNG file too?

Other advice?
I'm trying to manage a big research project, and am trying to find ways to manage and categorize typed notes and transcripts, audio interviews, and photos. My problems with textual and audio data have been resolved. Any advice and guidance on how to proceed with my photo collections will be very much appreciated! To reiterate, if I spend hundreds of hours cataloguing, ranking, and tagging photos with a carefully-developed keyword system, I want to make sure I do it correctly from the beginning, using a solid application, and in a way that the tags/keywords can be read by other software at some point in the future.

Thank you for any help you can provide. I do hope the questions I've presented are interesting.

[Franz Dur]

Hello,

This is a very limited answer since I never used the Bibble/ASP catalogue. I use Bibble/ASP and IMatch since a long time. in my opinion they work well together.

Basically IMatch can do all that you want, and much more. My workflow is to ingest my Nikon NEF and Panasonic RW2 with Photomechanic (rename and distribution of raw into folders/subfolders, geotagging and basic IPTC (Location, Title, Comments, Photographer etc.). Photomechanic can write all these metadata inside the NEF and, with some restrictions), inside the RW2 but is is safer to write it to an external xmp. Then I develop the raw in ASP and save to jpg. ASP reads the infile/xmp metadata and writes it to the jpg, so both raw and derived jpg(s) are in synchrony, metadata wise.

I catalogue the raw files in IMatch. The possibilities are really immense and will increase shortly because the long awaited Imatch 5 is "coming soon" (see IMatch forum). When the raw are fully catalogued I use one of the many IMatch scripts to copy everything from raw to jpg. Et voila....

I think it is not wise to catalogue with different programs, especially I would not mix IMatch with ASP, and any of these with Adobe programs because Adobe has its own way to manage IPTC/XMP (and it changes with time). ASP is an excellent raw converter (if your cameras are supported) and IMatch can easily handle all cataloging needs.

I suggest you ask more catalogue-related questions on the IMatch forum, you will be convinced that you will not waste your cataloging time and end up in a dead end. The full data can be retrieved in xmp form directly or with the help of some scripts, maybe even be simpler in IMatch 5.

Francis

[coolrat]

Thank you for your detailed reply!

Your workflow sounds well-thought out and efficient.
I am wondering however, if iMatch can do (or will be able to do) what you are currently doing with PhotoMechanic- "rename and distribution of raw into folders/subfolders, geotagging and basic IPTC". I'm impressed to learn that PhotoMechanic can write all of these to metadata inside the NEF file and inside RW2 (I also use both of these RAW formats!). Can iMatch do that too? If it can, I would have no need to use PhotoMechanic.

I'm trying to simply the number of tools I must use to process my files. I had originally hoped to use only Aftershot Pro, but its keyword manager and tagging system has aroused my suspicions- I'm not sure it will meet my long-term needs, or if its robust enough to allow me to properly tag photos with keywords because my keyword structure will change and adapt over time.

[afx]

Firstly, ASP's keyword manager seems to be very limited.

Calling it limited is an understatement ;-(

It seems that all of the photos that I had tagged using my old keyword tree-structure categories are broken.

I have yet to see a universal tree based keywording scheme that survives the test of time.
Simple flat keywords have many advantages:
They work everywhere, no dependency on interpretation.
There is no structure to worry about when one changes the working or keywording habits.
They are easier to search in simple expressions.

1. Is that work all lost now? Will the keywords in sidecar XMP files that describe each NEF file still be readable/searchable even now that the original keyword tree-structure categories no longer exist?

As long as you have them in the AS sidecars, you can convert them to anything else.
Even better if you also have them in standard sidecars.

2. Do I now have to re-create all of the JPEGS from the original NEF RAW files to get the keywords, and star-rankings re-embedded in the JPEGS?

Probably the easiest way.

3.[/b] Is Aftershot Pro's keyword manager, and overall DAM capabilities capable of maintaining elaborate keyword tree-structure categories for tagging and categorizing my photo collection?

Nope.

4. Why is the Keyword Manager designed this way?

The old Bibblelabs team never had enough resources to really make catalogs work.

1. Will iMatch be able to handle all of this better?

Yes. But be prepared to experiment and throw away your first catalogs and redo them.
You already went through some learning curve about general keywording, so this might actually be easier...

2. As I understand, iMatch, if I create a detailed list of keywords and tag my RAW files (Olympus E1 .ORF), Pentax K-7 .PEF and Nikon D800 .NEF) with keywords, they keywords assigned to each photo is stored in a sidecar XMP metadata file.

Yes

If in the future I do not want to use, or cannot use iMatch, will any other program be able to read these sidecar XMP metadata files?

Yes (if you stick to standard fields).

3. [/b]Is there a way to embed the sidecar XMP metadata files into RAW files as part of the IPTC metadata?

iMatch is one of the few programs I trust to write into my NEFs.
Still there is a corruption risk.

4. I had read that some photographers convert all of their RAW files by different camera manufacturers into Adobe DNG format because a) keyword metadata can be embedded into the DNG file; b) non-destructive edits made to the DNG file in Lightroom can are also embedded in the DNG file.

YUCK. How to lock yourself into the Adobe world....

Can Aftershot embed non-destructive edits made to the DNG file too?

Nope.

I'm trying to manage a big research project, and am trying to find ways to manage and categorize typed notes and transcripts, audio interviews, and photos. My problems with textual and audio data have been resolved.

iMatch allows you to associate text and audio files with images via buddy files. Makes it easier to manage in sync.

Any advice and guidance on how to proceed with my photo collections will be very much appreciated! To reiterate, if I spend hundreds of hou withrs cataloguing, ranking, and tagging photos with a carefully-developed keyword system, I want to make sure I do it correctly from the beginning, using a solid application, and in a way that the tags/keywords can be read by other software at some point in the future.

Experiment with a smaller set initially, a copy at best.
You should be able to to all you need with regards to image and metadata management in iMatch.

I am wondering however, if iMatch can do (or will be able to do) what you are currently doing with PhotoMechanic- "rename and distribution of raw into folders/subfolders, geotagging and basic IPTC". I'm impressed to learn that PhotoMechanic can write all of these to metadata inside the NEF file and inside RW2 (I also use both of these RAW formats!). Can iMatch do that too? If it can, I would have no need to use PhotoMechanic.

PM is the fastest way to keyword. It was designed for Sports Shooters.
But apart form that, I think iMatch can easily do everything that PM can do.

cheers
afx

[Franz Dur]

Thank you for your detailed reply!

Your workflow sounds well-thought out and efficient.
I am wondering however, if iMatch can do (or will be able to do) what you are currently doing with PhotoMechanic- "rename and distribution of raw into folders/subfolders, geotagging and basic IPTC". I'm impressed to learn that PhotoMechanic can write all of these to metadata inside the NEF file and inside RW2 (I also use both of these RAW formats!). Can iMatch do that too? If it can, I would have no need to use PhotoMechanic.

IMatch has an extensive rename feature, but it is not comparable to that of PM (Photomechanic) or then I don't know how to use it. In PM one can setup a preset for each camera (Nikon and Pana have different naming/numbering schemes) and have the NEF or RW2 flow to distinct folders named after session date (or any other Exif tag, or combination of these) i.e. if the card contains shots taken on different days they will go to subfolders on a one per day basis. It is excellent, also very easy for synchronizing the pix with gpx traces, etc.. But it is payware.

In fact there is a little gem in the XnView package (it is a standalone but you have to download the complete pack, not just XnView), it is called XnViewMediaDetector.exe and does much of this (not the gps), quite powerfull and free. It is what I used before buying a Panasonic, the problem is that XnViewMediaDetector cannot display the RW2 vignette, which is handy for a first cull. The GPS part can be handled with Geosetter, also free and very capable. XnViewMediaDetector can aply an IPTC template, don't remeber if it writes to the NEF, but certainly not to the RW2. Not all PM versions can write XMP to the RW2, but they (the PM poeple) promised me to include it as an option in a forthcoming version. The problem is that some Adobe programs don't like RW2 that have been written to. It is safe to stick to external xmp if you are in a mixed program mode. Don't know if IMatch 5 will allow writing to RW2, IMatch 3.6 writes to NEF in a very secure way, I have done it for thousands of D80 and D7000 NEF, no problem at all.

I'm trying to simply the number of tools I must use to process my files. I had originally hoped to use only Aftershot Pro, but its keyword manager and tagging system has aroused my suspicions- I'm not sure it will meet my long-term needs, or if its robust enough to allow me to properly tag photos with keywords because my keyword structure will change and adapt over time.

In my opinion the most important piece of pgm in such a workflow is the cataloguing program. It is much easier to switch from AFP to C1 or LR than to learn again a new DAM which might put different constraints on the way you organize your information tree. On the IMatch web page ther is a IM5 teaser, shows some features of the new version. But already 3.6 does well what I need. The trial is fully functionnal, have a look for yourself.

Francis

[ferdinand-paris]

The OP also asked his questions on the iMatch forum. I am replying there to the bits that relate to iMatch usage. I have never used the ASP keyword tool or DNG.

A lot of iMatch users also use Downloader Pro for ingest / renaming / folders / initial auto metadata entry. I was put off by PM's price, although I concede that one tool may be more attractive than several (Downloader Pro, FastPictureViewer, Geosetter).

F_P

[Franz Dur]

Hello Ferdinand,

Yes, PM is pricy, I was happy with XnViewmediadetector and bought Pm only to handle NEF and RW2 with the same software and in the manner I like.

Hopefully IM5 will allow the kind of workflow I like (everything essential in the image file, raw AND jpg/tif) and I will ditch PM which is overkill for my usage. For a press or sports photographer it makes a lot of sense, very powerful and configurable. And a responsive forum, on a par with the IMatch or the former Bibble 4 forum.

Francis

[benoit]

will allow the kind of workflow I like (everything essential in the image file, raw AND jpg/tif)

Did you tried Rapid Photo Downloader from Damon Lynch ?
It is taking care of all my photo and video files up to sorting in relevant folders, renaming, parallel backup ...
It can also use many tags (camera serial number for example !) as input to create downloading folders and file renaming scheme
Benoit

[Franz Dur]

No, I havn't cause now that I payed for PM I will stick with it unless the future IMatch5 handles ingestion as well as PM5.

Also, I have a dual boot Win7/Ubuntu, but all my photo programs are Windows, ans work well, so a Linux only program would be a hassle in my worflow.

But Rapid Photo Downloader looks fine although it is not evident from the online doc if it is able to read metadata (other than Exif) from the raw and to write metadata (other than Exif) to the raw. Dito for GPS data, I am not sure it cans import gpx files and tag the images.

Francis

[benoit]

Rapid Photo Downloader looks fine although it is not evident from the online doc if it is able to read metadata (other than Exif) from the raw and to write metadata (other than Exif) to the raw. Dito for GPS data, I am not sure it cans import gpx files and tag the images.
Francis

It reads the Exif but as files comes strait from the camera it is enough (I think).
It writes keywords if needed (but selected from a flat list only).
I am not using GPS data or gpx files, I don't think it has the feature.

[coolrat]

Thanks to everyone for sharing experiences! Very informative.

I tested out iMatch- its impressive. iMatch was able to import the XMP sidecar files from Aftershot Pro that contained my keywords and star-ratings.
After reading comments and suggestions, it seems clear that I should be adding XMP data using iMatch, and only use ASP for my RAW editing and adjustments

But I have a new question:
iMatch's ability to create off-line caches of the RAW images on my external harddrives is very useful.
This will be very useful in allowing me to show my collections to others and NOT have to carry my massive external harddrives around!

This, however raises a new question.
My photos are stored in folders like this:

2012_03_March_22_Village NNN Festival [NOTE: contains all the RAW files and xmp sidecar files with edits, and keywords)
.....2012_03_March_22_Village NNN Festival ASP 1800x1200 JPEGS [NOTE: contains the finished high-quality JPEGS created from edited RAW files from the folder above]

When iMatch creates the off-line cache, it scans photos from both folders: from "2012_03_March_22_Village NNN Festival" containing the RAW originals, and from the subfolder "2012_03_March_22_Village NNN Festival ASP 1800x1200 JPEGS".

Slightly perturbing to me is the fact that the off-line catalogue of the RAW images does not show any of the edits/adjustments. Only the finished "JPEGS in 2012_03_March_22_Village NNN Festival ASP 1800x1200 JPEGS" will reveal this.
Having an off-line catalogue that contains duplicate images of the original RAW image, and the finished JPEGS seems a bit counter-intuitive: In a perfect world, I'd like to be able to have the off-line catalogue display the actual edited RAWs and simply not bother displaying the finished JPEGS.

It seems more intuitive and sensible to use one single application that can
1. do RAW edits
2. function as a DAM allowing me to catalogue, use XMP editor to add keywords, stars, geo-data, rankings etc
3. allow the user to have off-line catalogue showing the edited RAWs,
4. allow the user to create different collections that I can show to different people (ie put all the photos of village NN in a one collection for viewing)

Many of you have thought about this far more deeply than I.
Will CaptureOne 7, LightRoom 4 are undoubtedly good RAW editors, but will they be able to provide satisfactory results as a DAM and for creating and showing offline catalogues of my photos?
Or might I be better off using ASP for now, and a DAM like iMatch? Or perhaps PhotoNinja and iMatch?

What I fear most now is that if I continue to use ASP, invest hundreds of hours in editing my RAW .NEF files, and ASP is abandoned, then I'll not have access to the edits 10 years from now unless I use my ancient copy of ASP. It seems unlikely that LightRoom or CaptureOne are going to disappear.

Great suggestions here so far! Thanks!

[afx]

It seems more intuitive and sensible to use one single application that can
1. do RAW edits
2. function as a DAM allowing me to catalogue, use XMP editor to add keywords, stars, geo-data, rankings etc
3. allow the user to have off-line catalogue showing the edited RAWs,

There are two ways to do this:
Use an integrated converter with a toy database like AS or LR or use the manufacturers converter and then switch on that codec in the DAM (works with Nikon NEFs in iMatch).

4. allow the user to create different collections that I can show to different people (ie put all the photos of village NN in a one collection for viewing)

Totally trivial with AS batch queues.
I create a shadow copy of my image tree on a local server with 1600pix JPGs for local viewing.

Will CaptureOne 7, LightRoom 4 are undoubtedly good RAW editors, but will they be able to provide satisfactory results as a DAM and for creating and showing offline catalogues of my photos?
Or might I be better off using ASP for now, and a DAM like iMatch? Or perhaps PhotoNinja and iMatch?

Pick your poison. Nothing is perfect. Personal preferences play a big role here.

What I fear most now is that if I continue to use ASP, invest hundreds of hours in editing my RAW .NEF files, and ASP is abandoned, then I'll not have access to the edits 10 years from now unless I use my ancient copy of ASP. It seems unlikely that LightRoom or CaptureOne are going to disappear.

PhaseOne depends on the sale of their digibacks. Their development team is small ;-(

Apart from that, so you really want to re-edit your images with settings that are ten years old? Why not just generate TIFFs or JPGs now and if you really need to produce a different edit, do it with whatever your current tool is then?
And, old software can be used for ages with a bit of care. I still have Bibble 4 on my Windows 7 box

Also, if a tool advances, it does not mean you can really re-use your old edits. Adobe did the right thing with LR, allowing the user to switch older and newer engine, but it will still need reworking of the images.

cheers
afx

[Franz Dur]

Hello,

On the IMatch forum you may ask Mario if the next version will allow to define buddy files at will.

If yes, you may define the xxxxx.RAW.xmp as buddies, Imatch will then handle them along with RAW, jpg/tiff, xmp. This would cache your edits and versions along with RAW and jpg/tiff.

Of course in ten years ASP may be dead but still functionnal on your computer. From my experience, if I open an eight years old Raw in Bibble/ASP, I am tempted to edit it again and end up with something different from the jpg I produced initially.Taste and perspective change over the years.

Keeping the Raw plus metadata/annotations in a strong database like IMatch is the best bet for the future, better in my view than going the DNG route.

Francis

[coolrat]

Thanks AFX and Francis for your replies

Good point about perhaps that in 10 years, photo editing technologies would be so different from now that there would be no need at all to even consider my current edits.
I guess the only think I really don't want to lose is the keywords (in XMP sidecar files) that I will assign to each photo.

Something I did not seem to understand however from your posts.

Afx, you mentioned,

"manufacturers converter and then switch on that codec in the DAM (works with Nikon NEFs in iMatch)"

Do you mean that I could use the Nikon editor, make my adjustments in that, and that iMatch would be able to read and display the adjustments? Can iMatch can read the adjustments made by Nikon's editor, but not by any other 3rd party editor? If that is the case, the perhaps using a Nikon proprietary editor is my best solution?

And Francis, when you wrote,

ask Mario if the next version will allow to define buddy files at will.
If yes, you may define the xxxxx.RAW.xmp as buddies, Imatch will then handle them along with RAW, jpg/tiff, xmp. This would cache your edits and versions along with RAW and jpg/tiff.

what are the xxxxx.RAW.xmp you are referring to? The ones created by ASP or by Nikon's proprietary converter? Could you please explain this a little more?

Thanks for your advice. I'm beginning to get a much clearer idea of the direction I should take.

[Franz Dur]

Hello,

Concerning AFX' comment: if you edit your NEF with Cnx or ViewXn the edits will be written into the NEF (along a full size standard quality jpg - that's why the NEF weight grows substantially after editing). The Nikon codec can read these edits and interpret the raw, and IMatch can be set to use this codec.

Concerning ASP: your edits (and versions etc) are saved by ASP in a xxx.NEF.xmp buddy file. The xmp is a bad idea because these are not "classical" xmp files, but anyway they are all that is needed for ASP to re-convert your NEF. When reading the imatch5_betateaser_2012.pdf document I had the impression that IM5 will extend the concept of buddy files (read the part on the File Relation configuration feature). Il it allows to declare the xxx.NEF.xmp as bona fide buddy files, Imatch 5 will probably handle them just as it will handle NEF and jpg/tiff ensembles, they should be cached, copied, deleted. This is my hope and personal interpretation, maybe Mario could say a word about this on the forum if you ask.

Francis