Where are the photos stored? Are you embedding XMP or using XMP sidecars?
-Kirk
Network server. The RAW conversion creates XMP sidecars. Just double checked: yes the timelag happens only with these files that consist as a triplet of NEF, JPG and XMP.
Setting a colour class (or whatever other metadata) has PM update the file. This can be slow over a network, especially if the files are big. In case of raw files, you can have PM just update the xmp file which is small and updates to it should be fairly quick (please double check your IPTC/XMP preferences that you are really not embedding IPTC/XMP into raw files). JPG files on the other hand will have to be gotten off the server (if not already in cache), updated, and written back. Depending on the size and network speed, this can take some time. D800 files (even just a jpg) are huge, so things are worsened.
The extend to which you are seeing lags (multiple minutes), leads me to suspect something else is at play here too. Does the contact sheet fill up pretty quick? If so, then the problem seems to be focussed on writing. Perhaps there is a problem with your server? How long does it take to copy a new file of several MBs (e.g., the size of a D800 file) to the server (in the same directory)?
Hope we can get a bit closer to a solution,
Hayo
