Author Topic: Massive data loss during ingest  (Read 3373 times)

Offline vAfotoriporter

  • Uber Member
  • ******
  • Posts: 1041
    • View Profile
    • Attila Volgyi photojournalist
Massive data loss during ingest
« on: October 24, 2010, 05:46:12 PM »
I experienced a massive data loss during ingest from an external HDD. There were a lot of photos copied onto the USB HDD I started to ingest in PM on a Win7 netbook and left it for a while to do it. Unfortunately Win7 decided to restart due to a system update. (I can't even imagine why Windows lets itself to restart while there is a copy of tens of gigabytes in progress)
The result is about 2600 RAW photos unreadable with zero byte file size.

The actual loss and the accident with the reset during ingest may not be connected but I checked all possibilites (status of hard drives, status of cards, etc) and cannot really imagine any other explanation. The same workflow repeated with the same cards, same computers, same harddrives and no other data loss just from this very first occasion interrupted by the restart.

Any idea how or why an unexpected system restart could cause such a data loss?
Working on Mac, OSX, iOS and with some Canons.
Allways shooting RAW.

http://www.volgyiattila.com

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24756
    • View Profile
    • Camera Bits, Inc.
Re: Massive data loss during ingest
« Reply #1 on: October 24, 2010, 06:05:04 PM »
I experienced a massive data loss during ingest from an external HDD. There were a lot of photos copied onto the USB HDD I started to ingest in PM on a Win7 netbook and left it for a while to do it. Unfortunately Win7 decided to restart due to a system update. (I can't even imagine why Windows lets itself to restart while there is a copy of tens of gigabytes in progress)
The result is about 2600 RAW photos unreadable with zero byte file size.

The actual loss and the accident with the reset during ingest may not be connected but I checked all possibilites (status of hard drives, status of cards, etc) and cannot really imagine any other explanation. The same workflow repeated with the same cards, same computers, same harddrives and no other data loss just from this very first occasion interrupted by the restart.

Any idea how or why an unexpected system restart could cause such a data loss?

Sure.  Windows (and other OSes) don't always write all data written to a file until the OS decides to 'flush' the data to the hard drive.  The OSes do this for reasons of performance but the trade off is that if the OS crashes or is hard reset then data loss can occur.

-Kirk

Offline Woodie

  • Full Member
  • ***
  • Posts: 147
    • View Profile
Re: Massive data loss during ingest
« Reply #2 on: October 25, 2010, 08:30:02 AM »
Kirk,

If the images were just being ingested from the external hard drive, i.e., no writing to the external drive, would they still be corrupted or would Erase Source Disk(s) need to be checked?  My Win is set to notify, not automatically download so I shouldn't run into the problem described, but other interruptions could happen.  Would the files being ingested become corrupted if set to Read Only?

(Sorry this is a little off topic as far as PM goes but does affect Ingest)

Woodie

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24756
    • View Profile
    • Camera Bits, Inc.
Re: Massive data loss during ingest
« Reply #3 on: October 25, 2010, 08:47:18 AM »
Woodie,

If the images were just being ingested from the external hard drive, i.e., no writing to the external drive, would they still be corrupted or would Erase Source Disk(s) need to be checked?  My Win is set to notify, not automatically download so I shouldn't run into the problem described, but other interruptions could happen.  Would the files being ingested become corrupted if set to Read Only?

The source images should never be written to and thus they could never be damaged even if a spontaneous reboot occurred.  Adding Erase Source Disk(s) would cause the source images to be deleted after they are Ingested, but only if no errors occurred during the Ingest process.

-Kirk