I was resizing my hard drive last night and was not successful.
My system and drives are local on an ESXi 5.1 vm.
I'm running Debian 6 x64 and have a 2TB mount that I was resizing. It was about 1.8T that I was resizing to the full 2TB.
I ran e2fsck on /dev/sdc1 and it had no issues. I than ran resize2fs and let it run for a few hours. After checking ps, it showed the state in sleep (D+) with no activity. I couldn't kill the process so I rebooted and now I cannot do anything with the drive.
Now I'm missing the /dev/sdc1 drive and I cannot do anything that will read that drive; fdisk, cfdisk, e2fsck, resize2fs all freeze attempting to do anything related to this drive.
What can I do to recover this drive? I unfortunately do not have a backup of this drive. I just want to be able to mount the drive to recover my data or at least read the data on the drive somehow.
dmesg output
[ 722.816205] sd 2:0:2:0: timing out command, waited 180s
[ 722.816329] sd 2:0:2:0: [sdc] Unhandled error code
[ 722.816336] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 722.816343] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 722.816352] end_request: I/O error, dev sdc, sector 0
[ 722.816443] Buffer I/O error on device sdc, logical block 0
[ 722.816534] Buffer I/O error on device sdc, logical block 1
[ 722.816619] Buffer I/O error on device sdc, logical block 2
[ 722.816703] Buffer I/O error on device sdc, logical block 3
[ 903.215677] sd 2:0:2:0: timing out command, waited 180s
[ 903.215802] sd 2:0:2:0: [sdc] Unhandled error code
[ 903.215805] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 903.215808] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 903.215817] end_request: I/O error, dev sdc, sector 0
[ 903.215943] Buffer I/O error on device sdc, logical block 0
[ 903.216037] Buffer I/O error on device sdc, logical block 1
[ 903.216124] Buffer I/O error on device sdc, logical block 2
[ 903.216211] Buffer I/O error on device sdc, logical block 3
[ 903.216337] ldm_validate_partition_table(): Disk read failed.
[ 1083.615097] sd 2:0:2:0: timing out command, waited 180s
[ 1083.615216] sd 2:0:2:0: [sdc] Unhandled error code
[ 1083.615218] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1083.615222] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1083.615231] end_request: I/O error, dev sdc, sector 0
[ 1083.615316] Buffer I/O error on device sdc, logical block 0
[ 1083.615411] Buffer I/O error on device sdc, logical block 1
[ 1083.615496] Buffer I/O error on device sdc, logical block 2
[ 1083.615580] Buffer I/O error on device sdc, logical block 3
[ 1264.014510] sd 2:0:2:0: timing out command, waited 180s
[ 1264.014667] sd 2:0:2:0: [sdc] Unhandled error code
[ 1264.014669] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1264.014672] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1264.014680] end_request: I/O error, dev sdc, sector 0
[ 1264.014766] Buffer I/O error on device sdc, logical block 0
[ 1264.014855] Buffer I/O error on device sdc, logical block 1
[ 1264.014939] Buffer I/O error on device sdc, logical block 2
[ 1264.015022] Buffer I/O error on device sdc, logical block 3
[ 1444.413968] sd 2:0:2:0: timing out command, waited 180s
[ 1444.414098] sd 2:0:2:0: [sdc] Unhandled error code
[ 1444.414105] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1444.414113] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1444.414124] end_request: I/O error, dev sdc, sector 0
[ 1444.414246] Buffer I/O error on device sdc, logical block 0
[ 1444.414363] Buffer I/O error on device sdc, logical block 1
[ 1444.414450] Buffer I/O error on device sdc, logical block 2
[ 1444.414536] Buffer I/O error on device sdc, logical block 3
[ 1624.813488] sd 2:0:2:0: timing out command, waited 180s
[ 1624.813596] sd 2:0:2:0: [sdc] Unhandled error code
[ 1624.813603] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1624.813611] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1624.813620] end_request: I/O error, dev sdc, sector 0
[ 1624.813699] Buffer I/O error on device sdc, logical block 0
[ 1624.813786] Buffer I/O error on device sdc, logical block 1
[ 1624.813870] Buffer I/O error on device sdc, logical block 2
[ 1624.813953] Buffer I/O error on device sdc, logical block 3
[ 1624.814071] Dev sdc: unable to read RDB block 0
[ 1805.212836] sd 2:0:2:0: timing out command, waited 180s
[ 1805.212951] sd 2:0:2:0: [sdc] Unhandled error code
[ 1805.212953] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1805.212956] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1805.212963] end_request: I/O error, dev sdc, sector 0
[ 1805.213052] Buffer I/O error on device sdc, logical block 0
[ 1805.213143] Buffer I/O error on device sdc, logical block 1
[ 1805.213230] Buffer I/O error on device sdc, logical block 2
[ 1805.213317] Buffer I/O error on device sdc, logical block 3
[ 1985.612363] sd 2:0:2:0: timing out command, waited 180s
[ 1985.612556] sd 2:0:2:0: [sdc] Unhandled error code
[ 1985.612558] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1985.612561] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1985.612568] end_request: I/O error, dev sdc, sector 0
[ 1985.612691] Buffer I/O error on device sdc, logical block 0
[ 1985.612785] Buffer I/O error on device sdc, logical block 1
[ 1985.612878] Buffer I/O error on device sdc, logical block 2
[ 1985.612966] Buffer I/O error on device sdc, logical block 3
[ 2166.011692] sd 2:0:2:0: timing out command, waited 180s
[ 2166.011806] sd 2:0:2:0: [sdc] Unhandled error code
[ 2166.011808] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2166.011811] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2166.011818] end_request: I/O error, dev sdc, sector 24
[ 2166.011899] Buffer I/O error on device sdc, logical block 12
[ 2166.011998] Buffer I/O error on device sdc, logical block 13
[ 2166.012083] Buffer I/O error on device sdc, logical block 14
[ 2166.012168] Buffer I/O error on device sdc, logical block 15
[ 2346.411194] sd 2:0:2:0: timing out command, waited 180s
[ 2346.411377] sd 2:0:2:0: [sdc] Unhandled error code
[ 2346.411384] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2346.411392] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2346.411403] end_request: I/O error, dev sdc, sector 24
[ 2346.411505] Buffer I/O error on device sdc, logical block 12
[ 2346.411628] Buffer I/O error on device sdc, logical block 13
[ 2346.411730] Buffer I/O error on device sdc, logical block 14
[ 2346.411829] Buffer I/O error on device sdc, logical block 15
[ 2526.810657] sd 2:0:2:0: timing out command, waited 180s
[ 2526.810824] sd 2:0:2:0: [sdc] Unhandled error code
[ 2526.810827] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2526.810831] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2526.810841] end_request: I/O error, dev sdc, sector 0
[ 2526.810978] Buffer I/O error on device sdc, logical block 0
[ 2526.811096] Buffer I/O error on device sdc, logical block 1
[ 2526.811184] Buffer I/O error on device sdc, logical block 2
[ 2526.811271] Buffer I/O error on device sdc, logical block 3
[ 2526.811391] unable to read partition table
[ 2526.811899] sd 2:0:2:0: [sdc] Cache data unavailable
[ 2526.811901] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[ 2526.811998] sd 2:0:2:0: [sdc] Attached SCSI disk
[ 2706.830303] VMware vmxnet3 virtual NIC driver - version 1.0.5.0-k-NAPI
[ 2706.833247] VMware PVSCSI driver - version 1.0.1.0-k
[ 2706.836002] VMware vmxnet virtual NIC driver
[ 2706.845581] PM: Starting manual resume from disk
[ 2706.845584] PM: Resume from partition 8:5
[ 2706.845586] PM: Checking hibernation image.
[ 2706.845921] PM: Error -22 checking image file
[ 2706.845923] PM: Resume from disk failed.
[ 2706.858105] kjournald starting. Commit interval 5 seconds
[ 2706.858117] EXT3-fs: mounted filesystem with ordered data mode.
[ 2707.043045] udev[390]: starting version 164
[ 2707.070265] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[ 2707.070276] ACPI: Power Button [PWRF]
[ 2707.075278] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 2707.077419] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 2707.093012] [415]: VMCI: shared components initialized.
[ 2707.093068] Probing for vmci/PCI.
[ 2707.093106] alloc irq_desc for 16 on node -1
[ 2707.093107] alloc kstat_irqs on node -1
[ 2707.093117] vmci 0000:00:07.7: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 2707.093196] Found vmci/PCI at 0x1080, irq 16.
[ 2707.093226] VMCI: using capabilities 0xc.
[ 2707.093327] [415]: VMCI: Host capability check passed.
[ 2707.093471] alloc irq_desc for 56 on node -1
[ 2707.093472] alloc kstat_irqs on node -1
[ 2707.093524] vmci 0000:00:07.7: irq 56 for MSI/MSI-X
[ 2707.093525] alloc irq_desc for 57 on node -1
[ 2707.093526] alloc kstat_irqs on node -1
[ 2707.093546] vmci 0000:00:07.7: irq 57 for MSI/MSI-X
[ 2707.093717] Registered vmci device.
[ 2707.093760] [415]: VMCI: Using guest personality
[ 2707.093762] [415]: VMCI: host components initialized.
[ 2707.093785] [415]: VMCI: Module registered (name=vmci,major=10,minor=59).
[ 2707.093786] [415]: VMCI: Using host personality
[ 2707.093787] [415]: VMCI: Module (name=vmci) is initialized
[ 2707.102777] processor LNXCPU:00: registered as cooling_device0
[ 2707.103353] processor LNXCPU:01: registered as cooling_device1
[ 2707.103522] piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!
[ 2707.104218] processor LNXCPU:02: registered as cooling_device2
[ 2707.105603] processor LNXCPU:03: registered as cooling_device3
[ 2707.107367] input: PC Speaker as /devices/platform/pcspkr/input/input3
[ 2707.117231] ACPI: AC Adapter [ACAD] (on-line)
[ 2707.148586] parport_pc 00:09: reported by Plug and Play ACPI
[ 2707.148765] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[ 2707.193924] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[ 2707.199564] Error: Driver 'pcspkr' is already registered, aborting...
[ 2707.209963] sd 2:0:2:0: timing out command, waited 180s
[ 2707.210018] sd 2:0:2:0: [sdc] Unhandled error code
[ 2707.210020] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2707.210022] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2707.210029] end_request: I/O error, dev sdc, sector 0
[ 2707.210077] Buffer I/O error on device sdc, logical block 0
[ 2707.210129] Buffer I/O error on device sdc, logical block 1
[ 2707.210179] Buffer I/O error on device sdc, logical block 2
[ 2707.210230] Buffer I/O error on device sdc, logical block 3
[ 3787.606904] sd 2:0:2:0: [sdc] Unhandled error code
[ 3787.606907] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 3787.606910] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 3787.606921] end_request: I/O error, dev sdc, sector 0
[ 3787.607032] Buffer I/O error on device sdc, logical block 0
[ 3787.607145] Buffer I/O error on device sdc, logical block 1
[ 3787.607256] Buffer I/O error on device sdc, logical block 2
[ 3787.607367] Buffer I/O error on device sdc, logical block 3
[ 4868.003267] sd 2:0:2:0: timing out command, waited 1080s
[ 4868.003392] sd 2:0:2:0: [sdc] Unhandled error code
[ 4868.003394] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 4868.003397] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 4868.003407] end_request: I/O error, dev sdc, sector 0
[ 4868.003495] Buffer I/O error on device sdc, logical block 0
[ 4868.003595] Buffer I/O error on device sdc, logical block 1
[ 4868.003685] Buffer I/O error on device sdc, logical block 2
[ 4868.003774] Buffer I/O error on device sdc, logical block 3
[ 5948.399889] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400005] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400012] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400019] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 5948.400028] end_request: I/O error, dev sdc, sector 0
[ 5948.400110] Buffer I/O error on device sdc, logical block 0
[ 5948.400209] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400294] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400295] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400298] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 5948.400304] end_request: I/O error, dev sdc, sector 2
[ 5948.400384] Buffer I/O error on device sdc, logical block 1
[ 5948.400473] Buffer I/O error on device sdc, logical block 2
[ 5948.400562] Buffer I/O error on device sdc, logical block 3
[ 7028.796518] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796633] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796640] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796648] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 7028.796657] end_request: I/O error, dev sdc, sector 0
[ 7028.796746] Buffer I/O error on device sdc, logical block 0
[ 7028.796864] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796948] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796949] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796952] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 7028.796958] end_request: I/O error, dev sdc, sector 2
[ 7028.797037] Buffer I/O error on device sdc, logical block 1
[ 7028.797122] Buffer I/O error on device sdc, logical block 2
[ 7028.797207] Buffer I/O error on device sdc, logical block 3
Best Answer
First,
resize2fsmay take several hours or even more than a day to run, depending on the size of your partition.If you interrupted it while it was running, that's a very good recipe for losing all your data. All of the filesystem data structures are probably in an inconsistent state.
Now, your drive has errors at the very beginning of the disk, where the partition table is stored.
At this point, what you should do is replace the disk and restore your backups. Since you don't have backups, you should update your CV. If this server was very important, the company might go out of business.
To avoid situations like this in future, always ensure that you have backups and that your backups are fully functional, that you can successfully restore with them.
So, aside from updating your CV and making backups, this is what you need to do today:
You can attempt to use
dd_rescueto copy the disk, skipping any errors, to another disk, and then work on rescuing data from the copy.Don't expect to get too much out of this, though.
If you can't rescue anything from the disk, send it to a professional data recovery company.