Abstract: This article discusses how I recovered data from Buffalo Drive Station Quattro (Model HD-Q2.0TSU2/R5).
The unit was working normally, then after a power cycle it no longer powered up. The unit was dead iron. After a power cycle, the array appeared to be turned off, no fan, no LEDs, no signs of life.
Considering the RAID array was functioning perfectly before this event, I had a high degree of confidence the data on the disc array was intact, merely unaccessable because of a hardware problem.
This recovery mechanism is non-distructive. Copies of all original data discs are first made, then the analysis and recovery is made from those copied disc images. This is safe (assuming no hardware disc failure exists in the array being recovered), but requires a lot of working scratch space.
This page contains a DIY RAID-5 executive summary of a Drive Station Quattro, for those who don't want to read my detailed recovery steps.
The Drive Station Quattro (referred to as RAID array, or just array in this article) was connected to an HP Pavilion workstation running Fedora Core 8, using the USB interface. Although slower, the USB interface allowed the workstation to boot cleanly; using the eSATA interface the S.M.A.R.T feature of the BIOS incorrectly warned the disk was failing. The array was configured RAID-5, formatted as an NTFS filesystem.
The Buffalo Drive Station Quattro (Model HD-Q2.0TSU2/R5) RAID-5 has the following parameters:
Diagnosis and Evaluation
The first recovery attempt was to verify the problem was not a failed power supply. I opened up the unit, and with the array powered up, measured the output from the power supply with a volt meter. +5v and +12v was present on the power connector, so this did not appear to be the problem.
My next step was to call Buffalo Technologies Support. Ultimately I was completely dissatisfied with their ability to help me. The first support technician issued me an RMA, but I never received the RMA confirmation by email. After numerous phone calls, I finally got the URL to the RMA site by phone. They also admitted having problems with their email system over the week (3/2-2/5 2010 I believe).
The RMA form wanted to know the vendor of the hardware and the date of purchase. They only issue RMAs and support their product for 1 year, something the first technician failed to mention. My unit being nearly 3 years old was out of warranty.
Given this revelation, I decided to look for a replacement Drive Station. I found a similar unit, Model HD-QS2.0TSU2/R5. I called Buffalo Support, specifically asking if this unit would work as a replacement chassis. They said the only difference was the firmware supported Turbo USB 2.0, otherwise they were the same, and I should be able to swap drives and be back on-line.
Given this information, I purchased the Model HD-QS2.0TSU2/R5. The unit was brand new, still with the Buffalo factory seal on the original Buffalo box. The first thing I did was pop the front cover (no screw driver required) to look at the disc connections. Immediately I knew this was not going to work. The HD-Q2.0TSU2/R5 (the broken array) has 4-500GB discs with EIDE interfaces. The HD-QS2.0TSU2/R5 (the replacement array) had 4-500GB discs with SATA interfaces. The information given to be by Buffalo support was just wrong.
I called Buffalo again to resolve this question. Apparently they don't log all customer contact calls. The last technician I spoke with, Brian A. (3/17/2010) had no idea why I was told this unit was compatible. Additionally, I was told swapping the chassis with a unit of identical model number would not work, all arrays are formatted differently. This sounded bogus to me, so I figured I was wasting my time with Buffalo support.
My options for data recovery were:
Using a commercial data recover service was cost prohibitive. The data on the array was not important enough to warrant the money.
Looking for RAID data recovery software, the best one appeared to be RAID Reconstructor (v4.02). This software analyzes the array parameters, recommends the best configuration probed, and then allows rebuilding the array from the data on disc. The evaluation copy will only analyze the array parmeters, but not recover data. You must buy the $99 full version for data recovery. Not a bad deal if it works and saves you time.
The analysis results from RAID Reconstructor were inconclusive. Reliably it showed my array disc order was 3-4-1-2. However, it could not figure out the stripe size and the parity ordering. I tried their recommended parameters first, which were inconclusive. I then tried stripe widths of 1/2/4/8 sectors. The software hung when it tried to perform the 1 sector stripe analysis. I wrote to to runtime.org support (Authors of RAID Reconstructor). They said 1 sector stripe widths are uncommon, and were not suprised this did not work. Bug?
Not having a backup (bad, bad, bad) of the array, my only option left was figure out how to recover the data myself.
My first steps in recovery involved:
Using an unused spare HP computer, booting with the Knoppix 6.2 Live CD, I proceeded to create disc image files from each of the 4 500GB discs onto the 1TB/2TB scratch space drives. I accomplished this by doing:
dd if=/dev/sda bs=1M > /mnt/space/buffalo/disc1.imgI repeated this operation for each of the 500GB discs, creating 4 500GB disc image files.
The next step was to accurately determine the RAID parameters. I had some hints from RAID Reconstructor, which I started from. Starting with disc 1, I looked for the NTFS partition header:
dd if=/mnt/space/buffalo/disk1.img count=10000 | strings | lessI found the string NTFS , so I knew the partition was on disc 1. Next I started a binary search to find exactly which sector contained the partition.
dd if=/mnt/space/buffalo/disk1.img count=4 skip=1000 | strings | less dd if=/mnt/space/buffalo/disk1.img count=4 skip=500 | strings | less dd if=/mnt/space/buffalo/disk1.img count=4 skip=250| strings | less dd if=/mnt/space/buffalo/disk1.img count=4 skip=125| strings | less dd if=/mnt/space/buffalo/disk1.img count=4 skip=125| strings | less (nothing found yet) dd if=/mnt/space/buffalo/disk1.img count=4 skip=50| strings | less (found) dd if=/mnt/space/buffalo/disk1.img count=4 skip=75| strings | less (not found) dd if=/mnt/space/buffalo/disk1.img count=4 skip=60| strings | less (found) dd if=/mnt/space/buffalo/disk1.img count=4 skip=70| strings | less (not found) dd if=/mnt/space/buffalo/disk1.img count=4 skip=65| strings | less (not found) dd if=/mnt/space/buffalo/disk1.img count=4 skip=62| strings | less (found) dd if=/mnt/space/buffalo/disk1.img count=4 skip=64| strings | less (not found) dd if=/mnt/space/buffalo/disk1.img count=4 skip=63| strings | less (found)The next step was to figure out the stripe width, and parity format. My idea here was to look for some "known data" on the drive, and use that as a "Rosetta Stone". I found a README.txt file for building Perl on AIX, by scanning a disk for strings. The file was "jumbled" when looking at just one disc, to be expected as this is a RAID-5 array. Knowing the offset into the first disc where this file existed, I then started looking around the other disc images at the same offset for the file. I found the other parts of the README.txt. Googling for some of the text was also useful. I found the file (not the same version, but close enough) here:
http://theory.uwinnipeg.ca/CPAN/perl/pod/perlaix.htmlUsing this file as a guide, I was able to determine the RAID parameters.
Unstriping the Array
The performance of the prototype unstriping script was very poor, so I knew I had to write a program to perform the unstriping. This simple C program is what I used to fully unstripe the RAID array. For reasons I don't fully understand (write I/O bound?) the data transfer was about 5.5MB/s. For a 1.5TB array, this took about 3 1/2 days to unstripe.
After the unstriping was complete, I was left with an output file that was about 1.5TB, diskimage.ntfs. I then attempted to mount this file as an NTFS file system:
mount -t ntfs -o loop,ro,offset=32256 /mnt/sdb1/diskimage.ntfs /mnt/ntfsIt mounted cleanly! I was able to look around the file system, and everything looked to be there.
Fortunately, I had some very large backup drive image files from laptop computers stored on the now recovered array. I also had MD5 checksums for these image files. I used those backups to verify the integrity of the restore.
cat dell_laptop_hd_backup.dd.gz | gzip -dc | md5sumThe resultant checksum matched the value saved in the checksum file. I did this verification for 3 other files, and they all were correct. These images were disc image files for 60GB hard drives. Given that about 180GB of data checked out, the recovery was a success. There were also some AVI files and MP3 files that played flawlessly, which also helped to prove the recovery was valid.
Backing up the data
Because the recovered array was stored as a disk image file, I needed to copy that data out the disc image.
rsync --progress -axv /mnt/ntfs /mnt/space2tbI connected all drives by SATA interfaces for this copy. The average transfer rate was about 45MB/s, so this completed in about 7 hours.
The final step in this recovery was to reformat the replacement Buffalo from RAID-0 to RAID-5, then copy the recovered data back onto the new array.
rsync --progress -axv /mnt/space2tb /media/HD-QS2.0
My recovery was easy, as the data on the array was not damaged, I suffered hardware failure. As a result, a simple-minded program to unstripe the array was sufficient.
Being a developer with a curious mind, I'm planning on fleshing out the unstripe program to utilize the parity data. That way, data being unstriped can also be validated as good. Additionally, damaged sectors could be repaired, at least in theory. That will be an experimental side project.
RAID-5 is not a substitute for data backup. Backup your data, even if it is an enormous data set!
I hope this information is useful to someone else, or serves as a starting point for how to perform a DIY RAID-5 recovery.
RAID Recovery Guide RAID Reconstructor 4.0 DIY RAID Array unstriping toolsCopyright (C) Adam Bernstein. All Rights Reserved