How a Corrupted USB Drive Was Saved by GNU/Linux

 in
Would SUSE and fsck be able to recover the data in a usable way?
Repair Attempt #2

To summarize exactly what fixed the USB device:

  • Step 1: create a filesystem image of the right size, with FATs and the directory in the right places:


    # dd if=/dev/zero of=/tmp/r2x bs=512 count=1001952 
    # losetup /dev/loop2 /tmp/r2x
    # mkfs.msdos -n mkfs__msdos -s 16  -R 64 /dev/loop2

  • Step 2: copy bytes from the corrupt image, except the boot sector, onto the filesystem image created in step 1:


    # dd if=r1 of=r2x bs=512 skip=1 seek=1

  • Step 3: execute filesystem repair on that image:


    # fsck.msdos -f -r /dev/loop2

Because I knew that FAT1 was bogus, I told it to use FAT2, and it reported success. It asked me whether to write the changes, and I said yes.

The filesystem images in /tmp/r2x and /dev/loop2 now were consistent. The acid test was to try to mount the filesystem:


    # mkdir /tmp/r2d
    # mount -t vfat /dev/loop2 /tmp/r2d
    # ls -lRA /tmp/r2d

After which all kinds of good stuff appeared.

Note: A good result to ls -lR showed that I was lucky in one other way: I didn't know if the boot sector had a good value for the size of the root directory, the -r parameter to mkfs.msdos. I simply used the default and it turned out fine.

Burning CDs

At this point, I decided I had better burn a CD. I burn and read CDs all the time on Linux, but I rarely burn CDs to be read by Windows. Again I did a Web search, and a page from IBM's DeveloperWorks site turned up. I had searched "linux burn CD windows" or something like that. So I tried this:


    # mkisofs -J -r -v /tmp/r2d | \
          cdrecord -v -pad -eject fs=4m speed=4 dev=0,0,0 -

I wasn't 100% sure that Windows would like this CD, but fortunately I have Windows95 under Win4Lin. Its sole purpose for me is to run Quicken and TurboTax, but I fired it up and pointed Windows Explorer at the just-burned CD-ROM. Explorer loved it. I used gimp(1) to capture a screenshot and e-mailed the image to my friend's brother--he was ecstatic.

APPENDIX: The Bash Script Explained

Shell jockeys need not read this.


  1 #!/bin/bash
  2 # parameters added to mkfs.msdos....
  3 ARGS="$*"
  4 if mount | grep /tmp/r2d; then umount /tmp/r2d; fi
  5 losetup -d /dev/loop2 
  6 losetup /dev/loop2 /tmp/r2x
  7 mkfs.msdos -n mkfs__msdos -s 16 $ARGS /dev/loop2
  8 mount -t vfat /dev/loop2 /tmp/r2d
  9 yes hello | dd bs=8192 count=3 of=/tmp/r2d/foo.txt
 10 umount /tmp/r2d

Line 1 identifies to exec(2) that this is supposed to be run by the shell. I've become accustomed to bash, the Bourne again shell.

Line 2 simply explains line 3, that the parameters you type after b.sh are parameters to add to the mkfs.msdos command line.

Lines 4-6 establish /dev/loop2 as the block device whose contents are in the filesystem image kept in /dev/r2x. Line 4 unmounts the artificial filesystem if it was mounted; this is done because we're about to make some changes to it. Lines 5-6 make sure that /dev/loop2 is connected to /tmp/r2x and only to /tmp/r2x.

Line 7 creates an artificial filesystem image with whatever additional parameters the user gave--remember $ARGS from line 3?.

Line 8 mounts the filesystem onto /tmp/r2d. Line 9 creates a file of about 24KB (three clusters), so I have a filename to look for at the beginning of the directory.

Line 10 then unmounts the artificial filesystem image, so the kernel does not think there are inconsistencies if I play with /tmp/r2x.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

How about multiple partitions?

Subh's picture

Collin,

It was an enlightening article for me. However, I have a slightly different problem. I have a USB disk (1GB) which when I fsck.vfat - shows the following:

dosfsck 3.0.3, 18 May 2009, FAT32, LFN
/dev/sdb1: 379 files, 55267/62958 clusters

I cannot see the files on the drive. Moreover, when I do:

$sudo sfdisk -l -f /dev/sdb1

Disk /dev/sdb1: 1015 cylinders, 32 heads, 62 sectors/track
Units = cylinders of 1015808 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System
/dev/sdb1p1 ? 392205+ 967563- 575358- 570754815+ 72 Unknown
start: (c,h,s) expected (1023,31,62) found (357,116,40)
end: (c,h,s) expected (1023,31,62) found (357,32,45)
/dev/sdb1p2 ? 85024+ 1060845- 975821- 968014120 65 Novell Netware 386
start: (c,h,s) expected (1023,31,62) found (288,115,43)
end: (c,h,s) expected (1023,31,62) found (367,114,50)
/dev/sdb1p3 ? 942480+ 1918301- 975821- 968014096 79 Unknown
start: (c,h,s) expected (1023,31,62) found (366,32,33)
end: (c,h,s) expected (1023,31,62) found (357,32,43)
/dev/sdb1p4 ? 0 1833279- 1833280- 1818613248 d Unknown
start: (c,h,s) expected (0,0,1) found (372,97,50)
end: (c,h,s) expected (1023,31,62) found (0,10,0)

I tried mounting each of the separate partitions by changing the partition ID to VFAT, but it still didn't work. I dont know if data can still be recovered from it. What do you suggest?
Regards,
Subh

Thanks

jo-fluff's picture

I was having issues with a flash drive not being detected just now. Wouldn't mount and fsck.msdos gave a funny error. following your steps to view the partition table and with a bit of help from google, the active flag was set to 81H. So I used fdisk to toggle it and now it works. :)

Flash drive recovery

ram's picture

I brought a 49GB ameco flash drive. after using of 2 months, by mistake I formated it in windows using NTFS. Then it goes to RAW file system. It is showing in windows but for formating is not supporting in FAT/FAT32. Then i tryied in Linux, but the drive is not showing. One of my friend said that, it having problem with '0" sector problem. what can I do?

LINUCKS SAVES TEH DAY!!1

Anonymous's picture

LINUCKS SAVES TEH DAY!!1

Who ever made the comment

Anonymous's picture

Who ever made the comment above is a pr*** yes linux did save the day and your an asshole.

stoped at the first step

katu's picture

Hi, I tried to follow your instructions to recovery data froma a camera, but when i try to make dd if=/dev/sda of=file from a live cd of kubuntu, my computer doesn´t work. any help? thanx a lot

This Makes sense, I've had a similar experience with Linux

MAPWORLD's picture

This makes sense. I once had a friend who had a Microsoft Word document on his flash drive which was corrupted on Windows. I stuck the flash drive in my laptop run Ubuntu v 6.06 (Freeware/shareware Linux) and opened the file up without any problems whatsoever. It just worked.

Hey iv lost all my work on my

TY's picture

Hey iv lost all my work on my lexar usb. Very important work. how do i recover all of it. please anyone help???

FAT was (probably) not corrupted

Helmut from Germany's picture

The many FF FF entries in the beginnning of the FAT were probably correct:

In FAT systems, unfragmenting utilities usually deploy directories at the beginning of the partition. If there are not too many entries in a directory, the 32-bit entries in the directories all fit in one block (e.g. in a system with 8k-blocks, there are 256 entries per block), so the directory "file" has the lenght of one block. It is marked in the fat with FFF (FAT12), FF FF (FAT16) or FF FF FF FF (FAT32).

So I would expect to see many FF bytes in the start of a FAT. Did you ever look whether the FAT contained FF bytes at the beginning or not?

The boot sector of a fat disk partition (not the MBR) contains drive parameters , if the FAT boot sector is damaged, you will have some problems. I suppose windows damaged that sector due to some bug, not the FAT. Writing there, e.g the information that there are about 165 fats ;-)

The disk parameters consist of head/cylinder/sector information (which can safely be ignored (and is ignored, I guess, even by Windows) since it is purely redundant, the relevant data being in the MBR, plus some routine information (e.g. bytes per sector, which is virtually alway 512), plus vital information as number reserved sectors at the start of the partition, which has to be known to calculate the start of the first FAT. Whether the information about the number of FATs is vital depends on the probability that someone created a FAT system with only one FAT ...

Collin - Thank you, thank you, thank you

Craig's picture

Collin - Thank you for posting this article! Yesterday, I accidently shut down my Windows XP computer without ejecting my Lexar Jumpdrive. Afterward, I got the "Drive not Formatted" error. Fortunately I found your article, and have a second computer running Redhat. Following essentially the same precedure you described, I was able to recover all the data from my disabled Jumpdrive. You saved the day. Thanks again!!

512MB USB corrupted in Windows XP

Sash's picture

Hey! i just happened to jump to this page, nice comments here. my problem is that this very incidence happened with me 2 weeks back. my 512 MB kingston usb disk got corrupted. i have tried numerous programs to read it. it is shown in windows xp as removabe drive but it cannot be formatted. the system is shown as raw. some hard disk tools do show the exact size of the disk but can anyone tell me how to get the disk back! i will appreciate the replies to my email at sash@highnoon.com.pk

Thankx!

Recovery Tool!!!

Surowi's picture

PPL before you try anything you must check this.

http://www.runtime.org/gdb.htm

I managed to recover my files after 3 mins and with beautifull interface. This software works perfect. I tryed dozens dos recovery tools and noone worked

Try this mmaybe it helps you out

Regards

GetDataBack Sucks

Anonymous's picture

this is just another hostage holding, overpriced data reocovery tools that won't recover jack until you pay the outrageous $$$

I totally agree i knew i

Anonymous's picture

I totally agree i knew i shouldn't have wasted my dollars on this crap software it completely fucked up my drive and none of the files it recovered actually worked!!!!!!!! useless piece of crap software i am demanding a refund but these fuckers will probably get away with it!!!

re:

Polin Marsley's picture

stuck the flash drive in my laptop run Ubuntu v 6.06 (Freeware/shareware Linux) and opened the file up without any problems whatsoever. It just worked.

gpart can help you too

Anonymous's picture

The information here was invaluable, because it gave me some encouragement in trying to recover the 250GB USB drive that Win2K suddenly didnt want to accept after I missed that a VMWare Virtual machine had mounted the drive...

I lost the partition table and have no disk large enough to create an image to.

With the above info I started dumping 50MB of the disk and found that no physical damage was evident, but still no way of finding the appropriate data for the approach above.
Knowing that it was a single 250GB FAT32 partition I downloaded GPart
http://www.stud.uni-hannover.de/user/76201/gpart/

It guessed what I believe correctly and after reboot the disk was mountable (in fact it automounted which maybe wasnt the best approach). I ran fsck.vfat with no reported errors!

Once again, I had great help of the info you supplied and hope that more people in desperate need will find your article.

fsck.msdos reporting to find 165 FATs

Joerg's picture

Thanks for the great article. When trying fsck.msdos on a damaged CF card image, I stumbled over the same problem:

the "disk" claimed to have something near 165 FATs

(actually 191 in my case). I then noticed that you mounted the MBR instead of the FAT partition. When specifying the partition offset with losetup (I believe 0x8000 in your case), e.g.

losetup -o 32768 /dev/loop2 /tmp/r1

then fsck.msdos should be able to find 2 FATs (the number of FATs should be at offset 0x10 from start of the partition) and your Rapair Attempt #1 might work.

If the partion table was intact, dd'ing /dev/sda1 instead of /dev/sda would give you the partition image only, thus eliminating the need to care about the partition offset.

Unfortunately, in my case FAT#2 amd the directory table is damaged. I'm going to write a restore program exploiting the FAT chains...

Thanks,
Joerg

links to FAT info?

sean darcy's picture

Great article. I'm now trying to recover an 11gb FAT32 partition from my wife's windows machine. The MBR seems OK ( fdisk sees the partition - as does w2k disk manager ). Somehow the partition itself is messed up ( virus? sunspots? ). windows doesn't think it has a file system. fsck.vfat thinks it has 255 FATs.

This article is exactly what I was looking for in trying to fix this. But it assumes a lot of info about FAT file systems:

"That looked like an allocation chain with 16-bit entries. If these had taken the form 31 dd 00 00 32 dd 00 00 rather than 31 dd 32 dd, I might have thought I was looking at FAT32."

Where do you find this info? This is way beyond my meager knowledge. Any references would be *really* useful.

re: links to FAT info?

collin's picture

Where do you find this info? This is way beyond my meager knowledge. Any references would be *really* useful.

Sorry, I don't remember. But I remember reading somewhere that what FATxx means is is that the cluster numbers take xx bits. Hence seeing two consecutive 16-bit numbers (31 dd 32 dd) gave me the clue that it was FAT16. Unless a FAT filesystem is severely fragmented, one would expect to see consecutive cluster numbers either in a file allocation chain (cluster numbers within a file) or the free list.

NOTE: I haven't verified the full technical accuracy of the following links, your mileage may vary, not a statement of my employer, don't try this at home, etc etc etc., but these are just links that appear useful, just based on a casual glance:

I just googled on "fat16 fat32 structure cluster number" (no quotes); the first couple of links look very useful. One of them contains a pointer to http://home.teleport.com/~brainy/fatgen102.pdf, which looks pretty useful.

Another place to look is http://www.storagereview.com/guide2000/ref/hdd/file/clustChaining.html, which I think you'll find helpful.

FAT info

KultiVator's picture

Here, very cool site!

http://home.freeuk.net/foxy2k/disk/disk1.htm

Of course, advisable to handle with care, preferrably messing with an image file first. :-)

What to do when FAT1/FAT2 aren't working?

rob's picture

Salve!
Thank you for this artikel, but what to do when both FATs are gone?

I have a problem wit a 128 MB SMartMedia Card (FAT16 formated with
axxpac, a memory extention for PalmIIIx http://www.axxpac.de),
probly due ejct during writing and befor unounting it with Linux.

What I found out (with help from Debian-User-German) ML befor finding your article:

cfdisk 2.12p
Disk Drive: /dev/hde
Size: 131072000 bytes, 131 MB
Heads: 16 Sectors per Track: 32 Cylinders: 500
Name Flags Part Type FS Type [Label] Size (MB)
Pri/Log Free Space 0,03*
hde1 Boot Primary FAT16 131,05*

From a fresh formateded Card wrote I 64 Bit to the Card I can't mounted
dd if=/hd1/smartmedia128/neu/ddsmart128-neu-050616 ibs=1 skip=24063 bs=1 \
count=64 seek=24063 of=/dev/hde

Now cfdisk tells:
hde1 Boot Primary FAT16 [AXXPAC/TH ] 131,05*

The card is now mountable, I see all directories and all files, *g*
but only files smaller then 0x4000Bit (16384) can be read without
read/write errors. dosfsck will truncate all files to <=16384 and
is no help. Is this 0x4000 limit based on cluster - that there is
information to jump to the next cluster missing?
With fsck I got also the errormessage that multiple files or
directories use the same cluster.

ls -lRA is working, but some files and dirs have the date of the
crash day - but they are older and not written on that day.

So "ls -lRA" is not the final "Acid Test" for the recovery, a "cp -Rp *"
would not work in my case.

-What can I do insted of msfsck?
-When I something with "ls -lRA" the FAT1/FAT2 are still there?
-How can I extract the starting adresses for each file I see
with ls -lRA?

I know that there are tools like autopsy, but I hope that
the FATs are still recoverable.

Greetings
rob

Continue.... What to do when both FAT not working...

rob's picture

dosfsck will truncate all files to <=16384 and
is no help. Is this 0x4000 limit based on cluster - that there is
information to jump to the next cluster missing?
With fsck I got also the errormessage that multiple files or
directories use the same cluster.

ls -lRA is working, but some files and dirs have the date of the
crash day - but they are older and not written on that day.

So "ls -lRA" is not the final "Acid Test" for the recovery, a "cp -Rp *"
would not work in my case.

-What can I do insted of msfsck?
-When I something with "ls -lRA" the FAT1/FAT2 are still there?
-How can I extract the starting adresses for each file I see
with ls -lRA?

I know that there are tools like autopsy, but I hope that
the FATs are still recoverable.

Greetings
rob

continue ... does the forum don't like "

rob's picture

read/write errors. dosfsck will truncate all files to smaller 16384 and
is no help. Is this 0x4000 limit based on cluster - that there is
information to jump to the next cluster missing?
With fsck I got also the errormessage that multiple files or
directories use the same cluster.

ls -lRA is working, but some files and dirs have the date of the
crash day - but they are older and not written on that day.

So "ls -lRA" is not the final "Acid Test" for the recovery, a "cp -Rp *"
would not work in my case.

-What can I do insted of msfsck?
-When I something with "ls -lRA" the FAT1/FAT2 are still there?
-How can I extract the starting adresses for each file I see
with ls -lRA?

I know that there are tools like autopsy, but I hope that
the FATs are still recoverable.

Greetings
rob

PS: Sorry for my fragmentated post ;)

The FATs

KultiVator's picture

From what I understood, you've overwritten both FATs with "clean" ones? Bad move, sorry..
FAT = File Allocation Table .. basically, holds the information of where different parts of files are on a disk.

The root directory (which is located right after the FATs), however, holds some other info: file names, attributes, sizes, dates, and starting clusters' numbers, that's why you can see the files and can access them, but only the first 16kB, since that is the size of a cluster on that particular disk..

Was the drive corrupted or filesystem?

mangoo's picture

This piece is confusing: "According to Lexar tech support, there is a bug with Windows 2000 (that MS never bothered to fix) and can corrupt the drive when it is removed without proper eject.".

Does this mean that the drive was damaged because of this bug (and you could say good bye my $70 USB), or was it the filesystem on this drive (so that reformatting that USB would make it usable again)?

re: Was the drive corrupted or filesystem?

collin's picture

Just the filesystem was corrupted. The electronics in the card were fine.

Prosoft engineering

Anonymous's picture

Prosoft engineering produces a range of data rescue software for the Mac which I have personally used with much success. They support everying from HDD to memory cards to USB flash drives.

address is: http://www.prosofteng.com

There are various products from Image Rescue to Data Rescue, they are not open source but there are trial versions available for download which will allow you to recover a file at a time. Of course all of this is for the mac only. They are famous for their official Novell client for Mac as well.

Shameless ad.

WeAllHateSpam's picture

Shameless ad.

Mounting under /tmp

Duncan Thomas's picture

Nice article, printed and saved for if I ever need to do something similar. A very quick point though - beware of mounting under /tmp on systems that aren't yours; some badly written tmp cleanup scripts can go in and clean up your newly mounted filesystem :-( I've lost filesystems to this a couple of time before I worked out what was going on...

JPEG images

Gerrit Hannaert's picture

I approached a similar problem I had differently - I knew I was looking for JPEG images and that there was a chance they were intact. I assumed the FAT was unreliable, and admittedly forgot about the second FAT copy.

I also assumed there was no fragmentation, and then proceeded to 'dd' anything that looked like an EXIF header +1.5MB to disk as individual files.

Something along these lines:
hexdump -C sda1_flashdisk.dd | grep -i "exif" >headers.hex
cat headers.hex | awk --non-decimal-data '{print (sprintf("%d","0x"$1)/512)}' > headers.512.dec
cat headers.512.dec | xargs --replace bash -c "dd skip={} if=sda1_flashdisk.dd bs=512 count=3000 of=./{}.jpg"

I was lucky on several counts: all files were of predicatable content, there was little if any fragmentation, and JPEG seems to tolerate excess junk added to the end of file :-)

I wish I had known about this

Anonymous's picture

I wish I had known about this about a month ago. My Father had a 128MB Sandisk USB flash disk which he had been using under Windows 2000 and it died in a rather spectacular way, the only difference here though is that when I fed it to my linux workstation, it did not find any partition records - it was royally goosed. I had an identical USB flash disk to my Father's and tried to overwrite the partition records from my chip to his, but with no success. Extremely interesting article though and very well researched. Congratulations on recovering your data - I challenge any Windows user to even come close to that without having to write a suite of programs.

Overall, pretty good

Azhrei's picture

I've done this type of recovery myself, using basically the same approach: look at hex dumps and compare the corrupted media with uncorrupted media, looking for areas that match, then copying individual sectors in an attempt to produce something readable. I wouldn't have thought of using the backup FAT table for fsck.msdos, though. :) Usually, fixing a partition table would do it. Or I would've copied FAT#2 on top of FAT#1 and just tried to mount it...

Overwriting FAT#1 with FAT#2 (or vice-versa)

KultiVator's picture

Overwriting FAT#1 with FAT#2 (or vice-versa) - this isn't necessarilly a good idea if there's a chance that both are more or less corrupted.
In such cases, trying to "merge" them (with extreme care, of course) would be better..
Well, just using the copy #1 at one time, and #2 at the other, while both times trying to access & restore files and see which time you get better results ("proper" file contents), would be the most cautious way.

thanks for your comment

Collin's picture

Thanks for posting your comment! Those are both good ideas, which I didn't think of. Well, I thought of trying to fix the partition table but I didn't know enough about the boot sector contents (and still don't).

The important thing is that GNU/Linux distros give us enough tools so that we can approach these problems in different ways... even problems caused by closed operating systems!

Thanks again for your note.

Question: How would gpart hav

Anonymous's picture

Question: How would gpart have performed on that job?

I recovered once a whole laptop for someone in the office with half a years work of source code on it...

"I recovered once a whole lap

Mason Deaver's picture

"I recovered once a whole laptop for someone in the office with half a years work of source code on it"

Can we infer from your comment that your colleague whose work you saved is a software developer who had not backed up his or her work in six months? Yeesh! I'm getting queasy just thinking about this; I need to go lie down for a while ....

Windows tools to test

Anonymous's picture

Hey, there are some windows based tools that exist but I don't think they work for flash drives, but you could always try.

Driverescue 1.9d (scans through bios in windows).

Testdisk - http://www.cgsecurity.org/testdisk.html

PCInspector - which looks exactly like Driverescue before driverescue went commerical http://www.pcinspector.de/file_recovery/UK/welcome.htm

My usb-disk (Trekstore, 8

Anonymous's picture

My usb-disk (Trekstore, 8 GB) was damaged by unpluging during a write-operation.
I tried some ways, mentioned above. Nothing worked.
Finally I found Testdisk and it worked greatly!
Many thanks for that advice!

Using windows Disk Manager

QiQiangZhu's picture

Using windows Disk Manager to creat a new partition for your usb drive, although may lost 8MB space, still is the best way to do the rescure for a USB flash drive.

Or using Partition Magic

QiQiangZhu's picture

Or using Partition Magic Boot Disk to repartition this USB drive, you will gain back all the space on it.

Recover HD after Ghosting

need help's picture

Could you use this same process to recover files after Ghosting a HD? If not; what would you suggest I use? If it's possible at all to recover files after Ghosting a HD.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix