Recently, when I tried to ssh to my AWS Linux AMI, I got an error that the key is not valid. First, I thought that somehow my private key was stolen, but upon further investigation in the logs, I found these errors.

/ containts a file system with errors, check forced

The AMI will try to fsck automatically. Most of the time, this will fix your problems. In my case it failed and I got a useless AMI that won’t boot. If fsck doesn’t work, check the console. I saw that one of the checks failed.

According to AWS and this troubleshooting AWS Troubleshooting, first stop the instance.

Go to Volumes, Detach volume and wait until it says available.

Create a new instance, same AMI type, same resources. Right click the newly created AMI, choose to Attach volume and select the instance where this volume will be attached.

Once attached, in the new AMI do fdisk -l to see the device name for your attached disk. In my case I see /dev/xvdf.

Try to repair it from here. In my case I got the following errors.

fsck.ext4 -v /dev/xvdf
e2fsck 1.42.12 (29-Aug-2014)
ext2fs_open2: Bad magic number in super-block
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/xvdf

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

We have to find the superblocks that are corrupt. Now, let’s see where my superblocks are kept.

mke2fs -n /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Found a gpt partition table in /dev/xvdf
Proceed anyway? (y,n) y
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: 350b4d59-28c8-422f-a3af-ede9daae43e3
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

NOTE: You’ll get different numbers. Use yours not mine.

Next step is to try to restore the superblocks.

e2fsck -b 1605632 /dev/xvdf
e2fsck 1.42.12 (29-Aug-2014)
e2fsck: Invalid argument while trying to open /dev/xvdf

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

I tried all superblock numbers and all failed. Finally, I tried with the last resort.

mke2fs -S /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Found a gpt partition table in /dev/xvdf
Proceed anyway? (y,n) y
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: 97fc1ad1-b8dc-44ec-ba46-33061541cd60
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing superblocks and filesystem accounting information: done

Force the checks on the filesystem.

e2fsck -f /dev/xvdf -y
/dev/xvdf: ***** FILE SYSTEM WAS MODIFIED *****
/dev/xvdf: 9001/524288 files (5.0% non-contiguous), 114693/2097152 blocks

At the end, mount the volume and see if you see your data. In my case, I got nothing.

ls -l
total 148
drwx------ 516 root root 147456 May 13 23:23 lost+found