Recover VMFS datastore with all data

Sometimes it might happen that you lost datastore with all date for some reasons, deleted by mistake, VMFS corruption. It happen to me as well. Fortunately, following VMwareKB article I was able to recover VMFS with all data.

The most important – DO NOT PANIC !! 🙂 and DO NOT create new VMFS on that LUN because it will overwrite old partition table.

 Recover VMFS3 datastore

To recover VMFS3 volumes or upgraded VMFS5 from VMFS3 (without formatting) we will use old good toll fdisk :-).

• type esxcfg-scsidev -c list LUNs and identify Console device /dev/sd* , in my example it is /dev/sdg with size 507940 MB
• type fdisk -l /dev/sd* (replace * with correct letter) to see info about device – you should not see partition table.
• type fdisk -u /dev/sd* (replace * with correct letter) and follow below steps:
  • type n – create new partition
  • type p – for primary
  • type 1 – for first partition on disk
  • type 128 – this is starting block for VMFS which was created using vSphere client
  • press Enter – to leave default values for last sector
  • press t – select partition type
    • optionally you can type L to list all partition types
  • type fb for VMware VMFS file system
  • type w to write partition on disk

[root@server01 ~]# esxcfg-scsidevs -c
Device UID                            Device Type      Console Device                              Size      Multipath PluginDisplay Name
eui.001738000a0b0000                  RAID Ctlr        /vmfs/devices/genscsi/eui.001738000a0b0000  0MB       NMP     IBM Fibre Channel RAID Ctlr (eui.001738000a0b0000)
eui.001738000a0b0049                  Direct-Access    /dev/sdg                                    507904MB  NMP     IBM Fibre Channel Disk (eui.001738000a0b0049)
eui.001738000a0b017e                  Direct-Access    /dev/sdf                                    2097151MB NMP     IBM Fibre Channel Disk (eui.001738000a0b017e)
[root@vsxet01 ~]# esxcfg-scsidevs -c |grep -i eui.001738000a0b0049
eui.001738000a0b0049                  Direct-Access    /dev/sdg                                    507904MB  NMP     IBM Fibre Channel Disk (eui.001738000a0b0049)
[root@server01 ~]#
[root@server01 ~]# fdisk -l /dev/sdg

Disk /dev/sdg: 532.5 GB, 532575944704 bytes
255 heads, 63 sectors/track, 64748 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
[root@server01 ~]# fdisk -u /dev/sdg

The number of cylinders for this disk is set to 64748.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (63-1040187391, default 63): 128
Last sector or +size or +sizeM or +sizeK (128-1040187391, default 1040187391):
Using default value 1040187391

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): L

 0  Empty           1e  Hidden W95 FAT1 80  Old Minix       bf  Solaris
 1  FAT12           24  NEC DOS         81  Minix / old Lin c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          82  Linux swap / So c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  83  Linux           c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     84  OS/2 hidden C:  c7  Syrinx
 5  Extended        41  PPC PReP Boot   85  Linux extended  da  Non-FS data
 6  FAT16           42  SFS             86  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS       4d  QNX4.x          87  NTFS volume set de  Dell Utility
 8  AIX             4e  QNX4.x 2nd part 88  Linux plaintext df  BootIt
 9  AIX bootable    4f  QNX4.x 3rd part 8e  Linux LVM       e1  DOS access
 a  OS/2 Boot Manag 50  OnTrack DM      93  Amoeba          e3  DOS R/O
 b  W95 FAT32       51  OnTrack DM6 Aux 94  Amoeba BBT      e4  SpeedStor
 c  W95 FAT32 (LBA) 52  CP/M            9f  BSD/OS          eb  BeOS fs
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi ee  EFI GPT
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a5  FreeBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a6  OpenBSD         f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a7  NeXTSTEP        f1  SpeedStor
12  Compaq diagnost 5c  Priam Edisk     a8  Darwin UFS      f4  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       a9  NetBSD          f2  DOS secondary
16  Hidden FAT16    63  GNU HURD or Sys ab  Darwin boot     fb  VMware VMFS
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT
Hex code (type L to list codes): fb
Changed system type of partition 1 to fb (VMware VMFS)

Command (m for help):
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

• Run vmkstools -V to discover VMFS back to system

[root@server01 ~]# vmkfstools -V
[root@server01 ~]#

• Check log  /var/log/messages . As you can kernel see new partition sdg1 which sits on /dev/sdg device . Mission complete successfully.

Sep 19 11:07:48 vsxet01 kernel: [7851834.657658] SCSI device sdg: 1040187392 512-byte hdwr sectors (532576 MB)
Sep 19 11:07:48 vsxet01 kernel: [7851834.664579] sdg: Write Protect is off
Sep 19 11:07:48 vsxet01 kernel: [7851834.673365] SCSI device sdg: drive cache: write through
Sep 19 11:07:48 vsxet01 kernel: [7851834.679887]  sdg: sdg1
Sep 19 11:07:50 vsxet01 kernel: [7851836.729279] SCSI device sdg: 1040187392 512-byte hdwr sectors (532576 MB)
Sep 19 11:07:50 vsxet01 kernel: [7851836.736230] sdg: Write Protect is off
Sep 19 11:07:50 vsxet01 kernel: [7851836.743157] SCSI device sdg: drive cache: write through
Sep 19 11:07:50 vsxet01 kernel: [7851836.749681]  sdg: sdg1

You should see datastore with new name on ESX server. Rescan HBA on all other servers to bring VMFS back on them.

• VMwareKB article 1002281