Mystery of storage misalignment
Usually when people hear that term and how important it is to avoid storage misalignment on environments, regardless if we talk about virtual or physical, they thought their “kingdoms” are not affected and their just ignoring it. Suddenly they end up in a performance problems on storage which was caused by storage misalignment. Anyway, in regards of misalignment in virtual environments and shared storage, it is really serious problem because it does not affect only single OS instance but bunch of virtual machines which runs together on shared storage with misalign virtual machine or machines. Of course one misalign VM, among hundreds, usually (depending on IOPS characteristic) it is not a problem. Problem is when there are more virtual machines with storage misalignment. This brings serious storage performance degradation.
What exactly is storage misalignment ?
Starting in 2009, storage vendors stared producing disks with 4096 bytes blocks instead of 512 bytes like it was before. However thanks to firmware, each 4096 block is devised by logical blocks of 512 bytes. In each 4096 physical disk block there are 8 x 512 bytes logical blocks which are presented to the operating system.
Most modern file systems use data structures that are 4096 bytes or larger in size. When VM wants to read or write one of these data structures on a new disk with 4096-byte sectors and the file system data structures happen to align perfectly with the underlying physical partition size, a read or write of a 4096-byte data structure results in a read or write of a single sector but when the file system data structures do not align perfectly with the underlying physical sectors, a read or write operation must access two physical sectors. Heaving misaligned Virtual machines significantly increase number of IOPS issued into underlying storage which in same cases might cause overall Virtual Machines performance drop down.
In figure below, single read\write from a misaligned virtual machine affects 2 physical chunks causing additional IO to underlying storage system. If it is only one low IOPS intensive VM, it is not a big problem, modern storage systems are fast enough to handle workload even from misalign VM. Problem starts when there more VMs with different IOPS characteristics and additional storage saving features such as deduplication are enabled.
After Virtual Machine storage alignment single write on virtual machine block filesystem issues single IO to underlying storage.
Side effects of misalignment
- drop down disk performance
- backup – longer backup window
- deduplication – longer window to perform data deduplication
How to identify storage misalignment
In Windows OS:
- Run msinfo32.exe, under Component –> storage –>Disks –>Partition starting offset has to be divisible by 4096. On a figure below is not divisible it means that storage is misaligned.
- Download Free tool from ctxadmtools.com called VM Check Alignment v1.0 and run it against all Windows machines in network.
Linux or Unix workloads:
SSH to system and run fdisk -lu device, look at Start bock, in example below partition starts on 63 sector, which means is not aligned because sector number 63 “belongs” to the last sector from the first 4096 block (see first figures on top).
# fdisk -lu /dev/sda
Device Boot Start End Blocks Id System
/dev/sda1 * 63 97659134 48829536 7 Linux83
Tools used to fix misalignment.
- Tool #1 is UberAlign, fantastic tool.
- Storage vendor tools such as: NetAPP Virtual Storage Console.
- GParted – free Linux partition manager
- VMware converter 5 and newer align storage during conversion
- Platespin, Quest vOptimizerPro, Quest vConverter
How to avoid storage misalignment.
- follow storage vendor best practices and technical papers
- use tools which are recommended by vendor to manage storage and vSphere environment
- prepare golden image (virtual machine template)
- if possible, integrate vSphere with storage system using designated plugins to bring more visibility about storage backend to vSphere and vice versa
- use “virtualization aware” operating systems on virtual machines, such as Windows 2008 and newer, RHEL 6 and newer and more
- If you have to use older OS, prepare virtual machine template with aligned storage
- periodic checking of infrastructure