Usually when people hear that term and how important it is to avoid storage misalignment on environments, regardless if we talk about virtual or physical, they thought their “kingdoms” are not affected and their just ignoring it. Suddenly they end up in a performance problems on storage which was caused by storage misalignment. Anyway, in regards of misalignment in virtual environments and shared storage, it is really serious problem because it does not affect only single OS instance but bunch of virtual machines which runs together on shared storage with misalign virtual machine or machines. Of course one misalign VM, among hundreds, usually (depending on IOPS characteristic) it is not a problem. Problem is when there are more virtual machines with storage misalignment. This brings serious storage performance degradation.
What exactly is storage misalignment ?
Starting in 2009, storage vendors stared producing disks with 4096 bytes blocks instead of 512 bytes like it was before. However thanks to firmware, each 4096 block is devised by logical blocks of 512 bytes. In each 4096 physical disk block there are 8 x 512 bytes logical blocks which are presented to the operating system.
storage block size
misalign storage
- storage properly align
Most modern file systems use data structures that are 4096 bytes or larger in size. When VM wants to read or write one of these data structures on a new disk with 4096-byte sectors and the file system data structures happen to align perfectly with the underlying physical partition size, a read or write of a 4096-byte data structure results in a read or write of a single sector but when the file system data structures do not align perfectly with the underlying physical sectors, a read or write operation must access two physical sectors. Heaving misaligned Virtual machines significantly increase number of IOPS issued into underlying storage which in same cases might cause overall Virtual Machines performance drop down.
Misaligned virtual machine filesystem
In figure below, single read\write from a misaligned virtual machine affects 2 physical chunks causing additional IO to underlying storage system. If it is only one low IOPS intensive VM, it is not a big problem, modern storage systems are fast enough to handle workload even from misalign VM. Problem starts when there more VMs with different IOPS characteristics and additional storage saving features such as deduplication are enabled.
R/W IO on misaligned storage
After Virtual Machine storage alignment single write on virtual machine block filesystem issues single IO to underlying storage.
After alignment
Side effects of misalignment
- drop down disk performance
- backup – longer backup window
- deduplication – longer window to perform data deduplication
How to identify storage misalignment
In Windows OS:
- Run msinfo32.exe, under Component –> storage –>Disks –>Partition starting offset has to be divisible by 4096. On a figure below is not divisible it means that storage is misaligned.
- Download Free tool from ctxadmtools.com called VM Check Alignment v1.0 and run it against all Windows machines in network.
disk offest on misaligned partition
Linux or Unix workloads:
SSH to system and run fdisk -lu device, look at Start bock, in example below partition starts on 63 sector, which means is not aligned because sector number 63 “belongs” to the last sector from the first 4096 block (see first figures on top).
# fdisk -lu /dev/sda Device Boot Start End Blocks Id System /dev/sda1 * 63 97659134 48829536 7 Linux83
Tools used to fix misalignment.
- Tool #1 is UberAlign, fantastic tool.
- Storage vendor tools such as: NetAPP Virtual Storage Console.
- GParted – free Linux partition manager
- VMware converter 5 and newer align storage during conversion
- Platespin, Quest vOptimizerPro, Quest vConverter
How to avoid storage misalignment.
- follow storage vendor best practices and technical papers
- use tools which are recommended by vendor to manage storage and vSphere environment
- prepare golden image (virtual machine template)
- if possible, integrate vSphere with storage system using designated plugins to bring more visibility about storage backend to vSphere and vice versa
- use “virtualization aware” operating systems on virtual machines, such as Windows 2008 and newer, RHEL 6 and newer and more
- If you have to use older OS, prepare virtual machine template with aligned storage
- periodic checking of infrastructure
Artur, I strongly disagree with this article and whole bullshit about misalignment stuff. What I noticed is that vendors (big ones like NetApp-direct experience and EMC) tend to find those lousy bullshit to bounce complaining customers even without checking what’s real customer issue. I went through this shit once and i know what i’m talking about. We had like hundreds of misaligned VM’s on our NetApp cluster and basically we had other issues related to 10Gbit throughput from particular luns (customer was complaining about his speeds from linux or whatever, don’t remember already). So we created case in NetApp about… Read more »
Nice writeup and refresher. Agree, templates should be the first defense, and the NetApp VSC can quickly analyze the environment for misaligned VMs.
Also agree with Marek. The first step in troubleshooting by vendors was, is your firmware up to date? Update and then call us back. Sometimes doing the initial search is valuable before calling support, as it helps me to gauge where we are going in the process. Just my 2 Euro:)