I will not try to explain PSA here, there is enough resources to read about it (if you asked me for recommendation I’d suggest Cormac Hogan’s excellent blog) and I don’t feel expert enough to “increase entropy” on that subject.
Instead I’d like to focus on what used to be particularly confusing to me: the claimrules, SATP rules and how to manage them with esxcli.
PSA behaviour can be controlled with two sets of rules (at least I know two of them – maybe there is more 😉 ) first there is set of “claimrules” that decide which PSA plugin (NMP or 3rd party) will handle any particular storage device. Once that is decided there is another set of rules – “SATP rules” which decide which SATP exactly (out of the set of SATPs available in selected MP plugin) will handle the I/O.
Initially I was confusing these two sets because esxcli syntax to manipulate them is very similar and I was easily loosing track of what is going on with PSA on my vSphere host.
Just to set the focus – all listings that will follow come from vSphere 5.5 host in my homelab.
NOTE: do not forget to do vmware backup before you start playing with PSA rules 🙂
So let’s have a look at “claimrules” first.
~ # esxcli storage core plugin list Plugin name Plugin class ----------- ------------ NMP MP ~ # ~ # esxcli storage core claimrule list Rule Class Rule Class Type Plugin Matches ---------- ----- ------- --------- --------- --------------------------------- MP 0 runtime transport NMP transport=usb MP 1 runtime transport NMP transport=sata MP 2 runtime transport NMP transport=ide MP 3 runtime transport NMP transport=block MP 4 runtime transport NMP transport=unknown MP 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport MP 101 file vendor MASK_PATH vendor=DELL model=Universal Xport MP 65535 runtime vendor NMP vendor=* model=* ~ # ~ # esxcli storage core path list --path=vmhba33:C0:T0:L2 iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434 UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434 Runtime Name: vmhba33:C0:T0:L2 Device: t10.F405E46494C45425930586A5B426D2133533D6D276559434 Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C45425930586A5B426D2133533D6D276559434) Adapter: vmhba33 Channel: 0 Target: 0 LUN: 2 Plugin: NMP State: active Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:host03-208b6b7a Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1 Adapter Transport Details: iqn.1998-01.com.vmware:host03-208b6b7a Target Transport Details: IQN=iqn.2006-01.com.openfiler:mylabfiler01.tier1 Alias= Session=00023d000001 PortalTag=1
As you can see I only have Native Multipathing Plugin (NMP) installed, the claimrule list is also pretty generic, with either NMP or MASK_PATH plugins applied and MASK_PATH is used to hide some specific (unsupported?) SAN device from vSphere. These claimrules are checked in descending order (so if you configure more than one rule that could match a device the one that is listed first will apply).
You can also see a “catch all” rule at the bottom of the list, it is there to make sure that any device detected yet not matched with previous rules is handled with NMP.
I also show information about specific storage path (which happens to be iSCSI LUN that I share from OpenFiler appliance) because I want to show how to change claimrules to mask this particular LUN.
~ # esxcli storage core claimrule add --autoassign --type=target --transport=iscsi --iqn iqn.2006-01.com.openfiler:mylabfiler01.tier1 --lun 2 --plugin=MASK_PATH ~ # esxcli storage core claimrule list Rule Class Rule Class Type Plugin Matches ---------- ----- ------- --------- --------- ------------------------------------------------------------------ MP 0 runtime transport NMP transport=usb MP 1 runtime transport NMP transport=sata MP 2 runtime transport NMP transport=ide MP 3 runtime transport NMP transport=block MP 4 runtime transport NMP transport=unknown MP 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport MP 101 file vendor MASK_PATH vendor=DELL model=Universal Xport MP 5001 file target MASK_PATH transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2 MP 65535 runtime vendor NMP vendor=* model=* ~ # ~ # esxcli storage core claimrule load ~ # esxcli storage core claimrule list Rule Class Rule Class Type Plugin Matches ---------- ----- ------- --------- --------- ------------------------------------------------------------------ MP 0 runtime transport NMP transport=usb MP 1 runtime transport NMP transport=sata MP 2 runtime transport NMP transport=ide MP 3 runtime transport NMP transport=block MP 4 runtime transport NMP transport=unknown MP 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport MP 101 file vendor MASK_PATH vendor=DELL model=Universal Xport MP 5001 runtime target MASK_PATH transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2 MP 5001 file target MASK_PATH transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2 MP 65535 runtime vendor NMP vendor=* model=* ~ # ~ # esxcli storage core claiming reclaim --device=t10.F405E46494C45425930586A5B426D2133533D6D276559434 ~ # ~ # ~ # esxcli storage core path list --path=vmhba33:C0:T0:L2 iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1- UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1- Runtime Name: vmhba33:C0:T0:L2 Device: No associated device Device Display Name: No associated device Adapter: vmhba33 Channel: 0 Target: 0 LUN: 2 Plugin: MASK_PATH State: dead Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:host03-208b6b7a Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1 Adapter Transport Details: Unavailable or path is unclaimed Target Transport Details: Unavailable or path is unclaimed Maximum IO Size: 131072 ~ #
I use esxcli storage core claimrule add command here to create new claimrule, let vSphere autoassign the rule number, then I specify type of “target” and provide as much details as possible that will allow device to be matched (so I declare the protocol needs to be iSCSI, coming from specific iqn with LUN 2)
I want this LUN to be masked, so I select MASK_PATH plugin.
The rule shows immediately in the list, yet only in “file” class, which means it is saved, but vmkernel is not using it at the moment and we need to (re)load the claimrules.
The final step after loading claimrules (our claimrule is listed twice now, both in “file” and “runtime” class) is to reclaim the device (to activate our new rule) and… we’re ready.
Information about the path shows now that plugin has changed to MASK_PATH and the path state is DEAD!
You have to believe me that this device disappears from vSphere Client immediately after what we just did.
OK, let’s try to revert this now and make LUN 2 visible again.
~ # esxcli storage core claimrule remove --rule=5001 ~ # esxcli storage core claimrule list Rule Class Rule Class Type Plugin Matches ---------- ----- ------- --------- --------- ------------------------------------------------------------------ MP 0 runtime transport NMP transport=usb MP 1 runtime transport NMP transport=sata MP 2 runtime transport NMP transport=ide MP 3 runtime transport NMP transport=block MP 4 runtime transport NMP transport=unknown MP 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport MP 101 file vendor MASK_PATH vendor=DELL model=Universal Xport MP 5001 runtime target MASK_PATH transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2 MP 65535 runtime vendor NMP vendor=* model=* ~ # ~ # esxcli storage core claimrule load ~ # esxcli storage core claimrule list Rule Class Rule Class Type Plugin Matches ---------- ----- ------- --------- --------- --------------------------------- MP 0 runtime transport NMP transport=usb MP 1 runtime transport NMP transport=sata MP 2 runtime transport NMP transport=ide MP 3 runtime transport NMP transport=block MP 4 runtime transport NMP transport=unknown MP 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport MP 101 file vendor MASK_PATH vendor=DELL model=Universal Xport MP 65535 runtime vendor NMP vendor=* model=* ~ # ~ # esxcli storage core claiming reclaim --device=t10.F405E46494C45425930586A5B426D2133533D6D276559434 Unknown Device name t10.F405E46494C45425930586A5B426D2133533D6D276559434 ~ # ~ # Duh! That's confusing, let's try other method (using paths) ~ # ~ # esxcli storage core claiming unclaim --type=path --path=vmhba33:C0:T0:L2 ~ # esxcli storage core path list --path=vmhba33:C0:T0:L2 iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1- UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1- Runtime Name: vmhba33:C0:T0:L2 Device: No associated device Device Display Name: No associated device Adapter: vmhba33 Channel: 0 Target: 0 LUN: 2 Plugin: (unclaimed) State: dead Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:host03-208b6b7a Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1 Adapter Transport Details: Unavailable or path is unclaimed Target Transport Details: Unavailable or path is unclaimed Maximum IO Size: 131072 ~ # ~ # esxcli storage core adapter rescan --all ~ # esxcli storage core path list --path=vmhba33:C0:T0:L2 iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434 UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434 Runtime Name: vmhba33:C0:T0:L2 Device: t10.F405E46494C45425930586A5B426D2133533D6D276559434 Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C45425930586A5B426D2133533D6D276559434) Adapter: vmhba33 Channel: 0 Target: 0 LUN: 2 Plugin: NMP State: active Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:deobhv03-208b6b7a Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1 Adapter Transport Details: iqn.1998-01.com.vmware:host03-208b6b7a Target Transport Details: IQN=iqn.2006-01.com.openfiler:mylabfiler01.tier1 Alias= Session=00023d000001 PortalTag=1 Maximum IO Size: 131072 ~ #
Now “unmasking” a LUN that we’ve hidden with MASK_PATH plugin can be a bit tricky, at the beginning we need to remove the rule (using just the rule number) from wherever it is stored, but as you can see “remove” operation does not immediately deactivate the claimrule (it is still listed with “runtime” class).
This is solved by (re)loading the whole claimrule set, but subsequent attempt to reclaim the device fails with “Unknown Device” error…
Well it makes sense, we’ve just masked it a moment ago, so t10 identifier we use disappeared from our system…
At first I thought host reboot is the only solution to make that LUN visible again, but after a few minutes of playing around with esxcli I discovered there is a way to avoid a reboot.
We can not reclaim a device, but we can set the storage path (using runtime identifier) that “leads” to device to be “unclaimed” and then rescan our storage adapters so that default “catch all” claimrule is matched and our LUN gets handled by NMP again, you can see all the steps listed above.
The thing worth to remember is that claimrules are really easy – it is just about which MP plugin to assign to storage devices.
SATP rules on the other hand can get really complicated, cause you can have many SATPs as “sub-plugins” to MP plugin that is selected. NMP itself (so the PSA plugin that you get “out-of-box”, just with “vanilla” vSphere installation) has 13 of these SATPs, as a result SATP rule set gets much bigger and if you consider that each SAN can require some specific options to be provided by the means of SATP rules… Well I suggest you just to execute
esxcli storage nmp satp rule list
from your host’s console and admire this whole list scrolling.
Yet the esxcli syntax to manipulate these SATP rules is very similar to what we exercised with claimrules (and this is what was confusing me a lot when I first started to use them).
As an example I will show you how to “make” an SSD out of traditional, magnetic hard disk (something you can probably hear about at every VMware training, also “tier-1” bloggers like Duncan or William have posted about it like ages ago).
# esxcli storage nmp satp rule add --satp=VMW_SATP_LOCAL --device=mpx.vmhba1:C0:T1:L0 --option=enable_ssd # esxcli storage nmp satp rule list --satp=VMW_SATP_LOCAL Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description -------------- ------------------- ------ ----- ------ --------- ---------- ---------- ------------- ----------- ----------- ---------------------------- VMW_SATP_LOCAL usb system USB Devices VMW_SATP_LOCAL ide system IDE Devices VMW_SATP_LOCAL block system RAID Block Devices VMW_SATP_LOCAL parallel system Parallel SCSI Devices VMW_SATP_LOCAL sas system Serial Attached SCSI Devices VMW_SATP_LOCAL sata system Serial ATA Devices VMW_SATP_LOCAL unknown system Unknown Devices VMW_SATP_LOCAL mpx.vmhba1:C0:T1:L0 enable_ssd user # # esxcli storage core claiming reclaim --device=mpx.vmhba1:C0:T1:L0 # esxcli storage core device list --device=mpx.vmhba1:C0:T1:L0 mpx.vmhba1:C0:T1:L0 Display Name: Local VMware, Disk (mpx.vmhba1:C0:T1:L0) Has Settable Display Name: false Size: 40960 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0 Vendor: VMware, Model: VMware Virtual S Revision: 1.0 SCSI Level: 2 Is Pseudo: false Status: on Is RDM Capable: false Is Local: true Is Removable: false Is SSD: true Is Offline: false Is Perennially Reserved: false Queue Full Sample Size: 0 Queue Full Threshold: 0 Thin Provisioning Status: unknown Attached Filters: VAAI Status: unsupported Other UIDs: vml.0000000000766d686261313a313a30 Is Local SAS Device: false Is Boot USB Device: false No of outstanding IOs with competing worlds: 32 #
The only thing we do here is to add a rule that assigns VMW_SATP_LOCAL and sets “enable_ssd” option for our mpx.vmhba1:C0:T1:L0 device. Luckily we don’t need to (re)load anything just reclaim the device – and here we go, our magnetic drive will be considered SSD by vSphere from now on (you can see “Is SSD” property has value of “true”).
To be honest – reclaiming device just for SATP rule change also used to confuse me a lot – even though we only modify some SATP option it looks like whole PSA “stack” needs to be reset for this operation (including initial MP plugin selection for device we are working on).
The other observation I have – it seems like SATP rules are applied in different manner than “claimrules”, I’ve written above that “claimrules” are applied in descending order (and they have numbers!), as for SATP I think these rules are applied according to “best match” policy (same way like network gateway to forward traffic to is selected if you have multiple TCP/IP static routes defined). If you look at SATP rules list for VMW_SATP_LOCAL above there are at least two rules (“transport = sata” configured by default and our “device = mpx.vmhba1:C0:T1:L0” rule) that match, yet our trick works, so it seems “best match” policy is used (cause our rule is more precise, so makes better match than generic “transport” class rule).
If not “best match” policy, maybe the order in which rule classes are presented in listing above (Device > Vendor > Model > Driver > Transport) is the key: so if you include device name in your rule definition it overrides all rules with matching vendor or driver?
Oh and have you noticed? SATP rules don’t have numbers!
But again – this is only an “educated guess” based on some of my experiments, I wasn’t able to find any documentation that would prove me right or wrong and explain in what order SATP rules are really applied.
OK, it is fun to make a fool of vmkernel for a while, but just having SSD enabled for a spindle hard drive will not give you any performance boost, in fact it will give you nothing but trouble if you continue like this.
The only real-life scenario you might want to use this trick is when you have SSD from some “out-of-mainstream” vendor and capabilities of this disk are not recognized properly by vSphere, then you have to configure this SATP rule to get the performance you paid for.
Since I used this rule against magnetic disk, I’m going to revert my settings to their original state.
~ # esxcli storage nmp satp rule remove --satp=VMW_SATP_LOCAL --device=mpx.vmhba1:C0:T1:L0 --option=enable_ssd ~ # esxcli storage core claiming reclaim --device=mpx.vmhba1:C0:T1:L0 ~ # esxcli storage core device list --device=mpx.vmhba1:C0:T1:L0 mpx.vmhba1:C0:T1:L0 Display Name: Local VMware, Disk (mpx.vmhba1:C0:T1:L0) Has Settable Display Name: false Size: 40960 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0 Vendor: VMware, Model: VMware Virtual S Revision: 1.0 SCSI Level: 2 Is Pseudo: false Status: on Is RDM Capable: false Is Local: true Is Removable: false Is SSD: false Is Offline: false Is Perennially Reserved: false Queue Full Sample Size: 0 Queue Full Threshold: 0 Thin Provisioning Status: unknown Attached Filters: VAAI Status: unsupported Other UIDs: vml.0000000000766d686261313a313a30 Is Local SAS Device: false Is Boot USB Device: false No of outstanding IOs with competing worlds: 32
Removing SATP rules is not as easy as with claimrules – you can not use rule number (they don’t have numbers, right?), so there is a need to provide the exact syntax the rule was created with.
Once the rule is removed – just reclaim the device again and everything is back to normal.
I hope you find this lengthy post useful, any comments are welcome!
Small digression. Would you be able to configure PSA related settings via host profile? It might be useful if you think about VCAP examination. Really.
I have learnt to vcp exam and this post excellent explain claimrules to me 🙂 thanks 🙂