The biggest operational pain point in standard 3 Tier architecture was (and still is) maintenance. Regardless if we talking here about Hypervisor upgrade\update or firmware update, or any other maintenance activity. Below you can find example of actions\to-do’s, admin has to perform before start of Hypervisor upgrade, to upgrade 3-Tier architecture:
- Go to multiple vendors website, check hardware vs firmware vs hypervisor compatibility for all components
- BIOS version
- HBA firmware and drivers
- NIC firmware and drivers
- supported hypervisor version
- FC fabric switch firmware version
- and so on and so on
- download all necessary binaries
- create dependency list
- install one by one from several different management consoles
In addition, at least several resources are involved in activity such as:
- storage administrator
- check all FC fabric components firmware level and upgrade if necessary, usually only one team (in this case Storage team) has access to management tools
- virtualization\vSphere\Hyper-V administrator or engineer
- check all hardware firmware level and upgrade if necessary
- perform hypervisor upgrade
- In some extreme cases, you have to engage vendor Professional Services (and paid for it, of course) to perform any upgrade on your environment because you are NOT ALLOWED to do it by yourself.
At the end of this preparation, if one is lucky and all went well, then the mission is accomplished. Question is – how much time is your operational\engineering team spent on such trivial task like this, or how much money you have to spend to pay for Profession Service to do upgrade? Answer is – definitely to much. Days and days of preparation work and an additional few days to perform upgrades and patches. At the end, lot of money wasted on something which should be as easy as OneClick upgrade on Nutanix.
That’s why Nutanix, in the NOS 4.X release bring one of the best features ever – One click upgrade. Where from single console you can upgrade not only Acropolis Operating System (AOS) but also:
- Hypervisor
- ESXi with major releases and minor patches
- Hyper-V
- Acropolis AHV – minor and major releases are supported
- Firmware
- disks firmware – in the current releases
- NIC firmware – in the future releases
- BIOS – in the future releases
- IPMI\BMC firmware – in the current releases
- LSI cards firmware – in the future releases
- Tools
- NCC – Nutanix cluster check
- Foundation
- Prism Central – you can upgrade Prism Central with Nutanix OneClick upgrade too.
With Nutanix OneClick upgrade, above actions with:
- ZERO downtime for virtual machines and its applications, !!!!.
- performed by one person, from single console
- no need to check compatibility matrix, Nutanix OneClick upgrade does it for you
As you can see, Acropolis software does everything for you. You do not have to worry about compatibility matrix, dependencies, or upgrade\patching of all those components. All you have to do is just press a few buttons and enjoy orchestrated upgrade while playing 2048 game 🙂 .
An example below how Nutanix OneClick upgrade helps avoid bad surprises during AOS upgrade. In below example, the Pre-Upgrade check step detected that some of the data is under replicated and needs to be replicated. In the large majority of cases, trying the upgrade after waiting some time will allow the pre-upgrade check to complete without warnings
Nutanix Cluster is under-replicated
Below is procedure how to quickly solve above error.
NOTE: below procedure is not mandatory, as AOS is self healing
Step #1 – check if it you can see under replication from curator command line.
nutanix@NTNX-15SM65300246-B-CVM:10.4.91.57:~$ curator_cli get_under_replication_info summary=true Using curator master: 10.4.91.56:2010 +------------------------------------------+ | Disk Id | Under replication data (bytes) | +------------------------------------------+ | 35 | 9077780480 | | 46 | 17132949504 | +------------------------------------------+
Step #2 – find out ID for last successful curator scan
nutanix@NTNX-15SM65300246-B-CVM:10.4.91.57:~$ curator_cli get_last_successful_scans 2>/dev/null Using curator master: 10.4.91.56:2010 +----------------------------------+ | Job Name | Full Scan | | Job id | 0 | | Execution id | 421 | | Master handle | 10.4.91.58:2010 | | Incarnation id | 3422 | | Status | 0 | | Start time | Mar 14 04:02:44 | | End time | Mar 14 04:08:09 | +----------------------------------+ +----------------------------------+ | Job Name | Partial Scan | | Job id | 1 | | Execution id | 470 | | Master handle | 10.4.91.56:2010 | | Incarnation id | 437789 | | Status | 0 | | Start time | Mar 14 08:12:50 | | End time | Mar 14 08:14:34 | +----------------------------------+
Step #4 – check where curator muster is running
nutanix@NTNX-15SM65300246-B-CVM:10.4.91.57:~$ curator_cli get_master_location Using curator master: 10.4.91.56:2010 +----------------------------------+ | Master handle | 10.4.91.56:2010 | | Incarnation Id | 437789 | +----------------------------------+ nutanix@NTNX-15SM65300246-B-CVM:10.4.91.57:~$
Step #5 – log in to CVM with curator master and run command to manually triger curator scan
nutanix@NTNX-15SM65300246-B-CVM:10.4.91.56:~$ allssh curl http://localhost:2010/master/api/client/StartCuratorTasks?task_type=2 Executing curl http://localhost:2010/master/api/client/StartCuratorTasks?task_type=2 ================== 10.4.91.56 ================= % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 ================== 10.4.91.57 ================= % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 112 225 112 225 0 0 19488 0 --:--:-- --:--:-- --:--:<html> 0 <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>Title not set</title> </head> <body bgcolor="white" text="black"> <pre>Unable to handle request: kInvalidOperation</pre></body> </html> -- 109k ================== 10.4.91.58 ================= % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 112 225 112 225 0 0 15484 0 --:--:-- --:--:-- --:--:-- 219k <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>Title not set</title> </head> <body bgcolor="white" text="black"> <pre>Unable to handle request: kInvalidOperation</pre></body> </html> nutanix@NTNX-15SM65300246-B-CVM:10.4.91.57:~$
Step #6 – wait 5 – 10 minutes and check again if there is any under replicated data.
nutanix@NTNX-15SM65300246-B-CVM:10.4.91.57:~$ curator_cli get_under_replication_info summary=true Using curator master: 10.4.91.56:2010 +------------------------------------------+ | Disk Id | Under replication data (bytes) | +------------------------------------------+ nutanix@NTNX-15SM65300246-B-CVM:10.4.91.57:~$
[hr]
References
In my experience, “one click upgrade” rarely works as advertised