Performance issues HP Microserver gen8 and VMWARE EXSI 6.5

upgraded ESXI to 6.5 U1 which is version 6.5.0 build 5969303

You are running HPE Customized Image ESXi 6.5.0 version 650.10.1.0.47 released on July 2017 and based on ESXi 6.5.0 Vmkernel Release Build 5310538

unfortunately the HP Microserserver Gen8 is than running with: HPE_bootbank_scsi-hpvsa_5.5.0.102-1OEM.550.0.0.1331820

As I was (and still am but as of writing I am in maintenance mode) seeing spikes in my CPU usages. It might be caused by the bad performance

checking with:
cd /vmfs/volumes/datastore1
time dd if=/dev/zero of=tempfile bs=8k count=1000000

It took a very long time to see some output. (Very long time!)

than I tried to downgrade to the hpvsa-5.5.0-88.zip driver

guidelines:

  1. enter maintenance mode
  2. I do a reboot, but you can do it probably without
  3. copy the downloaded driver to /tmp/ and run the following command
  4. esxcli software vib install -d /tmp/hpvsa-5.5.0-88.zip
  5. the old driver will be removed and the 5.5.0-88 driver installed
  6. Now important: if you do reboot this way: you will not see your DataStores anymore, only your NFS datastores (in my situation) this is caused by VMWARE ESXI as it will be using vmw_ahci driver for the datastore.
  7. so disable the usage of this ‘default’ driver: esxcli system module set –enabled=false –module=vmw_ahci
  8. now you can reboot and your device will be using the 5.5.0-88 driver
  9. you can see this by using; cat /proc/driver/hpvsa/hpvsa0

update (taken from: https://homeservershow.com/forums/topic/9602-esxi-notes/)

When installing VIB you first need to un-install the old driver

How to downgrade the hpvsa driver in 5.5 and 6.0
The b120i driver from 5.5.0-90.0 to 5.5.0.98-1 has performance issues and requires a downgrade to 5.5.0-88.0. Newer versions may still be affected so check the forum or this thread for more details.

Download scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib from the HP VIB depot and upload it to your datastore. It is the same driver for 5.5 and 6.0.

Enter maintenance mode and run the following command. You’ll need to reboot after changing the driver and then leave maintenance mode. Do NOT restart without reinstalling the driver after removing it.

important: remove the old driver !!

esxcli software vib remove -n Hewlett-Packard:scsi-hpvsa

install the NEW driver:

esxcli software vib install –viburl=/vmfs/volumes/[NAME OF DATASTORE]/scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib

If you get an error saying bootbank cannot be live installed
The error will be similar to the following:

The transaction is not supported:
VIB VMware_bootbank_esx-base cannot be live installed.
VIB VMware_bootbank_esx-base cannot be removed live.
The bootbank is the system image that ESXi loads into a ramdisk and boots off.
It is only used during startup and during updates.

The error is generally due to two things.
First is corruption and second is if the boot device is not accessible for some reason.
On the Gen8 Microserver, doing an iLO update whilst the system is running may cause the micro-SD reader to temporarily stop working.
The solution is to shutdown the host, unplug the machine for 5 seconds, plug it back in and attempt the update again

the speed should have been returned.

 

The hassle of upgrading ESXI 6.0 to ESXI 6.5 on a HP Microserver gen8

Wauw ..

yesterday and today I tried to upgrade my HP Microserver Gen8 from VMWARE ESXI 6.0.0 to 6.5
what a trouble ..

Steps:
shut down all vms (hosts) and enter maintenance mode. If you do this there are a number of online blogs with help to install from online depots but in all my tests it was too slow or I was to impatient to wait to end. But in the end I always ended up with an system with errors:

the transaction is not supported: VIB Hewlett-Packard_bootbank_scsi-hpvsa_5.5.0-88OEM.550.0.0.1331820

Do not try to force the installation. You will end up with an system where it seems that your EXSI is updated to 6.5 but actually is is running in a ramdisk environment. Easy to see cause you have lost your datastores (NFS datastores are still mounted).

If you than reboot: you are back to 6.0.0 .. so how to solve this:

easy: when entering maintenance mode: reboot your machine. In many guidelines this part is not mentioned.

After I rebooted I first tried the online depot installation documentation but ended up waiting and waiting. I got a VMWARE image for update from VMWARE but I got all kinds of different issues again:

“The upgrade contains the following set of conflicting VIB” When using the standard image of VMWARE: many conflicting vibs a no go for me. So I read some blogs and I found out that it is best to keep using your HPE image vmware files. (So in short: if you used the HPE VMWARE ESXI ISO installing Esxi onto your HP Microserver Gen8 keep using the update files with HPE in it and not the plain VMWARE onces. It can give issues !

After I used the HPE image I only got 1 issue: one vib was still complaining. On this blog I read what I needed to do: remove this vib (partner supported, so not native).

After removing this VIB I could upload the various ZIP bundle files like

VMware-ESXi-6.5.0-5310538-HPE-650.10.1.0.47-Jul2017-depot.zip
VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.0.14-Jul2017-depot.zip

with the command:
esxcli software vib install -d “/vmfs/volumes/datastore1/patch-directory/VMware-ESXi-6.5.0-5310538-HPE-650.10.1.0.47-Jul2017-depot.zip”

I was able to install finally the 6.5 version. Due to issues of Storage driver I immediately updated to the U1 update
with the command:

esxcli software vib update -d “/vmfs/volumes/datastore1/patch-directory/VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.0.14-Jul2017-depot.zip”

the blog nxhut showed me some info that with the U1 version the storage speed performance issues should be over.

A simple test downloading a file towards a VM showed me a 18MB/s so the 10MB/s barrier was not seen.

A last reboot and get the machine out of maintenance mode. Finally I have a running 6.5U1.