Performance issues HP Microserver gen8 and VMWARE EXSI 6.5

upgraded ESXI to 6.5 U1 which is version 6.5.0 build 5969303

You are running HPE Customized Image ESXi 6.5.0 version 650.10.1.0.47 released on July 2017 and based on ESXi 6.5.0 Vmkernel Release Build 5310538

unfortunately the HP Microserserver Gen8 is than running with: HPE_bootbank_scsi-hpvsa_5.5.0.102-1OEM.550.0.0.1331820

As I was (and still am but as of writing I am in maintenance mode) seeing spikes in my CPU usages. It might be caused by the bad performance

checking with:
cd /vmfs/volumes/datastore1
time dd if=/dev/zero of=tempfile bs=8k count=1000000

It took a very long time to see some output. (Very long time!)

than I tried to downgrade to the hpvsa-5.5.0-88.zip driver

guidelines:

  1. enter maintenance mode
  2. I do a reboot, but you can do it probably without
  3. copy the downloaded driver to /tmp/ and run the following command
  4. esxcli software vib install -d /tmp/hpvsa-5.5.0-88.zip
  5. the old driver will be removed and the 5.5.0-88 driver installed
  6. Now important: if you do reboot this way: you will not see your DataStores anymore, only your NFS datastores (in my situation) this is caused by VMWARE ESXI as it will be using vmw_ahci driver for the datastore.
  7. so disable the usage of this ‘default’ driver: esxcli system module set –enabled=false –module=vmw_ahci
  8. now you can reboot and your device will be using the 5.5.0-88 driver
  9. you can see this by using; cat /proc/driver/hpvsa/hpvsa0

update (taken from: https://homeservershow.com/forums/topic/9602-esxi-notes/)

When installing VIB you first need to un-install the old driver

How to downgrade the hpvsa driver in 5.5 and 6.0
The b120i driver from 5.5.0-90.0 to 5.5.0.98-1 has performance issues and requires a downgrade to 5.5.0-88.0. Newer versions may still be affected so check the forum or this thread for more details.

Download scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib from the HP VIB depot and upload it to your datastore. It is the same driver for 5.5 and 6.0.

Enter maintenance mode and run the following command. You’ll need to reboot after changing the driver and then leave maintenance mode. Do NOT restart without reinstalling the driver after removing it.

important: remove the old driver !!

esxcli software vib remove -n Hewlett-Packard:scsi-hpvsa

install the NEW driver:

esxcli software vib install –viburl=/vmfs/volumes/[NAME OF DATASTORE]/scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib

If you get an error saying bootbank cannot be live installed
The error will be similar to the following:

The transaction is not supported:
VIB VMware_bootbank_esx-base cannot be live installed.
VIB VMware_bootbank_esx-base cannot be removed live.
The bootbank is the system image that ESXi loads into a ramdisk and boots off.
It is only used during startup and during updates.

The error is generally due to two things.
First is corruption and second is if the boot device is not accessible for some reason.
On the Gen8 Microserver, doing an iLO update whilst the system is running may cause the micro-SD reader to temporarily stop working.
The solution is to shutdown the host, unplug the machine for 5 seconds, plug it back in and attempt the update again

the speed should have been returned.