Performance issues HP Microserver gen8 and VMWARE EXSI 6.5

upgraded ESXI to 6.5 U1 which is version 6.5.0 build 5969303

You are running HPE Customized Image ESXi 6.5.0 version 650.10.1.0.47 released on July 2017 and based on ESXi 6.5.0 Vmkernel Release Build 5310538

unfortunately the HP Microserserver Gen8 is than running with: HPE_bootbank_scsi-hpvsa_5.5.0.102-1OEM.550.0.0.1331820

As I was (and still am but as of writing I am in maintenance mode) seeing spikes in my CPU usages. It might be caused by the bad performance

checking with:
cd /vmfs/volumes/datastore1
time dd if=/dev/zero of=tempfile bs=8k count=1000000

It took a very long time to see some output. (Very long time!)

than I tried to downgrade to the hpvsa-5.5.0-88.zip driver

guidelines:

  1. enter maintenance mode
  2. I do a reboot, but you can do it probably without
  3. copy the downloaded driver to /tmp/ and run the following command
  4. esxcli software vib install -d /tmp/hpvsa-5.5.0-88.zip
  5. the old driver will be removed and the 5.5.0-88 driver installed
  6. Now important: if you do reboot this way: you will not see your DataStores anymore, only your NFS datastores (in my situation) this is caused by VMWARE ESXI as it will be using vmw_ahci driver for the datastore.
  7. so disable the usage of this ‘default’ driver: esxcli system module set –enabled=false –module=vmw_ahci
  8. now you can reboot and your device will be using the 5.5.0-88 driver
  9. you can see this by using; cat /proc/driver/hpvsa/hpvsa0

update (taken from: https://homeservershow.com/forums/topic/9602-esxi-notes/)

When installing VIB you first need to un-install the old driver

How to downgrade the hpvsa driver in 5.5 and 6.0
The b120i driver from 5.5.0-90.0 to 5.5.0.98-1 has performance issues and requires a downgrade to 5.5.0-88.0. Newer versions may still be affected so check the forum or this thread for more details.

Download scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib from the HP VIB depot and upload it to your datastore. It is the same driver for 5.5 and 6.0.

Enter maintenance mode and run the following command. You’ll need to reboot after changing the driver and then leave maintenance mode. Do NOT restart without reinstalling the driver after removing it.

important: remove the old driver !!

esxcli software vib remove -n Hewlett-Packard:scsi-hpvsa

install the NEW driver:

esxcli software vib install –viburl=/vmfs/volumes/[NAME OF DATASTORE]/scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib

If you get an error saying bootbank cannot be live installed
The error will be similar to the following:

The transaction is not supported:
VIB VMware_bootbank_esx-base cannot be live installed.
VIB VMware_bootbank_esx-base cannot be removed live.
The bootbank is the system image that ESXi loads into a ramdisk and boots off.
It is only used during startup and during updates.

The error is generally due to two things.
First is corruption and second is if the boot device is not accessible for some reason.
On the Gen8 Microserver, doing an iLO update whilst the system is running may cause the micro-SD reader to temporarily stop working.
The solution is to shutdown the host, unplug the machine for 5 seconds, plug it back in and attempt the update again

the speed should have been returned.

 

The hassle of upgrading ESXI 6.0 to ESXI 6.5 on a HP Microserver gen8

Wauw ..

yesterday and today I tried to upgrade my HP Microserver Gen8 from VMWARE ESXI 6.0.0 to 6.5
what a trouble ..

Steps:
shut down all vms (hosts) and enter maintenance mode. If you do this there are a number of online blogs with help to install from online depots but in all my tests it was too slow or I was to impatient to wait to end. But in the end I always ended up with an system with errors:

the transaction is not supported: VIB Hewlett-Packard_bootbank_scsi-hpvsa_5.5.0-88OEM.550.0.0.1331820

Do not try to force the installation. You will end up with an system where it seems that your EXSI is updated to 6.5 but actually is is running in a ramdisk environment. Easy to see cause you have lost your datastores (NFS datastores are still mounted).

If you than reboot: you are back to 6.0.0 .. so how to solve this:

easy: when entering maintenance mode: reboot your machine. In many guidelines this part is not mentioned.

After I rebooted I first tried the online depot installation documentation but ended up waiting and waiting. I got a VMWARE image for update from VMWARE but I got all kinds of different issues again:

“The upgrade contains the following set of conflicting VIB” When using the standard image of VMWARE: many conflicting vibs a no go for me. So I read some blogs and I found out that it is best to keep using your HPE image vmware files. (So in short: if you used the HPE VMWARE ESXI ISO installing Esxi onto your HP Microserver Gen8 keep using the update files with HPE in it and not the plain VMWARE onces. It can give issues !

After I used the HPE image I only got 1 issue: one vib was still complaining. On this blog I read what I needed to do: remove this vib (partner supported, so not native).

After removing this VIB I could upload the various ZIP bundle files like

VMware-ESXi-6.5.0-5310538-HPE-650.10.1.0.47-Jul2017-depot.zip
VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.0.14-Jul2017-depot.zip

with the command:
esxcli software vib install -d “/vmfs/volumes/datastore1/patch-directory/VMware-ESXi-6.5.0-5310538-HPE-650.10.1.0.47-Jul2017-depot.zip”

I was able to install finally the 6.5 version. Due to issues of Storage driver I immediately updated to the U1 update
with the command:

esxcli software vib update -d “/vmfs/volumes/datastore1/patch-directory/VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.0.14-Jul2017-depot.zip”

the blog nxhut showed me some info that with the U1 version the storage speed performance issues should be over.

A simple test downloading a file towards a VM showed me a 18MB/s so the 10MB/s barrier was not seen.

A last reboot and get the machine out of maintenance mode. Finally I have a running 6.5U1.

 

If you want to connect to your local server what is behind openvpn and yourself are connected with a VPN to your local network

Yeah .. wierd title

So in my case: I am at location A and I am making a VPN to my local network what is at location B

I can access all my local servers and computers, but one !

“The One” is a server which has a dedicated openvpn connection to location C, when this is active my VPN from location A cannot access “The One” at location B. Only when the OpenVPN connection is terminated

Solution: add a static route on “The One”

check with: ip route show

and add the route like this: ip route add 10.10.10.0/24 via 10.1.1.1 dev eth0

where 10.10.10.0 is the VPN IP range what your local network is giving VPN users and via 10.1.1.1 is the IP of your gateway (router)

… easy peacy ..

Google vind iets .. probeer dat dan maar eens recht te zetten

Google zegt dat deze site mogelijk gehacked is ..

en laat dat nu eens niet het geval zijn

Deze site is mogelijk gehackt., zegt Google. Ja maar …. en dan ga je zoeken, testen en je weet me god niet waar je moet zoeken. Want .. tsja je krijgt wel een link, maar als je link naar de weblog sectie van deze site verwijst dan staan daar een x aantal berichten onder met wat ‘technisch’ karakter, maar dat is het dan wel .. heb dus totaal geen idee waar ik moet zoeken of wel stukje code wat ik getikt hebt een trigger is voor Alphabet’s Google ..

Mocht je iets vreemds bemerken, ik hoor het wel, tot die tijd blijven we ons afvragen wat er mis zou zijn .. (Ik heb het theme is al veranderd .. ) dus een theme hack sluit ik even uit

 

update: Na eerst 1 pagina aan te merken als iets wat niet goed zou zijn, vond Google dat dan de gehele site wel even geïnfecteerd was. Maar echte duiding wat er dan mis is: nope ..

Dus ..

update: diep in de spelonken van dit weblog was wel wat aan de hand met wat gemodificeerde bestanden maar zou niet zo 1-2-3 kunnen zien of dat Google zou signaleren .. maar goed , WordPress maar even waar nodig geheel verwijderd en opnieuw geïnstalleerd zodat er geen ‘historie’ van oude bestanden meer aanwezig was.

Tevens een aantal folders die niet meer gebruikt werden ook verwijderd zodat dat ook geen belemmering zou kunnen geven ..Zie hier . site is weer ok .. zegt Google dan ;)

My First PHP: een website met data van Schiphol

Nadat ik las dat Schiphol data van vertrekkende en landende vliegtuigen beschikbaar ging maken dacht ik: laat ik eens kijken of ik deze data ook kan verwerken in een website.

Na een paar weken stoeien is tie dan als een soort van 1st release beschikbaar: AroundMyRoom’s Schiphol page

Vooral het het begin lopen te stoeien hoe ik de JSON data kom omzetten in arrays en objecten. Met wat hulp van Tweakers.net kwam ik daar wat verder mee.  Daarna liep ik best wel vast met wat for each loops en daar heeft Peter aka zaph.nl mijn mee kunnen helpen.

Op het laatst ging het bijna mis met de checkin balies, want ik wilde wat meer dan wat de API makkelijk mij kon voorschotelen. Ik maak namelijk een verschil tussen normale checkin-balies en andere soorten. Ik had het op basis van Type kunnen doen, maar ik koos onbewust voor de beschrijving waarbij hier vele verschillende schrijfwijzen voorbij kwam.  Door niet op te geven en door te blijven zoeken kwam ik wat functies tegen die ik kon gebruiken. De code is zeer zeker niet geoptimaliseerd, maar hey, het werkt ..

Wat ik als wens heb ik dat ik data wat sneller van de API zou kunnen krijgen / voorschotelen. Wellicht data tijdelijk in een DB stoppen zodat de data snel gepresenteerd kan worden. ..

Message to self: No password in Access when there is an ODBC connection to an SQL server needed

Message to self: Important

When you create a userspace alike ODBC connection in Windows 2008 Server (whatever) it is likely that Access does not allow you to use linked tables as it is requiring a password.
Although you have the ability to store passwords in Access when linking tables through ODBC it is not always working when the ODBC connection is set in the field ‘user’ rather than Machine DataSource/System

The trick is to execute as admin the file c:\windows\SysWOW64\odbc32.exe
Then create the System ODBC connection towards the SQL database, return to your Access file and link the tables (and keep in mind to save the password as well when linking)

I tried above multiple times when the ODBC was in userspace or was (probably) not correctly created (32 versus 64 bits??)
Possible it can also be related that the application pool is 32bits rather then 64bits and the ODBC connection needs 32bits as well .. Hell: if this occurs again, you have some tips ;_)

Edgemax Lite with local domain issues through VPN

I was using a dnsmasq DHCP/DNS server but that version broke due to an software update where it was running. So I changed towards the DHCP and DNS of Ubiquity Edgemax Lite

The issue I had was that local domainnames (FQDN) could not be looked up by its name but only through IP. In the old situation where not the router was the DNS server it was working but in the new environment not. I changed the DNS server to the correct local IP for the local hostnames, but no success.

Than I found a hint in a forum

open a SSH session towards the Edgemax

configure
show service dns

forwarding {
cache-size 400
listen-on eth1
listen-on eth2
name-server 10.1.1.1
name-server 8.8.8.8
name-server 8.8.4.4
system
}

Above is the non working environment for VPN (for all other situations it was ok)

the local name-server was defined but it was probably not available through L2TP VPN

The solution was to add

options listen-address=10.1.1.1 with this command:
configure
set service dns forwarding options "listen-address=x.x.x.x"
commit
save

where x.x.x.x is the IP of the router. It should be the IP address that PPTP/L2TP is returning to the clients for DNS server