Fixing Proxmox boot ending in grub prompt with ZFS disks

I have removed my outdated how-to as it was not functional for my proxmox version (6.4x) during a planned reboot on 5-7-2021 .. I am lucky I did not had a power outtage earlier as that would have given me more troubles ..

Proxmox has made a great tool: proxmox boot tool .. and following the guidelines in the link below gave me a working proxmox in only minutes (it took some hours before I had a USB thumbdrive working (one defect one) ..

But there is a great how to here

Important parts

Repairing a System Stuck in the GRUB Rescue Shell

If you end up with a system stuck in the grub rescue> shell, the following steps should make it bootable again:

Boot using a Proxmox VE version 6.4 or newer ISO
Select Install Proxmox VE (Debug Mode)
Exit the first debug shell by typing Ctrl + D or exit
The second debug shell contains all the necessary binaries for the following steps
Import the root pool (usually named rpool) with an alternative mountpoint of /mnt:
```
zpool import -f -R /mnt rpool
```
Find the partition to use for proxmox-boot-tool, following the instructions from Finding potential ESPs
Bind-mount all virtual filesystems needed for running proxmox-boot-tool:

mount -o rbind /proc /mnt/proc

mount -o rbind /sys /mnt/sys

mount -o rbind /dev /mnt/dev

mount -o rbind /run /mnt/run
change root into /mnt
```
chroot /mnt /bin/bash
```
Format and initialize the partitions in the chroot – see Switching to proxmox-boot-tool
Exit the chroot-shell (Ctrl + D or exit) and reset the system (for example by pressing CTRL + ALT + DEL)
Note: The next boot can end up in an initramfs shell, due to the hostid mismatch (from importing the pool in the installer).
If this is the case, simply import it again with using the force -f flag:
```
# zpool import -f rpool
```

This first part gives you the start to fix grub again

Find the partitions:

lsblk -o +FSTYPE

(in my situation) with the vfat > 512MB

format it again

# proxmox-boot-tool format /dev/sda2
# proxmox-boot-tool format /dev/sdb2

NB. I used the –forced option as there is data on the vfat

the init part

# proxmox-boot-tool init /dev/sda2
# proxmox-boot-tool init /dev/sdb2

and if needed (what was the case in my situation)

# proxmox-boot-tool clean

Reboot and I was up and running again with my node in my cluster ..

The next GUIDE to install NGINX-PROXY-MANAGER and not having bad gateway database issues

Before reading: the main reason why this nginx-proxy-manager was not running in my environment was the fact that I was running my Linux version as LXC under Proxmox and not as a VM under proxmox. After failing a 2nd time with exact the same config files what was working I noticed that I was using LXC and that the Proton VM was actually a VM, by changing it to a normal Debian VM I was able to get a working version fast again.

This is also probably the reason that Portainer was not able to start the database as well. So in the end: using a VM ..

I was reading:
another website that was telling me how to install nginx-proxy-manager. But I failed. I kept getting ‘ bad gateway’ and if you read the github posts about this issue you will not understand why all is failing.

So yes, I did install proton VM, a sucking virtual machine under my proxmox as it was used by the guy from that other website. As docker is available I had to start it during boot. Those guidelines were described fine. But installing my own mysql or mariadb was failing time after time. Especially as mariadb or mysql was not having a root password. So I failed. Buy why?

So in the end (lucky I had a snapshot, so that I could go back when messing some things really bad up. I restarted the machine and thought about what I read on another website: nginx-proxy-manager is ‘now’ providing a mysql instance itself. AHA .. so if that is true than I have to forget all info about previous own installed docker stuff with databases. So i removed those failures from the system.

I checked the website of nginx-proxy-manager and thought: let start over ..

In the end to make this story short

I made sure the server pointed to “host”: “127.0.0.1”, in the config.json

make sure there is a config.json
place this config.json where you use the ‘docker-compose up -d’
I did it in /home/nginx-proxy-manager/

And probably here is the catch as the default example is telling

 # Make sure this config.json file exists as per instructions above:
      - ./config.json:/app/config/production.json

the /app/config/production.json is a location where you did not put your own config.json. So this part is totally wrong. So the config.json with your database settings can never be found, so you get issues, but the ‘ make sure this config.json exisist as per instructions above ‘ gave me no clue, cause what is stated above?

So I tried what I did before in the docker-compose.yml I changed the location of my config.json to

-./config.json:/home/nging-proxy-manager/config.json

now I restarted the docker container again but I made an error the container was started with docker-compose up without the -d (DAEMON) … so I got output in my screen and suddenly I saw that there was a connection to the database but my password was not accepted.

I made sure I shut down the docker container again, removed the contents from the directories and restarted it again .. YEAAAHHH .. finally .. it was working

In short, 2 things to notice

config.json: change the host part to: “host”: “127.0.0.1”,
in the docker-compose.yml pinpoint the config.json to the actual location on your HDD wher you put it.

Now start with ie. docker-compose up -d

have fun