Upgrading Log Insight to 8.12

I use vRealize Log Insight in my homelab for collecting logs from my vCenter and ESXi hosts, and as a syslog server. Unfortunately it hasn't gotten that much love for some time and it's been stuck on version 8.6 which is over a year old.

That, and of course this KB article explaining an issue with the internal certs expiring on April 30th 2023, had me prioritizing the upgrade.

I have replaced the web server certificate with my own custom certificate issued by my own CA, but still wanted to expedite the upgrade to be on the safe side.

A quick check of the internal cert shows that it expires on the given date

1openssl x509 -noout -in /storage/core/loginsight/cidata/cassandra/config/cacert.pem -enddate

The cert issue is present in all versions prior to 8.12.

When checking the release notes of version 8.12, the supported upgrade path is from 8.10.x

This hade me check the release notes for version 8.10.2 which shows that the supported upgrade path is from 8.8.x

Digging down to the release notes for version 8.8.2 shows finally a supported way to upgrade from my 8.6.0 installation.

So, for me this means, 8.6.0->8.8.2->8.10.2->8.12

After taking a snapshot of the VM and downloading the bits needed from Customer connect I started my upgrade process.

8.6.0 to 8.8.2 upgrade

My first upgrade failed

I did some quick troubleshooting and suspected that the custom cert might be the reason

I noticed that the cert in the log message was the internal self-signed certificate and not my own custom one. Digging further I tried to curl the url from the vRLI VM itself and found that it didn't trust the cert

With that I tried to add the Root CA to the VMs trust store. However when trying to save the Root CA cert I got a message stating that it couldn't be saved due to a full disk.

So after cleaning out some space on the root partition and adding the Root CA cert I retried the upgrade

Again it failed without giving anything more exciting stuff in the upgrade log so I decided to give it a try from the command line.

During that upgrade the script complained about missing 16KB on the root filesystem!

So after cleaning up a bit more I reran the upgrade from the CLI and it finally went through

And the Cluster status page shows the updated version. Note also the new UI layout

8.8.2 to 8.10.2 upgrade

After the successful upgrade to 8.8.2 the next step was upgrading to 8.10.2.

I tried the upgrade through the UI once more, and this time I had no issues

8.10.2 to 8.12.0 upgrade

After letting the instance rest for about an hour I continued with the final step, upgrade to 8.12

Unfortunately this did not succeed on the first try

When checking the update log I found a message with an error on the SSL certificate

I also checked the disk space since that had been the cause on previous upgrades, but this time that seemed fine.

I decided to remove my custom SSL cert by reverting the SSL config to the default as the Release notes mentions some considerations around custom certs

After restarting my browser session I confirmed that the SSL cert was now using the VMware self-signed so I tried the upgrade again, this time through the CLI to fetch any error messages

And score! The vRLI is now Aria Operations for Logs and we are running on version 8.12

Account locked after upgrade

When trying to log in as the admin user I got an error stating that the credentials was invalid or that the account was locked. I waited for about one hour before trying again, but I still got the same message

I confirmed this by checking the admin account status with the CLI and could also reset it

1#Check status
2/usr/lib/loginsight/application/sbin/li-reset-admin-passwd.sh --checkAdminStatus
3
4#Unlock account
5/usr/lib/loginsight/application/sbin/li-reset-admin-passwd.sh --unlockAdmin

And finally I could log in with my admin account and verify the operation of the vRLI Aria Operations for Logs status

Addin a new SSL certificate

Final step in the upgrade process was to change the default SSL certificate to my own custom CA signed cert.

Since the upgrade log mentioned some field not found on the existing one I decided to create a new one by following the steps found in the VMware documentation

I got an issue with getting the server to accept the certificate because of "No SSL client purpose set", which is the same kind of error that blocked the upgrade in the first place.

After digging around a bit I found a way to update the certificate, and since this might occur also outside of upgrades I decided to write a blog post on that specifically. In short the issue was that the Certificate template I had used was missing the "Client Authentication" policy which at the time of this writing is not listed as a requirement to the certificate.

After adding this my custom cert could be uploaded.