Exploring monitoring endpoints in the vCenter Server Appliance (VCSA) REST API

For a long time, actually since we migrated to the VCSA in 6.5 last year, I've wanted to utilize the REST API in the appliance to have some monitoring of them.

For several reasons I've had to put that on hold, one of them being that there seems to be something wrong with the back-end authentication calls. I get authentication errors on certain calls no matter which user I am logged in with (also the vsphere.local admin account).

{
  "type": "com.vmware.vapi.std.errors.unauthenticated",
  "value": {
    "messages": [
      {
        "args": [],
        "default_message": "Unable to authenticate user",
        "id": "vapi.security.authentication.invalid"
      }
    ]
  }
}

After deciding to check it more closely I eventually found a few errors in the VCSA logs which in turn led me to this article by Ryan Harris. This error is exactly the same as I have in my environment so I tried the solution given to Ryan by VMware support and this also worked for me. I have not found any mentions on this error in the Release notes / Known issues of the VCSA so I do think the workaround might be overwritten when updating the appliance but I'll worry about that later.

If you have the same error please make sure you have** backups of your VCSA, and consider contacting support before implementing the fix, especially in a production environment.**

With a working API we can now explore the VCSA REST API.

I'll use Powershell as my REST client. Please note that you can also use the PowerCLI modules for this (CiSServer cmdlets), check out William Lam's blog series for more info.

To start of I'll build a couple of variables for the base REST API Url and the specific URL for authenticating.

$vcenter = "vcenterserver.your.domain"
$BaseUri = "https://$vcenter/rest/"
$SessionUri = $BaseUri + "com/vmware/cis/session"

Now, I'll create a credential object which I will then create an authentication header from.

$Cred = Get-Credential
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Cred.UserName+':'+$Cred.GetNetworkCredential().Password))
$header = @{
  'Authorization' = "Basic $auth"
}

With that in place I can now login to create a session key which I'll store in a variable for use in a new header which will be used in all subsequent API calls

$authResponse = (Invoke-RestMethod -Method Post -Headers $header -Uri $SessionUri).Value
$sessionHeader = @{"vmware-api-session-id" = $authResponse}

Now I can play around with the API just as I can through the API Explorer (https://vcenterserver.your.domain/apiexplorer)

Let's first check out the health endpoints.

PS C:\> $healthEndpoint = Invoke-Restmethod -Method Get -Headers $sessionHeader -Uri ($BaseUri + "appliance/health/applmgmt")
$healthEndpoint.value
green

As we can see the different health endpoints returns a string value. While this is ok for a quick status it would be nice if we can read out the actual performance values. To do this we'll use the appliance/monitoring endpoints.

To check the monitoring instances available we'll start by querying the montioring endpoint

PS C:\> $metrics = Invoke-RestMethod -Method Get -Headers $sessionHeader -Uri ($BaseUri + "appliance/monitoring")
$metrics.value | select -First 1


instance    : eth0
name        : com.vmware.applmgmt.mon.name.net.rx.activity.eth0
description : com.vmware.applmgmt.mon.descr.net.rx.activity.eth0
units       : com.vmware.applmgmt.mon.unit.kb_per_sec
id          : net.rx.activity.eth0
category    : com.vmware.applmgmt.mon.cat.network

I've only outputted the first result, but as we can see we get some information about the different instances, what they are and their ID which we'll use later on when querying specific resources.

One thing to note is that as we are working with Powershell here and as Powershell is best-friends with JSON objects we can easily output just the different names of the available metrics/resources:

PS C:\> $metrics.value | select name

name
----
com.vmware.applmgmt.mon.name.net.rx.activity.eth0
com.vmware.applmgmt.mon.name.cpu.util
com.vmware.applmgmt.mon.name.net.tx.activity.lo
com.vmware.applmgmt.mon.name.storage.totalsize.filesystem.vcdb_seat
com.vmware.applmgmt.mon.name.storage.totalsize.filesystem.root
com.vmware.applmgmt.mon.name.storage.used.filesystem.netdump
com.vmware.applmgmt.mon.name.storage.used.filesystem.invsvc
com.vmware.applmgmt.mon.name.storage.used.filesystem.root
com.vmware.applmgmt.mon.name.storage.used.filesystem.log
com.vmware.applmgmt.mon.name.storage.totalsize.file
...

In totalt we have 43 metrics available

PS C:\> $metrics.value.count
43

Let's try to query a specific metric and check the results of that. For this to work we need to know how to work with the parameters of the API. This is not possible through the API Explorer so you might need to experiment in your own environment first. While you get some information on how it works in the API Explorer be sure to check out the vdc-repo from VMware which has more details for each endpoint.

The API endpoint takes a few parameters that all need to be included. The interval, names, function, start and end time: Interval defines the granularity/interval of the returned values. It ranges from 5 minutes to 24 hours. Check the vdc-repo mentioned above for details. The names parameter is the actual metrics that you want to retrieve. We can query multiple by adding a sequential number to the parameter (item.names.1, item.names.2 etc). Function specifies the aggregation function, can be COUNT, AVG, MAX, MIN Finally _start- _and end_time specifies the range of the query, both must be filled.

We'll use the "CPU Util" metric in this example and query for 5 minute interval data inside a 10 minute timeperiod.

PS C:\> $uri = $BaseUri + "appliance/monitoring/query?item.interval=MINUTES5&item.names.1=cpu.util&item.function=COUNT&item.start_time=2018-08-13T12:50:00.000Z&item.end_time=2018-08-13T12:59:00.000Z"
$response = Invoke-RestMethod -Method Get -Headers $sessionHeader -Uri $uri
$response.value


start_time : 2018-08-13T12:50:00.000Z
data       : {5, 4}
function   : COUNT
name       : cpu.util
end_time   : 2018-08-13T12:59:00.000Z
interval   : MINUTES5

The result shows 2 values in the data attribute, which corresponds to the 5 minute interval we requested. Note that we've used COUNT as the function which in this case should be the SUM inside that interval. If we instead ask for the AVG we'll get the following output

PS C:\> $uri = $BaseUri + "appliance/monitoring/query?item.interval=MINUTES5&item.names.1=cpu.util&item.function=AVG&item.start_time=2018-08-13T12:50:00.000Z&item.end_time=2018-08-13T12:59:00.000Z"
$response = Invoke-RestMethod -Method Get -Headers $sessionHeader -Uri $uri
$response.value


start_time : 2018-08-13T12:50:00.000Z
data       : {2.66247797254, 2.79290581324}
function   : AVG
name       : cpu.util
end_time   : 2018-08-13T12:59:00.000Z
interval   : MINUTES5

And with MAX as the function

PS C:\> $uri = $BaseUri + "appliance/monitoring/query?item.interval=MINUTES5&item.names.1=cpu.util&item.function=MAX&item.start_time=2018-08-13T12:50:00.000Z&item.end_time=2018-08-13T12:59:00.000Z"
$response = Invoke-RestMethod -Method Get -Headers $sessionHeader -Uri $uri
$response.value


start_time : 2018-08-13T12:50:00.000Z
data       : {3.61491764901, 3.2818270841}
function   : MAX
name       : cpu.util
end_time   : 2018-08-13T12:59:00.000Z
interval   : MINUTES5

With this we have a nice starting point for building out a monitoring solution for our VCSA's. More on that in a later post. Stay tuned!