Issues with Get-Spbm* PowerCLI cmdlets and vCenter 6.5

We are using Storage Policies on all VMs in our environment and rely on those to give the VM's the correct storage. Storage Policy based management is a key area for VMware going forward it seems, especially with things like vSAN and VVOL.

In addition to matching a VM and it's storage we are using the policies to set limits on the VM's. We have a shared environment and we have to have some limitation on the VM's so that one VM couldn't impact all the others. To ensure this we have a couple of scripts that retrieves all VM's and their corresponding policy and based on the policy naming convention we are setting IOPS limit and the Bandwidth Cap for that VM. The scripts has been working for a long time, aside from some bugs with the limits it self which VMware support is looking into.

After upgrading and migration some of our vCenters from Windows vCenter Server 6.0 U2 to the vCenter Server Appliance 6.5 it seems we broke the Get-SpbmStoragePolicy cmdlet. PowerCLI is updated to the latest available version as well through the PowershellGallery.

What's strange is that when issuing the Get-SpbmStoragePolicy to a connected vCenter server nothing happens. It just sits there "forever".

There is a TCP connection from the Powershell process to the vCenter and there is some network traffic going, but not a lot..

We have one vCenter left on Windows. The same command from the same machine works perfectly. I can't figure out what's wrong and have filed a Support Request with VMware to see if they can figure it out.

Update!

Before I filed the SR I also posted the issue on VMware communities PowerCLI forum. The Powershell MVP and PowerCLI guru LucD thought it seemed like a bug and thought I should file that SR to have VMware look at it.

VMware support couldn't replicate the issue in their lab and told me they didn't support issues with PowerCLI. I posted this on the forum and LucD didn't agree with VMware not supporting PowerCLI and would take it to the PowerCLI PM. Thanks Luc!

Anyways I continued troubleshooting this as this is kind of key to us keeping our environment healthy.

I tested the cmdlet from several Windows 2012 R2 servers which all showed the same behavior against the 6.5 VCSAs. I was using the latest PowerCLI available PowerCLI 6.5.1 build 5377412, downloaded from the Powershell Gallery.

I also tested with PowerCLI 5.8 and 6.0 build 2548067. With these two I get some SOAP errors and I found some blog posts about a TLS change which I think is related.

The support tech at VMware used a slightly older PowerCLI version than me, 4624819, so I tested that as well. Same result (or lack of result..).

When testing against a lab vCenter not available from the Windows servers mentioned I tested with a Windows 10 laptop with the latest PowerCLI version.

Here the command would produce an error!

This pointed me in direction of some regional settings fun. We are using Norwegian settings which has a date format of: dd.MM.yyyy

So I changed the date format on the laptop to English US and VOILA!

There was our Storage policies!

I went over to one of the 2012 servers and did the same there just to confirm. Surprisingly this “fix” didn’t work there..

I checked further and found that the 2012 servers and the Win10 laptop had different Powershell versions

Then I found a newly built Windows 2016 server and checked it's PS version

This is the same version as the laptop!

I downloaded and tested the PowerCLI cmdlets from this server and there was the dateformat issue again. I quickly changed to EN-US and sure enough the cmdlet worked on that server.

I then built a new 2016 server to confirm my findings. With my default Norwegian regional settings I get the dateformat error. So I changed the settings to English US and expected things to work, but no luck. Same behavior as before, nothing happened.

I compared PS versions and PowerCLI versions between the two 2016 servers – everything was the same. I had a third 2016 server which was built on the same time as the first one that worked so I tried this one as well only to find that this didn’t work either.

Then I did everything on step at a time on all three servers to see if I could figure out what was going on. I noticed that on the server that worked I had previously fixed the PowerCLIConfiguration, specifically I had set ParticipateInCEIP to false and set the InvalidCertificateAction to Ignore. On the two servers this was not done.

So I did the same on my newly built 2016 server and to my surprise that did the trick!

To confirm I tested the same on the third server. First I set the InvalidCertificateAction to Ignore as this was my prime suspect.

To confirm I put my regional settings back to Norwegian to check the dateformat issue, but then I got the error.

I also tested on my 2012 servers, and got it to work there as well.

And with that I had found my workaround.

To summarize, this is what worked for me:

• Set InvalidCertificateAction to Ignore (would probably work with a valid cert as well which of course would be the best practice)
• Set the dateformat to English US

I've updated the VMware support request with my findings so hopefully they will provide a fix for this soon.