Windows Server 2022 Hyperconverged cluster sizing and performance – S2D IOPS

I as most of other IT pros have always been a great fan of Vmware, the reason has been it stability, simplicity, features and price. I guess the price and customer adaption are mainly the main drivers that boosted Vmware from becoming the most sold or largest virtualization provider at least for on-prem scenarios. Ever since 2008 R2, Microsoft has been trying to compete with Vmware without really being able to be right up there. But at the same time, Microsoft has done a pretty darn good job with Hyper v. I see a lot of customers are using Hyper v for non-critical production environments as for example management systems being one of top of the list. The feedback is almost always the same, that Hyper v works very well, but it you must use it with SCVMM (Systemcenter virtual machine manager) it becomes a complex task. Not only must you have to manage and maintain a Vmware infrastructure, but you also require to maintain SCVMM. The good news is the introduction of Admincenter! Ever since it was introduced as being the Hanalulu project it has been a great addition to IT pro’s/admins toolkit. A lot of customers are using admincenter and managing Hyper v standalone servers as well as clusters. Hyper-converged cluster are also a very popular flavor amongst SMB customers. I too personally have been like most Microsoft customers, sitting there on the fence and using Hyper v to some extent but not fully utilizing its potential, until Admincenter became the game changer.

Thinking about the licensing for Vmware one has to ask himself if it really is worth the price! Or is it that, because you first are a Vmware shop, you are bound to be trapped into the ever increasing costs. The prices keep on rising, and features are being moved more and more to enterprise licensing side. Which means, if you are going to use vSAN, you are paying for Vmware, vSAN, but if you are going to use strech cluster or DRS you have to license everything as being enterprise edition software. On top of that you still have to pay Windows server licenses. God forbid that you get anything chip in the tech greedy era! Anyhow if you are a person or company which wants to implement SDN (software defined networking) you have to buy another module called NSX. Is it cheap or available for testing in lab environments? The simple and short answer is no, if you are a SMB, just forget about it. So now, what options do you have left? It turs out that here is where you should start looking at other alternatives, if you still want to be happily married with Vmware, you can choose to do that or start looking at Nutanix. Or you can do as other SMB’s are doing you can start looking at Microsoft Windows server 2019 or Windows server 2022 in a hyper converged infrastructure form. You still would require the hardware and Windows licenses, but you would not be needing any more Vmware licenses. Being an SMB and shaving of a annually cost of about 35-50k dollars is not bad. Are you worried about the performance and stability of Hyper v? Read on and this blog post might be able to clear you doubts and confusions. If you never have considered it, you might be very surprised.

Blog post details

I will not be covering how to actually set up the S2D cluster as you will find numerous blog posts describing this, I might write a blog post about it if someone requests me, but there is no use writing about something that has already been written a lot about. I will rather be focusing on how to measure the performance of your S2D cluster. This blog post will also focus a bit on how to use Admincenter for you S2D implementation as it binds all of it together. In the future there will be a blog post about how you can use SDN with Windows server 2022 hyper converged infrastructure.

Prerequisites for S2D performance – Planning hardware

We shall not be spending too much time on this section, however it is imperative to understand that same hardware can provide very different performance depending on the configuration. You can tune it to your needs so that it gives best performance and your given scenario. There is the three basic pillars that apply Compute, Storage, Network.

Compute

You must have enough compute resources so that the workload will not perceive this layer as bottleneck. Normally you would want to take into account Microsoft’s licensing model which focuses on Core based model. You always pay for 16 Cores. This started back in Windows server 2016 i believe and has been the same ever since. You should be considering use of either 1 or 2 processors containing a total of 16 cores at a bare minimum or 32, which is the next stop. You would be using Windows server datacenter edition, which costs about is about 5-6k dollars for each 16 core license. As it is the physical core that are licensed and not virtual you can build a quite decent host without spending too much on Windows server licenses. When sizing the server choose performance.

Storage

You should at least have raid level with failover for the Operating system. Hyper v and Hyper v running on Windows server 2022 core editions do not require more that 20-30 GB of storage. However, it could be a good idea to have some extra storage available. Ideally the hypervisor should be able to withstand disk failure. If you can, you should aim for Nvme or M2 disk in a mirrored configuration for you OS. The next part is storage for the workload or VMs that would run on your cluster. You can choose to have Nvme/M2, SSD, HDD for your main storage tier. You can also mix and match, but as a general rule of thumb, the quickest and most performant storage medium must be used as cache layer and other storage layers can be either of same type of any other slower medium. If you have lets say 2 Nvme disks and rest is SSD, the choice is obvious the Nvme/M2 disks for cache and then SSD for storage. If you only have SSD and HDD, then use SDD for cache and HDD for storage. If all is flash disk, then there is no need for cache! There is also another layer of cache which is memory that can be utilized. It is before the disk cache layer, so if you have a lot of RAM which is not being utilized otherwise it can function as the cache, this is configurable both through admincenter as well as Powershell. We will get back to this at a later point.

S2D storage for Windows 2022 hyper converged cluster
S2D storage for Windows 2022 hyper converged cluster

Network

This is one of the most important components in a hyper converged infrastructure. You can have a lot of performance from disk and onboard storage components, but these do not get the proper utilization as the network layer might be a potential bottleneck. Furthermore, configuring this layer incorrectly can also cause drainage of CPU resources, as the CPU will be interrupted for handling the network traffic. The first thing you need to make sure is that the NICs support SR-IOV which Microsoft describes here

https://docs.microsoft.com/en-us/windows-hardware/drivers/network/overview-of-single-root-i-o-virtualization–sr-iov-

it basically is a feature in hardware which allows bypassing the software layer so that VMs can utilize the functionality found in the hardware without needing to overwhelm the software switch. The next important component or feature is RDMA. This too is similar to SR-IOV but instead focuses on the physical processors, instead of processing the data it is directly bypassed down to NICs which handles the SMB traffic freeing up the processors to work with other tasks. Mind you this is quite important in a hyper converged scenarios as the Hyper v nodes will be sending mirroring data to the other nodes in the cluster. You can read more about RDMA here

https://docs.microsoft.com/en-us/windows-hardware/drivers/network/background-reading-on-rdma

However, be aware that when working with RDMA you would have to consider the implementation options as iWarp and RoCE which both require the your network layer supports these protocols and are properly configure.

Considering the SMB scenario, if we are talking about a 2-3 node cluster, you can rather use peer to peer connections instead of connecting through a switch. This really simplifies the implementation.

S2D cluster nodes Peer to Peer communication without switch
S2D cluster nodes Peer to Peer communication without switch

RDMA gives about 30-37% increase in performance both with regards to bandwidth and latency. The minimum recommended requirement for S2D implementation by Microsoft is 25Gbps Nics for storage and additional for any VM traffic. Which in layman approach translates to 40Gbps NICs and at least 2 of there to have redundancy.

S2D cluster nodes inter communication with switches in place
S2D cluster nodes inter communication with switches in place

Again, if you do not have support for 40Gbps in your network, peer to peer connection option is available between nodes for storage and then you can connect to network with 10 Gbps links to your network. This is by far the most common implementation that my SMB customers use. In Hyper v you should be utilizing SET (switch embedded teaming) which allows you fully utilize the both uplinks giving you 20 Gbps connection to the VMs. Needless to say that you backbone network must support this, but if you are mostly with the SDN realm, you could expect excellent performance.

Now that we have covered the basics for S2D hyper converged infrastructure we can focus on how to measure the performance of the infrastructure. As already mentioned, you should have an idea of what kind of workload you would be running so that you can plan accordingly. Consider a scenario where you access a 500 MB file 10 times vs accessing 5 MB files 1000 times. Both result in 5GB of data being accessed however the IOPS would be very different in first scenarios 10 vs 1000 (this is not the actual number as it would depend on block size, read, write as well as random or sequential reading – however gives an idea of the concept of requirements for workload type).

Tools available and utilized during the procedure

There are numerous tools available to utilize IOPS and hardware performance, but in this blog post we will be focusing on DiskSpd. DiskSpd is a tool that is designed by Microsoft which is the recommended tool to measure S2D performance. However, when it comes to documentation and elaboration of the tool, I believe Microsoft could have done a much better job. Not convinced? Try reading the help files and description of the tool here https://github.com/microsoft/diskspd after you have read the entire document, let me know where you are supposed to run what and in what order.

DiskSpd tool overview

DiskSpd is available at github in the link provided just above. It is supposed to be run on the Host, which will then stage the VMs and insert the program and commands that you enter while working from one of the physical hosts. This is illustrated in the diagram below

S2D performance and IOPS testing with DiskSpd
S2D performance and IOPS testing with DiskSpd

First step in this process is to download the DiskSpd tool along with all it files from GitHub. Extract the files and put them in a folder. I normally create a folder called DiskSpd on C: drive of the host.

DiskSpd folder on Host-01
DiskSpd folder on Host-01

Other folder required is for Vmfleet, you do not need to create it either, but this is just a practice that I use, as the customer would still be able to run testing of performance if required after I have left them with the initial setup.

VMfleet folder for S2D performance testing
VMfleet folder for S2D performance testing

Windows 2022 Core VM setup – Golden image setup step by step

Now er have created the folders and files necessary in order to copy tools into the VMs. The next step is to create a VM. This VM has to be a Windows Core installation. It is very important that the size of the VM vhdx file is at least 20 GB as the tool generates a workload consuming about 10GB. If you create a VM with less disk space you will be greeted with an error message while running the diskspd powershell scripts stating

#failed creation of initial load file for – during the specialize phase

failed creation of initial load - while using create-vmfleet.ps1
failed creation of initial load – while using create-vmfleet.ps1

After you have created the VM and the installation process is complete you can set up a password and shutdown the VM. It would be a good idea to not use special characters and use a rather simple password. That due to the fact that tool creates problems if you do not run the password parameter value in three quotes. You read it right ”’ are required. So if you are using P@$$word! you must specify the password in the format ”’P@$$word!”’ more on this as we move along. Power off the VM, we will only be requiring the Vhdx file and not the VM itself in the next steps. The VHDX file will be used as a template to clone as many instances as you want to use for testing.

Collect and VM storage folder for DiskSpd

The structure we want to create is a disk pool which is provisioned from disks on our hosts. As we want to check the performance for each individual host and the S2D cluster we would require Clustershared volume visible to all nodes but specific for each host. Simpler when illustrated by a diagram, as below

DiskSpd clustershared storage
DiskSpd clustershared storage

In my case I have a host TSTHYP03 and TSTHYP04 along with Collect Virtual disks that would be used for performance testing. vDisk01 is just a repository for ISO files and VHD templates. It can also be used for VM storage, but for the testing of performance it is not required.

#To create TSTHYP03 and TSTHYP04 vhdx

Get-ClusterNode |% { New-Volume -StoragePoolFriendlyName S2D* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size 300Gb }

#Storage should be sufficient for at least 15 VMs if you have allocated 20GB of space.
#Now we create a new volume for collection of data called Collect

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Collect -FileSystem CSVFS_ReFS -Size 100GB

#To install VMfleet run the following commands

cd C:\Vmfleet\diskspd-master\Frameworks\VMFleet1.0
.\install-vmfleet.ps1 -Source C:\VMFleet\diskspd-master\Frameworks\VMFleet1.0

Remember to add the storage to ClusterShared volume, so that it is visible and accessable for all the hosts in question. Now we start to copy the tools and Windows 2019/Windows 2022 core image to Collect folder as the scripts expect these pieces to be available here.

#Choose 64 bit version of the tool and copy it under tools
Copy-Item C:\DiskSpd\amd64\diskspd.exe -Destination c:\clusterstorage\collect\control\tools
 
#Copy VHDX with Windows Server 2019 or Windows 2022 Core (not syspreped!) to the Collect folder
 
Copy-Item "C:\ClusterStorage\vDisk01\Hyper-V\Core01\Virtual Hard Disks\Core01.vhdx" -Destination C:\ClusterStorage\Collect

If you have followed the same steps as this blog post you should have the VHDX file available as

Windows 2022 Core VHDX file for DiskSpd
Windows 2022 Core VHDX file for DiskSpd

As you installed VMfleet, the contents of the control folder should be something similar to the image below

DiskSpd Control folder for Windows 2022
DiskSpd Control folder for Windows 2022

Currently the flag and result folders will be empty, but these will be populated as we run the tests. Files are created and placed in these folder. Now we are ready, that is almost. To make sure that permissions are correct on the newly created clustershared volumes run the following command

# This also changes ownership of test CSV volumes if so required.
 
.\update-csv.ps1 -renamecsvmounts $true

Now we are ready to start creating VMs. This is done by running the following command

#AdminPass is the local password for VHDX golden image file that you created.
#Connect user is domain user to connec to Hyper v Host and -ConnectPass is you password
#If special characters have been used you must use three quotes with the password

.\create-vmfleet.ps1 -BaseVHD 'C:\ClusterStorage\Collect\Core01.vhdx' -VMs 10 -AdminPass 'Abc123Xyz' -ConnectUser 'zeglory.com\admin' -ConnectPass '''ComplexP@$$w0rd!'''

If you for some reason do not include the three quotes with the password, you will most probably end up with 20 VMs, which cannot be used to run disk performance tests on the S2D cluster. The only hint you get is that while running the tests, you will never get a message stating “Pause not enforce” and “Start Go epoch:0”

The VM creation will take some time. However, when it is done you would now find the VMs in cluster manager in Powered Off state. Min have been Powered on. You can increase the processor and ram of the VMs using the following command and then turn them on.

.\set-vmfleet.ps1 -ProcessorCount 2 -MemoryStartupBytes 4GB -MemoryMinimumBytes 4GB -MemoryMaximumBytes 4GB

#To Power on VMs
.\start-vmfleet.ps1

After you have powered on the VMs you should just check that they are up and running. After a couple of minutes you are ready to generate som load.

DiskSpd VMs for Windows 2022 S2D cluster performance testing
DiskSpd VMs for Windows 2022 S2D cluster performance testing

In a new powershell windows which you start as administrator, run the

.\watch-cluster.ps1

It will start displaying the current read and write workload on the hosts along with IOPS. You should disable the in memory caching for the sake of testing. This can be either done in Powershell or Admincenter. You are now ready to start the workload by entering the following command

#To view the current memory caching size
(Get-Cluster).Blockcachesize
#To set the memory caching size to 0 for performance testing. 
(Get-Cluster).Blockcachesize = 0

#To start processes 
.\start-sweep.ps1 -b 4 -t 8 -o 8 -w 0 -d 300 -p r

# b=> block size t=> threads o=> list of IO counts w=> writes d=> duration p=> read/write type or IO pattern

Now you would be presented with messages similar to these

start-sweep generating workload on S2D cluster
start-sweep generating workload on S2D cluster

You can not observe the performance of your individual servers as well as the entire cluster in the Powershell windows where you are hopefully running .\watch-cluster.ps1

S2D IOPS and performance measured with DiskSpd
S2D IOPS and performance measured with DiskSpd

As you can see I am only running read tests, which is generating about 185k IOPS at most. The value will fluctuate and change the entire time as the load changes. But realistically speaking, the cluster in question should be fine delivering somewhere between 150-160K IOPS as the read queue is already in 2 seconds. My experience with the tools shows that you can achieve enough load just by running 4-5 VMs on each Host, if you increase the number of threads. However, you might benefit from spinning up more VMs if you have very performant system consisting of all flash or NVme/M2 disks etc. While the tests are running you can look at the Windows 2022 Core VMs which automatically login and start executing the commands issued by you on the hosts in Powershell.

DiskSpd VMfleet running for S2D performance testing
DiskSpd VMfleet running for S2D performance testing

You can also view the impact of stress testing you environment in Admincenter, which retains the historical data.

Admincenter registering Windows 2022 S2D performance
Admincenter registering Windows 2022 S2D performance

Housekeeping and clean up

After you have performed testing, you would see that you have consumed a lot of storage space in your S2D cluster. In order to reclaim the disk, delete the volums and then remove the virtual disks. Or the correct description of procedure would be to remove the Clustered shared Volumes, after then have been removed you can delete the VHD. If you want to leave the volumes for further testing, that too is fine as long as you are not holding off important storage that otherwise is required by the production environment.

#To poweroff VMs and remove them completely 
.\stop-vmfleet.ps1

#To delete the VMs
.\destroy-Vmfleet.ps1 

Now you are ready to remove the volumes, this can either be done by using Powershell or manually by using cluster manager. I tend to lean towards using Cluster manager, just to be sure not to delete/remove something that is not supposed to be removed. And then at last to remove the VHDS you can run the following commands


#Disk reclaims after removing volumes

$virDisks = Get-ClusterNode |% { Get-VirtualDisk | Where-Object {$_.FriendlyName -eq 'Collect'} }

foreach($virDisk in $virDisks)
{
$vhdUID = $virDisk.UniqueId
Remove-VirtualDisk -UniqueId $vhdUID
}

# Process is repeated for Hosts in the cluster. In our blog post that would be for TSTHYP03 and TSTHYP04 $virDisks = Get-ClusterNode |% { Get-VirtualDisk | Where-Object {$_.FriendlyName -eq 'TSTHYP03'} }
$virDisks = Get-ClusterNode |% { Get-VirtualDisk | Where-Object {$_.FriendlyName -eq 'TSTHYP04'} }

#Lastly set the memory caching for cluster back to whatever you had before performance testing
(Get-Cluster).Blockcachesize = 10240 

After running the commands for copying the image to local storage and running the install command, make sure to boot the switch. The first time, it will be rebooted automatically, but just to be on the safe side just reboot it. Delete the old image file to free up space.

Additional resources

For more useful resources visit technet.

Leave a Reply

Your email address will not be published. Required fields are marked *