Windows Server Storage Spaces tier write-back cache, how to configure threshold that determines if write is performed on write-back cache?

Question

The Storage Spaces feature included with Windows Server 2012 and the upcoming Windows Server 2016 consists of a virtual disk. In a tiered storage deployment a write-back cache is created (the default size is 1GB). There is a threshold that is configured in Storage Spaces which determines whether a particular write operation will go to the write-back cache or not. This threshold is very small and makes it so that the majority of writes do not land on the write-back cache.

As an example, let's assume that the write-back cache can achieve 500MB/s while the slower tier operates at only 75MB/s. After creating the virtual disk and creating a freshly formatted volume, it is immediately noticeable that a copy operation of a 386MB file takes longer than one second to finish on the Storage Spaces Virtual Disk. One would hope that the file would be copied in under a second assuming that the write-back cache would consume the operation, but it doesn't. Instead the storage tier detects that the write size exceeds the pre-defined threshold for writing to the write-back cache and forwards the write to the slowest tier which then takes around 5 seconds to complete. I find it disappointing that the write-back cache will not accelerate this file copy operation.

Background information can be found here:

• https://technet.microsoft.com/en-us/library/dn387076(v=ws.11).aspx#bkmk_wbc
• https://social.technet.microsoft.com/Forums/en-US/e6868c95-2f6a-4433-a012-174cadba13cd/windows-server-writeback-cache-storage-tier-how-do-you-configure-the-threshold-for-how-many

My goal is to accelerate more write operations, not just those very small in size. Here is a list of some other software providers that have accomplished this:

• VirtuCache: http://virtunetsystems.com/virtucache-overview/
• Intel CAS: https://software.intel.com/en-us/articles/intel-cache-acceleration-software-a-primer
• PrimoCache: (write-deferred caching mode) https://www.romexsoftware.com/en-us/primo-cache/index.html

If Microsoft had a way to simply allow write-deferred caching, or to modify the threshold for what is to be written to the write-back cache Storage Spaces speeds could be improved by 20-fold (according to the benchmarks provided by these 3rd party vendors).

Two questions:

1) What is the default threshold (in Kilobytes) for Storage Spaces to choose to perform writes to the write-back cache?

2) What command can be issued in PowerShell to modify the threshold so that I can instruct Storage Spaces to perform larger writes to the write-back cache?

Solution 1

There's no known threshold for that. If WBC enabled all writes will go to flash regardless of their size. The only case they go to disk bypassing flash is when flash is already full and flushing it before acknowledging newer write is pointless: latency will spike all over the roof.

Some good reading on topic:

https://blogs.technet.microsoft.com/larryexchange/2015/12/02/understand-storage-space-tiering-in-windows-server-2012-r2/

https://technet.microsoft.com/en-us/library/dn789160.aspx

I'd rather avoid third-party caching software unless they have something of a real value to bring in, like f.e. distributed DRAM write-back cache. You have to run a cluster with something like StarWind vSAN for that though as huge DRAM cache on a single server is begging for troubles.

https://www.starwindsoftware.com/caching-page

You can experiment with CSV cache but it's read-only so far.

https://blogs.msdn.microsoft.com/clustering/2013/07/19/how-to-enable-csv-cache/

Upcoming Windows Server 2016 has some optimizations here as well so worth giving it a try.

https://blog.cdemi.io/caching-and-storage-tiering-in-storage-spaces-direct/

Solution 2

If you want to customize the write-back cache size, use the Windows PowerShell cmdlets and the -WriteCacheSize parameter, for example:

New-VirtualDisk -StoragePoolFriendlyName "My Storage Pool" -FriendlyName TieredSpace -StorageTiers @($ssd_tier, $hdd_tier) -StorageTierSizes @(50GB, 300GB) -ResiliencySettingName Mirror -WriteCacheSize 2GB

Note: After a virtual disk is created, its WriteCacheSize can't be modified.

Source: http://windowsitpro.com/windows-server-2012-r2/set-windows-server-2012-r2-storage-space-write-back-cache

Generally speaking, Storage Spaces WBC is not optimized for large cache capacity and I, personally, would not assign more than 5GBs in any case.

Also, as @BaronSamedi1958 noticed, distributed RAM cache is far better way to speed up write I/O and StarWind's cache implementation should definitely worth a try.

Comments

[Cammy Boy]

"Generally speaking, Storage Spaces WBC is not optimized for large cache capacity and I, personally, would not assign more than 5GBs in any case" is there a source for this?

[Monstieur]

In Simple and Mirror spaces, only random writes go to the write cache. Sequential writes above 256 KiB bypass the write cache completely, so a 100 GB write cache is useless unless you have a workload that can accumulate that many random writes. In Parity spaces, all writes go to the write cache or Journal disk until its full.

[Monstieur]

@wbkang The write cache is bypassed for sequential writes on Simple and Mirror spaces.

[BaronSamedi1958]

You don't get sequential writes with anything except video capture and one target per LUN. Read about so-called "I/O blender" effect. blog.fosketts.net/2012/05/23/…

[Monstieur]

I'm using a simple tiered storage space on a desktop, and pasting large files bypasses the WBC and writes at the speed of the HDD.

[Bill Silner]

The link is dead

[Bill Silner]

Also running this one Win2019 returns an error New-VirtualDisk : Failed to run CIM method CreateVirtualDisk on the MSFT_StoragePool (ObjectId = "{1}\\STORAGE2\root/Microsoft/Windows/St...) CIM object. CIM array cannot contain null elements.