Over the past few weeks, we have talked a lot about different key performance indicators and how these may indicate an underlying issue associated with your performance problems. One of the recurring issues brought up was forced flushing. So, today we are going to dive a little deeper into what forced flushing is.
How Does Forced Flushing Occur?
Let’s say you have an environment that has very high IOPS (Input/Output Operations per Second) requirements that is currently being run on SATA disks. These SATA disks will most likely not have the IOPS capability that is required for your environment.
So, what happens? Your storage processor, or SP, identifies the hot blocks on these LUNs and keeps all of the data in cache. As all of the transactions and I/Os are coming in, everything is being written to cache. The SATA disks are simply not fast enough for the storage processor to upload all the data onto the disks to allow new data to come into the cache. So, a key performance indicator of your system will be the number of dirty pages. Dirty pages are pages of cache that have not yet been written to disk.
The number of dirty pages has a low and high watermark, typically 60% and 80% respectively. You will usually see peaks and valleys in your performance graphs that will typically fall between these watermarks of 60% and 80%. This is an indication of healthy cache utilization.
What is essentially happening here is that your SP cache is hitting the 80% mark and saying, “Ok, I need to start dumping some data off to disk.” This brings the amount back down to 60% where it will start working its way back up. This is very healthy and the way it is designed to work.
High Cache Utilization – Forced Flushing
However, in a situation where you have very high cache utilization, the percentage of dirty pages will eventually climb up to 100% and then it will flat line. This means that the array is not able to flush the data onto disk fast enough. This is when you are in danger of forced flushing.
When your array sees that it is at 100% dirty pages for over a certain amount of time, it will initiate a process called forced flushing. During forced flushing, the array is basically saying, “I am not taking any more IO into my cache until I can dump what is currently existing in the cache to disk.” So, any additional IO coming to the array no longer has the ability to use the right cache. Instead it will go directly to the RAID group or the storage pool where the LUN resides.