The local storage is presented by a Fujitsu D2616 (almost entirely based on LSI SAS2108) with 4 7200rpm 500GB SAS disks. (THe HBA is not on the VMware HCL but the 2108 is)
I assume the RAID controller on your storage has a battery-backed write cache? What RAID configuration is configured? I really wouldn't expect much from a RAID5 with four 7200k RPM disks for example.
What exactly is the HBA model and which driver is it running? Run this to check:
# esxcli storage core adapter list
HBA Name Driver Link State UID Description
-------- -------- ---------- ------------------------------------ -------------------------------------------------------------------------
vmhba0 ata_piix link-n/a ide.vmhba0 (0:0:31.1) Intel Corporation 631xESB/632xESB IDE Controller
vmhba1 cciss link-n/a block.cciss/c0d0:0 (0:6:0.0) Hewlett-Packard Company Smart Array P400i
If it's using the mpt2sas driver then it might be worth updating the driver:
attempting a snapshot of a smallish <40GB thin provisioned VM was taking 30 minutes to complete.
I've monitored ESXTOP during one of these snapshot attempts and found the write latency to be the obvious figure of concern - QAVG and KAVG stats are at 0ms with DAVG running between 20 and 30 for the duration of the write.
Really 30 minutes for the Create Snapshot operation (quiesced?)? Or a removing Snapshot Operation?
20-30ms are, while being quite bad, actually not that horrible.
We still have some local datastores consisting of four 15k RPM SAS drives in RAID10 on a controller with battery-backed write cache and performance is quite satisfying for what it's worth, though we usually don't run VMs on there.