While doing some lab testing and benchmarking for my upcoming VCAP-DCD exam I came across some interesting results when messing about with NFS and iSCSI using FreeNAS. I plan to re-run the same set of tests soon using the EMC Celerra simulator once I have it set up.
The results are from very simplistic testing using a simple buffered read test only (it would be reasonable to expect write times to be the same or slower, and this is just a quick test for my own info). For this I used the following sample hdparm command in some Ubuntu VMs:
sudo hdparm -t /dev/sda
A sample output from this command would be:
/dev/sda: Timing buffered disk reads: 142 MB in 3.11 seconds = 45.59 MB/sec
As this was a quick performance comparison I only repeated the test three times per storage and protocol type, but even with this simplistic testing the results fairly conclusive.
Test HW was based on my 24GB RAM ESX and ESXi cluster in a box solution [hence some results will be faster than you can achieve over a gig network as this is all running on one host] running under Windows 7 64-bit with VMware Workstation 8. Components are:
- 4x ESX/ESXi 4.1 hosts running in Workstation 8 with an SSD datastore. 4GB RAM and 2x vCPUs each.
- 1x FreeNAS 0.7.2 instance running in Workstation 8 with an SSD datastore and a SATA datastore. I use this over FreeNAS 8 as it has a significantly smaller memory footprint (512mb instead of 2GB). 1vCPU and 512 MB RAM.
- 64-bit Ubuntu Linux VMs running nested under the ESX(i) virtual hosts. 1vCPU and 512 MB RAM each.
Storage components are:
- SATA 2 onboard ICH10R controller
- 1x Crucial M4 128GB SSD (500MB/sec Read, 175MB/Sec Write)
- 1x Seagate 250GB 7200RPM SATA
The results of the testing are as follows:
Protocol | Storage Type | Read MB/sec |
Local VMFS | SSD | 383 |
Local VMFS | SATA | 88 |
FreeNAS 0.7.2 w/ NFS | SSD | 11 |
FreeNAS 0.7.2 w/ NFS | SATA | 5 |
FreeNAS 0.7.2 w/ iSCSI | SSD | 175 |
FreeNAS 0.7.2 w/ iSCSI | SATA | 49 |
As you can see, FreeNAS with NFS does not play nice with ESX(i) 4. I can confirm that I have seen stats and posts confirming these issues are not aparent in the real world, with NetApp FAS or Oracle Unified Storage (which is aparently awesome on NFS) but for your home lab, the answer is clear:
For best VM performance using FreeNAS 7, stick to iSCSI!