Tag Archive for iSCSI

Tech Startup Spotlight – Hedvig

Hedvig

After posting this comment last week, I thought it might be worth following up with a quick post. I’ll be honest and say that until Friday I hadn’t actually heard of Hedvig, but I was invited along by the folks at Tech Field Day to attend a Webex with this up and coming distributed storage company, who have recently raised $18 million in their Series B funding round, having only come out of stealth in March 2015.

Hedvig are a “Software Defined Storage” company, but in their own words they are not YASS (Yet Another Storage Solution). Their new solution has been in development for a number of years by their founder and CEO Avinash Lakshman; the guy who invented Cassandra at Facebook as well as Amazon Dynamo, so a chap who knows about designing distributed systems! It’s based around a software only distributed storage architecture, which supports both hyper-converged and traditional infrastructure models.

It’s still pretty early days, but apparently has been tested to up to 1000 nodes in a single cluster, with about 20 Petabytes, so it would appear to definitely be reasonably scalable! 🙂 It’s also elastic, as it is designed to be able to shrink by evacuating nodes, as well as add more. When you get to those kind of scales, power can become a major part to your cost to serve, so it’s interesting to note that both x86 and ARM hardware are supported in the initial release, though none of their customers are actually using the latter as yet.

In terms of features and functionality, so far it appears to have all the usual gubbins such as thin provisioning, compression, global deduplication, multi-site replication with up to 6 copies, etc; all included within the standard price. There is no specific HCL from a hardware support perspective, which in some ways could be good as it’s flexible, but in others it risks being a thorn in their side for future support. They will provide recommendations during the sales cycle though (e.g. 20 cores / 64GB RAM, 2 SSDs for journalling and metadata per node), but ultimately it’s the customer’s choice on what they run. Multiple hypervisors are supported, though I saw no mention of VAAI support just yet.

The software supports auto-tiering via two methods, with hot blocks being moved on demand, and a 24/7 background housekeeping process which reshuffles storage at non-busy times. All of this is fully automated with no need for admin input (something which many admins will love, and others will probably freak out about!). This is driven by their philosophy or requiring as little human intervention as possible. A noteworthy goal in light of the modern IT trend of individuals often being responsible for concurrently managing significantly more infrastructure than our technical forefathers! (See Cats vs Chickens).

Where things start to get interesting though is when it comes to the file system itself. It seems that the software can present block, file and object storage, but the underlying file system is actually based on key-value pairs. (Looks like Jeff Layton wasn’t too far off with this article from 2014) They didn’t go into a great deal of detail on the subject, but their architecture overview says:

“The Hedvig Storage Service operates as an optimized key value store and is responsible for writing data directly to the storage media. It captures all random writes into the system, sequentially ordering them into a log structured format that flushes sequential writes to disk.”

Supported Access Protocols
Block – iSCSI and Cinder
File – NFS (SMB coming in future release)
Object – S3 or SWIFT APIs

Working for a service provider, my first thought is generally a version of “Can I multi-tenant it securely, whilst ensuring consistent performance for all tenants?”. Neither multi-tenancy of the file access protocols (e.g. attaching the array to multiple domains for different security domains per volume) nor storage performance QoS are currently possible as yet, however I understand that Hedvig are looking at these in their roadmap.

So, a few thoughts to close… Well they definitely seem to be a really interesting storage company, and I’m fascinated to find out more as to how their key-value filesystem works in detail.  I’d suggest they’re not quite there yet from a service provider perspective, but for private clouds in the the enterprise market, mixed hypervisor environments, and big data analytics, they definitely have something interesting to bring to the table. I’ll certainly be keeping my eye on them in the future.

For those wanting to find out a bit more, they have an architectural white paper and datasheet on their website.

Cannot See Any iSCSI Devices on Synology from a vSphere Host

Just a quick fix I discovered this weekend. It’s probably quite specific but hopefully if you come across this in future it will save you some time.

I had just finished rebuilding the second node in my lab from 5.1 with a fresh install. I added the software iSCSI initiator and connected it to my iSCSI target (Synology DS412+) using Dynamic Discovery. I then rescanned for a list of devices, and although I was picking up the IQN for the iSCSI server, I couldn’t see any devices!

I tried lots of things including removing and re-adding the initiator, messing with iSCSI bindings, but nothing! Very frustrating.

After a bit of googlage, I came across this KB article from VMware:

Cannot see some or all storage devices in VMware vCenter Server or VirtualCenter (1016222)

Although this was specific to VI3/vSphere 4, it did trigger a thought! Just before I built the new node, I rejigged all of my storage LUNs, deleting 3 old ones in the process (which just so happened to be the first 3 LUNs on my NAS). What I believe this caused is that the LUNs were viewed by node 1 as different LUN IDs on node 2, so they refused to show up!

So now, the fix. Incredibly simple as it turns out:

  1. Created three new temp 10GB LUNs on my old NAS which would then assume LUN IDs 1/2/3 as they originally had (before I deleted them).
  2. Rescan the new node of the cluster for storage
  3. Confirmed all of the LUNs are now visible
  4. Deleted the three temp LUNs from the Synology (I don’t plan to add any more nodes for now so I have no need of these temp LUNs, but as they’re thin provisioned anyway it actually wouldn’t hurt to leave them there).
  5. Rescan the ESXi host again to ensure it can still see the LUNs.
  6. Job done!

Synology iSCSI Devices

Not much to it, but worth a quick post I thought as this simple issue wasted a chunk of my time!

FreeNAS 0.7.2 NFS and iSCSI Performance in a vSphere 4.1 Lab

While doing some lab testing and benchmarking for my upcoming VCAP-DCD exam I came across some interesting results when messing about with NFS and iSCSI using FreeNAS. I plan to re-run the same set of tests soon using the EMC Celerra simulator once I have it set up.

The results are from very simplistic testing using a simple buffered read test only (it would be reasonable to expect write times to be the same or slower, and this is just a quick test for my own info). For this I used the following sample hdparm command in some Ubuntu VMs:

sudo hdparm -t /dev/sda

A sample output from this command would be:

/dev/sda:  Timing buffered disk reads: 142 MB in  3.11 seconds =  45.59 MB/sec

As this was a quick performance comparison I only repeated the test three times per storage and protocol type, but even with this simplistic testing the results fairly conclusive.

Test HW was based on my 24GB RAM ESX and ESXi cluster in a box solution [hence some results will be faster than you can achieve over a gig network as this is all running on one host] running under Windows 7 64-bit with VMware Workstation 8. Components are:

  • 4x ESX/ESXi 4.1 hosts running in Workstation 8 with an SSD datastore. 4GB RAM and 2x vCPUs each.
  • 1x FreeNAS 0.7.2 instance running in Workstation 8 with an SSD datastore and a SATA datastore. I use this over FreeNAS 8 as it has a significantly smaller memory footprint (512mb instead of 2GB). 1vCPU and 512 MB RAM.
  • 64-bit Ubuntu Linux VMs running nested under the ESX(i) virtual hosts. 1vCPU and 512 MB RAM each.

Storage components are:

  • SATA 2 onboard ICH10R controller
  • 1x Crucial M4 128GB SSD (500MB/sec Read, 175MB/Sec Write)
  • 1x Seagate 250GB 7200RPM SATA

The results of the testing are as follows:

Protocol Storage Type Read MB/sec
Local VMFS SSD 383
Local VMFS SATA 88
FreeNAS 0.7.2 w/ NFS SSD 11
FreeNAS 0.7.2 w/ NFS SATA 5
FreeNAS 0.7.2 w/ iSCSI SSD 175
FreeNAS 0.7.2 w/ iSCSI SATA 49

As you can see, FreeNAS with NFS does not play nice with ESX(i) 4. I can confirm that I have seen stats and posts confirming these issues are not aparent in the real world, with NetApp FAS or Oracle Unified Storage (which is aparently awesome on NFS) but for your home lab, the answer is clear:

For best VM performance using FreeNAS 7, stick to iSCSI!

%d bloggers like this: