Archive for Storage

Words Mean Things, Apparently – Deduplication Myths Explored

A rose by any other name would smell as sweet?

We might all agree that this is most definitely the case, but in the technology industry we have a problem, and it was highlighted across a number of the sessions we attended at Storage Field Day 9 this week.

Specifically, the use of certain terms to describe technology features, when the specific implementations are very different, and have potentially very different outcomes. This is becoming more and more of a problem across the industry as similar features are being “RFP checkboxed” as the same, when in reality they are not.

For example most of the vendors we saw support deduplication in one form or another, and in many cases there was a significant use of the word “inline”.

What do we mean by “inline deduplication”, and what impact to performance can this have?

One of the other delegates at SFD9, W Curtis Preston, had very strong opinions on this, which I am generally inclined to agree with!

UPDATE 08/04/2016: Curtis has recently published an article detailing his thoughts here.

If a write hits the system and is deduplicated prior to being written to its final non-volatile media, be it flash or disk, then it can generally be considered as inline.

Dedupe-Inline

Inline Deduplication

If deduplication is running in hardware (for example as 3PAR do in their Gen4+ ASIC), the deduplication process has minimal overhead on the system, and by not needing to send all writes to the back end storage it can actually improve performance overall, even under sustained high throughput where it can actually improve it by reducing back end writes.

Most non-inline deduplication would typically be referred to as “post-process”, and as a general rule are either run on a schedule or as a lower priority 24/7 system maintenance task. It can also run immediately after the write has gone to disk. This is still post-process, not inline.

It’s worth noting that any of these post-process methods can potentially have an impact on back-end capacity management, as dumping large quantities of data onto a system can temporarily spike capacity utilisation until the dedupe process has time to work its magic and increase storage efficiency. Not ideal if your storage capacity is approaching critical.

depu

In addition, the block has been written to an NVRAM device which should protect it from power loss etc, but the problem we have is that cache is an expensive and finite resource. As such, by throwing a sustained number of IOs at the system, you end up potentially filling up that cache/NVRAM faster than the IOs can be flushed and deduplicated, which is exacerbated by the fact that post-process dedupe generates yet more IOPS on the back end storage (by as much as 2-3x compared to the original write!). The cumulative effect causes IO to back up in the system like a dodgy toilet, thereby increasing latency and reducing your maximum capable IOPS from the system.

Worse still, in some vendor implementations, when system performance is maxed out deduplication in the IO path is dropped altogether, and inbound data is dumped out to disk as fast as possible. Then is then post-processed later, but this could obviously leave you in a bit of a hole again if you are at high capacity utilisation.

Dedupe-post

Post-Process Deduplication

None of this is likely to kick in for the vast majority of customers as they will probably have workloads generating tens of thousands of IOPS, or maybe low hundreds of thousands on aggregate. As such, for most modern systems and mixed workloads, this is unlikely to be a huge problem. However, when you have a use case which is pushing your array or HCI solution to its maximum capability, this can potentially have a significant impact on performance as described above.

[HCI – yet another misappropriated computing acronym, but I’ll let that one slide for now and move on!]

VMware VSAN Deduplication

In the case of one of one of the vendors we saw, VMware, they joked that because of the fact that they initially write to the caching flash tier prior to deduplication, they spent more time arguing over whether it was valid to call this inline than it took them to actually develop the feature! In their case, they have been open enough not to call it “inline” but instead “nearline”.

In part this is because they are always written to a flash device prior to dedupe, but also because not all of the writes to their caching tier actually get sent to the capacity tier. In fact some may live out their entire existence in an non-deduplicated state in flash cache.

dedupe.png

I applaud VMware for their attempt to avoid jumping on the inline bandwagon, though it would have perhaps been better to use a term which doesn’t already mean something completely different in the context of storage! 🙂

You can catch the full VMware session at the link below – it’s well worth a watch!
VMware Storage Presents at Storage Field Day 9

Further Reading

Some of the other SFD9 delegates and VMware staffers had their own takes on the presentation we saw. Check them out here:

Disclaimer/Disclosure: My flights, accommodation, meals, etc, at Storage Field Day 9 were provided by Tech Field Day, but there was no expectation or request for me to write about any of the vendors products or services and I was not compensated in any way for my time at the event.

You had me at Tiered Non-Volatile Memory!

Memory isn’t cheap! Despite the falling costs and increasing sizes of DRAM DIMMS, it’s still damned expensive compared to most non-volatile media at a price per GB. What’s more frustrating is that often you buy all of this expensive RAM, assign it to your applications, and find later through detailed monitoring, that only a relatively small percentage is actually being actively used.

For many years, we have had technologies such as paging, which allow you to maximise the use of your physical RAM, by writing out the least used pages to disk, freeing up RAM for services with current memory demand. The problem with paging is that it is sometimes unreliable, and when you do actually need to get that page back, it can be multiple orders of magnitude slower returning it from disk.

Worse still, if you are running a workload such as virtual machines and the underlying host becomes memory constrained, a hypervisor may often not have sufficient visibility of the underlying memory utilisation, and as such will simply swap out random memory pages to a swap file. This can obviously have significant impact on virtual machine performance.

More and more applications are being built to run in memory these days, from Redis to Varnish, Hortonworks to MongDB. Even Microsoft got on the bandwagon with SQL 2014 in-memory OLTP.

One of the companies we saw at Storage Field Day ,  Plexistor, told us that can offer both tiered posix storage and tiered non-volatile memory through a single software stack.

The posix option could effectively be thought of a bit like a non-volatile, tiered RAM disk. Pretty cool, but not massively unique as RAM disks have been around for years.

The element which really interested me was the latter option; effectively a tiered memory driver which can present RAM to the OS, but in reality tier it between NVDIMMs, SSD and HDDs depending on how hot / cold the pages are! They will also be able to take advantage of newer bit addressable technologies such as 3D XPoint as they come on the market, making it even more awesome!

PlexistorArch.jpg

Plexistor Architecture

All of this is done through the simple addition of their NVM file system (i.e. device driver) on top of the pmem and bio drivers and this is compatible with most versions of Linux running reasonably up to date kernel versions.

It’s primarily designed to work with some of the Linux based memory intensive apps mentioned above, but will also work with more traditional workloads as well, such as MySQL and the KVM hypervisor.

Plexistor define their product as “Software Defined Memory” aka SDM. An interesting term which is jumping on the SDX bandwagon, but I kind of get where they’re going with it…

SDM_vs_SDS2.png

Software Defined Memory!

One thing to note with Plexistor is that they actually have two flavours of this product; one which is based on the use of NVRAM to provide a persistent store, and one which is non-persistent, but can be run on cloud infrastructures, such as AWS. If you need data persistence for the latter, you will have to do it at the application layer, or risk losing data.

If you want to find out a bit more about them, you can find their Storage Field Day presentation here:
Plexistor Presents at Storage Field Day 9

Musings…
As a standalone product, I have a sneaking suspicion that Plexistor may not have the longevity and scope which they might gain if they were procured by a large vendor and integrated into existing products. Sharon Azulai has already sold one startup in relatively early stages (Tonian, which they sold to Primary Data), so I suspect he would not be averse to this concept.

Although the code has been written specifically for the Linux kernel, they have already indicated that it would be possible to develop the same driver for VMware! As such, I think it would be a really interesting idea for VMware to consider acquiring them and integrating the technology into ESXi. It’s generally recognised as a universal truth that you run out of memory before CPU on most vSphere solutions. Moreover, when looking in the vSphere console we often see that although a significant amount of memory is allocated to VMs, often only a small amount is actually active RAM.

The use of Plexistor technology with vSphere would enable VMware to both provide an almost infinite pool of RAM per host for customers, as well as being able to significantly improve upon the current vswp process by ensuring hot memory blocks always stay on RAM and cold blocks are tiered out to flash.

plexistorvmware

The homelab nerd in me also imagines an Intel NUC with 160GB+ of addressable RAM per node! 🙂

Of course the current licensing models for retail customers favour the “run out of RAM first” approach as it sells more per-CPU licenses, however, I think in the long term VMware will likely move to a subscription based model, probably similar to that used by service providers (i.e. based on RAM). If this ends up being the approach, then VMware could offer a product which saves their customers further hardware costs whilst maintaining their ESXi revenues. Win-Win!

Further Reading
One of the other SFD9 delegates had their own take on the presentation we saw. Check it out here:

Disclaimer/Disclosure: My flights, accommodation, meals, etc, at Storage Field Day 9 were provided by Tech Field Day, but there was no expectation or request for me to write about any of the vendors products or services and I was not compensated in any way for my time at the event.

Violin Memory have cool technology, but do they have a future? I hope so!

Since visiting Violin Memory at Storage Field Day 8, it has taken me a while to get around to writing this post, and I guess the reason is because I am both frustrated, and a little bit sad.

They were the first guys who (for me at least) truly tamed the beast that is flash storage, packed it up into a blisteringly fast product with insanely low latencies, and released it into the big wide world.

An organisation I worked at took that product and threw thousands of desktop users at it, a number of very busy SQL TempDBs, and some other frankly evil workloads. It didn’t even blink an eye! With Violin Storage on the back, I have seen RDS desktop customers stress test the platform with up to 100 users in a single VM!

The thing that makes me sad is that to date, Violin have not yet turned a profit.

This is not some upstart company out of the boondocks! This is a mature company, founded in 2005, who provide their product to some of the biggest enterprises in the world. Yet so far they have not managed to make a penny!

Anyway, there are other guys in the industry who are far better at financial analysis than me (for example Justin Warren did a post on this very subject just last week). So I will leave it to them try to work out why this is the case, because I like to talk tech!

Violin Memory

Looking at the latest incarnation, once again Violin have “evolutionized” [sic] something which is technically very impressive.

  • The original Violin OS has been given the boot and has been replaced by something they call Concerto OS, which blends elements of the original software with some updated features. In development is also support for a multi-controller configuration beyond the current dual, though this is obviously not GA yet.
  • The new dual controller Flash Storage Platform supports FC, iSCSI and Infiniband, as well as RDMA and ROCE.
  • It is all packaged in a 3U appliance with up to 64 redundant flash modules.
  • Thanks to their own custom backplane design, it is capable of 10-12GB/sec of throughput!
  • At 100% sustained writes they have measured 400,000 IOPSat RAID5, which is more than many of their competitors can achieve with 100% reads!It should be noted that this was on the performance model, which does not support dedupe.

All in all the new solution just screams FAST! In fact, I’m surprised they didn’t paint a red stripe down the side of the chassis!

low latency

So why on earth are Violin not ruling the AFA world right now? As a frickin cool technology with hyper speed storage, they deserve to be up there at the very least!

If I had to hazard a guess, it’s because for most companies, good is good enough.

With the smorgasbord of All Flash Arrays available today, if you don’t need latency measured in microseconds and massive IOPS/bandwidth, then you have a huge array of choices (pardon the pun). At that point features, price and support become more important than straight line drag racer performance.

If Violin want to compete with the general market whilst servicing their high-speed clients, then they need to concentrate on continuing to developing a wider range of data services and providing entry-level options to consume their products. The last thing you want to do is lose business just because you were missing a check box in an RFP…

If Violin can stop burning cash and break even, then perhaps they have a future. I for one, hope so!

If you want to catch Violin’s presentation from SFD8, check them out here:
http://techfieldday.com/appearance/violin-memory-presents-at-storage-field-day-8/

They will also be presenting again at SFD9 this week, and I’m looking forward to finding out what they plan to do next!

Further Reading
Some of the other SFD8 delegates have their own takes on the presentation we saw. Check them out here:

Disclaimer/Disclosure: My flights, accommodation, meals, etc, at Storage Field Day 8 were provided by Tech Field Day, but there was no expectation or request for me to write about any of the vendors products or services and I was not compensated in any way for my time at the event.

Pure Storage goes All Cloudy

Conversations (and arguments) are prevelant in the industry today about moving to the cloud, be it public cloud, private cloud or hybrid cloud.

People are developing into the usual fervent “religious” groups depending on their personal biases and use cases, and today Pure Storage joined the discussion by announcing a new feature for their FlashStack product. Working in partnership with Equinix, they will enable you to effectively colo a replicated FlashStack device in an Equinix DC, then use it to mount volumes via iSCSI, straight into your instances on AWS and Azure.

FlashStackHybrid.png

This is an interesting solution and having discussed this with several other guys at Pure Accelerate, the general impression is positive for two main reasons/uses:

  • Legacy solutions which depend on traditional enterprise infrastructure / storage patterns and performance.
  • As a sales enabler for Pure – Ticking checkboxes for FlashStack when”Is it cloud integrated?” comes up in an RFP.

It’s a cool idea, but personally, I am still struggling to find a really compelling use case for this, and I do slightly worry for Pure that they are providing an easy on-ramp to the public cloud for their customers. Once the compute is up there, comfort with the public cloud grows, and applications are refactored to be more cloud-compatible, the primary storage will undoubtedly follow. Pure do run the risk eroding a percentage of their customer base in the long term.

I guess only time will tell, but I hope this little gamble does not turn around to bite Pure in the rear later!

Hybrid FlashStack integration is available on Azure today, with AWS coming soon…

Disclaimer/Disclosure: My accommodation, meals and event entry to Pure Accelerate were provided by Pure Storage, and my flights were provided by Tech Field Day, but there was no expectation or request for me to write about any of the products or services and I was not compensated in any way for my time at the event.

%d bloggers like this: