Archive for 27th September 2016

Index of Blog Posts on Docker and Containers

Docker Logo

Hard to believe I wrote my first blog post on Docker in September 2014! Life moves pretty fast. If you don’t stop and look around for awhile, you could miss it! 🙂

The regularity with which my posts are likely to feature Docker or other container technologies is increasing quickly, so I thought it was worthwhile indexing all of the current posts and providing an updated list as this grows. Hopefully, this should make them easier for people to find!

Docker HOWTO Series
Tips and Fixes
Random Posts

Just in case I forget to keep this page updated:

docker and container posts

Guide to OpenStack for VMware and AWS Admins – Part 1 – Intro

As a newcomer to the OpenStack world, with quite a bit of VMware and some AWS experience, I thought it would be worthwhile documenting some of the basics as I learn. Hopefully, this will provide something useful for others with a background in either technology, who choose to follow the same path in the future. In many ways, this is planned to be as much to solidify my understanding, as anything else!

Also, it’s probably worth noting that I may express some opinions throughout this series as to where one technology may suit specific workloads better than another. This certainly does not constitute me expressing a preference for one or another! I’m lucky enough to have a day job where I work with a huge range of great technologies; each has their own place in today’s enterprises. The technology should fit the use case – Technology Agnosticism FTW! 🙂

The Basics

Firstly, a few key basics and observations!

  • OpenStack is a collection of different tools and technologies, most of which are entirely interchangeable and / or optional. For example, you could choose to use any of a huge number of hypervisors, such as KVM, Xen, VMware, etc, each of which will have their own pros and cons. I will try to dig into one or two of these when I do a post on Nova, later.
  • OpenStack is quite a complex beast, and most certainly not a simple monolithic stack. Within each of the separate elements of OpenStack (known as “OpenStack services“) there are actually multiple independent processes, all of which do different tasks within their specific service. Here is a quick conceptual diagram which describes a typical solution and all of the interconnecting services:OpenStack Conceptual Architecture
  • Taking that a stage further, there is also a far more complex diagram which shows all of the logical processes in a typical architecture as well!
    OpenStack Logical Architecture
  • Keystone is the most critical service of all, as it is the glue which binds all other OpenStack Services together. As you add more services into your stack, they all register back to Keystone to provide service discovery, API client authentication and a number of other functions. The closest equivalent in vSphere is the PSC. In AWS it would be IAM, but IAM is mainly about the permissions and security elements only, compared to the broad set of functions Keystone provides.
  • If you are an AWS developer and don’t want to have to re-learn or re-write all of your API calls for your software, you don’t have to! You could use HPE Helion Eucalyptus to effectively provide an AWS-compatible API for your OpenStack cloud – that’s pretty cool!
  • OpenStack is still being developed at a huge rate of knots! The releases come out every 6 months and are named alphabetically. We are already at M (Mitaka), with N (Newton) coming out imminently! It’s definitely getting pretty mature as a platform, and I suspect that’s probably why many more enterprises are being quite vocal about looking at it for their private clouds these days.
Private Cloud
Building OpenStack

The control plane and proxy services can all be run as containers. A typical highly-scalable design pattern is therefore a set of physical hosts running containers for all management / API / control processes. You then add one or more separate compute and storage clusters based on your scalability and resilience requirements. For a test lab, you can collapse these onto as little as a single physical host if you use nested instances.

In fact, it will even install in as little as 8GB of RAM, as Eric Wright described in his blog post here about installing on top of OSX. This was based on the 2nd Edition of the awesome OpenStack Cloud Cookbook from Kev Jackson and Cody Bunch. I also did a recent review of the book, for those who are interested.

Vagrant is an excellent way to help get started quickly as it will pull down images and spin up machines very quickly, with minimal effort. It supports multiple environments from VirtualBox and VMware to Docker and even AWS.

The fact that OpenStack is designed from the ground up with automation in mind means you can do some really amazing stuff with it. For example, the other day I was at a presentation where my colleague @the_cloudguru deployed a development stack on his laptop using just 3 lines of OpenStack Ansible code! Very impressive!

Closing Thoughts

I’m still really early in my OpenStack learning journey, but as my knowledge builds I will expand on this series. If you do see any errors in the information in this series, please don’t hesitate to let me know!

Amazon AWS Tips and Gotchas – Part 8 – AWS EC2 Reserved Instances

Continuing in this series of blog posts taking a bit of a “warts and all” view of a few Amazon AWS features, below are a handful more tips and gotchas when designing and implementing solutions on Amazon AWS, including AWS EC2 Reserved Instances.

For the first post in this series with a bit of background on where it all originated from, see here:
Amazon #AWS Tips and Gotchas – Part 1

For more posts in this series, see here:
Index of AWS Tips and Gotchas

AWS Tips and Gotchas – Part 8

Reserved Instances are a great way to save yourself some money for instances you know you will require for a significant period of time (from 12-36 months). One really cool fact which AWS don’t announce enough, in my opinion, is that reserved instances can actually be shared across consolidated billing accounts!

If you wanted to, you could purchase all of your reserved instances from your primary consolidated billing accounts, however, it doing this has some potentially unexpected results:

  1. Reserved instances don’t just provide you with a better price, they also provide you with guaranteed ability to spin up an instance of your chosen type, regardless of how busy the AZ in question actually is.
    If there is an AZ outage, other AWS customers will scramble to spin up additional instances in other AZs in the same region, either manually or via ASGs, and this has the potential to starve the compute resources for one or more instance types!
    Yes, that’s right, even AWS do not have an infinite compute resources!AWS Infinity Reserved InstancesBy using reserved instances, you are still guaranteed to be able to run yours regardless of available capacity for on-demand instances. They are truly reserved.
    If however, you centralise your reserved instances into your CB account, you will get the reservation pricing benefits at the top of the account tree, but you don’t get the capacity reservations as these are account specific.
  2. Reserved instances are specific to individual Availability Zones, so ensure you spread these evenly across your AZs to avoid wasting them (you are of course designing your apps to be resilient across AZs, right?) and give you maximum reserved coverage in the unlikely event of a full AZ outage.
  3. And finally… Reserved instances are a commercial tool applied after-the-fact, not against a specific instance. When using consolidated billing for reserved instances, the reservations are therefore effectively split evenly across all accounts. If you actually want to report back to each business unit / account owner on their billing including reserved instance, this could be tricky.

Find more posts in this series here:
Index of AWS Tips and Gotchas

Amazon AWS Tips and Gotchas – Part 9 – Scale-Up Patching

What I read on my holidays – Uber Geek Edition!

Having only started in my new role at the start of July, I was fortunate enough to sneak in a cheeky week off work at the end of the kids summer holidays. My wife and I have done a fair bit of travelling in the past, but being parents of young children, we do not currently go in for big sightseeing tours. My ears can only survive hearing “my feet hurt” and “I need a wee” so many times before I give in to temptation and leave the kids by the side of the road!…

As I would prefer not to go to prison, instead we had a pretty chilled out week at a resort and I was able to get a wee bit of reading in; which was nice!

readingTypically I like to vary my reading between something for enjoyment, followed by something educational, then rinse and repeat. The former is generally some kind of fiction, especially science fiction / fantasy / humour.

IMO, Terry Pratchett was a true genius and is my favourite author by a huge margin, and he manages to achieve all three of these categories, and then some! Unfortunately, Terry passed away in March last year, leaving millions of fans deeply saddened. The two fiction books below were in fact originally recommended by him, and I would certainly echo this recommendation!

  • openstack-explainedOpenStack Explained – Giuseppe Paternò
    • I was fortunate enough to see Guiseppe present on OpenStack at this year’s Tech Unplugged event in London (see playlist of YouTube vids here and Guiseppe’s session recording is here), at the end of which he gave everyone a copy of his book for nothing, except the ask that we donated some money to charity for it. Very honourable indeed!

      I suggest if you do download the ebook from the above link, you do the same for your favourite charity! If you are struggling to choose one, I suggest Willen Hospice, who provided amazing care to a family member of mine recently (Donation Link Here).Anyway, the session was excellent and Guiseppe gave some insights into the growing adoption of OpenStack in the Enterprise today. In fact it led me to post the following tweet at the time:

      Guiseppe’s book is a great intro to all of the basics elements of OpenStack and what they do; well worth the cost of a donation for a download!

  • leaky4The Leaky Establishment – David Langford (or eBook here)
    • As an ex-press officer in the civil nuclear industry, Pratchett described this as the book he should have written!
      The satirical black comedy focuses around our hero, Roy Tappen, who accidentally smuggles a “pit” (i.e. a nuclear warhead core!) out of the nuclear weapons research facility he (regrettably) works in!

      Needless to say, his wife is none too impressed with him keeping a multi-megatonne explosive source in the house, and hilarity ensues as Roy plots to smuggle it back into work!

      Parts of this book had me in stitches; well worth a read!

  • openstack-cloud-computing-cookbookThe OpenStack Cookbook – Kev Jackson & Cody Bunch
    • I currently have the second edition of their book so it’s not 100% up to date, but as I was on holiday I wasn’t actually running through the labs specifically. Instead, I read the main content in each section to get a better understanding of how each of the OpenStack components connect together.

      The book is very well researched and written, with clear and easy to follow instructions for you to build your own OpenStack homelab. I will definitely be upgrading to the Third Edition when it comes time to build my own lab!

  • evolutionmanThe Evolution Man, Or, How I Ate My Father – Roy Lewis
    • This is one of the strangest books I have read in a long time, but a really enjoyable read! Originally written in 1960, it is a story about a tribe of cavemen of the Pleistocene era, trying to pass through multiple evolutionary leaps within a single generation, and covers everything from their discovery of fire, cooking, improved hunting techniques, domestication of animals, etc, but ultimately it is a story about the friction between progress and those who wish to avoid it!You might be wondering how the author manages any compelling dialogue with prehistoric tribespeople? The good news is, that’s the best bit!

      All of the characters speak as if out of the pages of a 1920’s period drama, or perhaps even the drawing room of Charles Darwin himself! The juxtaposition of the characters and their dialogue is really what makes the book so special in my opinion.

      AFAIK this isn’t available in eBook format, but in this case, I think good old fashion print just adds to the anachronistic experience! 🙂

  • SecondMachineAgeThe Second Machine Age – Erik Brynjolfsson & Andrew McAfee
    • This book blends analysis of the history of technical innovations, with economics. It’s not my usual type of read, but it turned out to be fascinating on multiple levels.

      The geek in me enjoyed reading about the developments in technology and analyses of how they impacted the modern world, along with the predictions about where and how the authors believe technology will change our future.

      The parent in me took a lot of great ideas about how to advise and guide my children when they get to the age that they need to start thinking about their careers and university choices. One of the key recommendations made in the book was how people can remain valuable knowledge workers in the new machine age: “work to improve the skills of ideation, large-frame pattern recognition, and complex communication instead of just the three Rs”. If you want to understand this more either for your children or yourself, I definitely recommend you read this book!

So what’s next on my list I hear you ask? (Well maybe not, but I’m going to tell you anyway!)… The Tin Men by Michael Frayn (another Pratchett recommendation), most likely followed by Google’s recent Site Reliability Engineering publication.