Tag Archive for architecture

TekBytes #2: The Complexity of Public Cloud Architecture


For many organisations, the cloud and cloud-native application refactoring is attractive. This is often due to the belief that it will reduce complexity and risk for them, when compared to running their own DCs. The theory being that public cloud architecture is simpler.

By going all in,¬†however, many modern “cloud-native” applications are built upon a multitude of solutions, services and elements. This could be anything from a third party PaaS / SaaS provider for ID management, to “rolling your own” caching and search solution. It could even be simply implementing a broad set of management tooling for code and infrastructure automation.

The diagram below represents the technologies involved in one such solution. It’s clearly a highly distributed application with dependencies across many different platforms and cloud-vendors! It’s also not the only example of a solution I have seen in the new cloud-native world!

The risk is, the failure of any single one of those SaaS, PaaS or IDM platforms, automation tools or API gateways could leave an application offline and its owners potentially powerless to resolve it! Developers are exchanging the complexity of building elements into their applications natively, for the risk of distributing (out-sourcing?) them out to other cloud platforms.

Public cloud architecture isn’t always simples!

That is not to say this is not a reason to go to cloud and refactor applications to be more cloudy! The relative benefits to an organisation may far outweigh the risks. The key thing is that in any organisation, requirements from the business will always trump any expectations of simplicity or even consistency!

We are simply exchanging one set of complexities for another!

Thoughts? Feel free to discuss in the comments below!

TekBytes: A Blogging Experiment


I don’t know about you, but some of my best and worst ideas come to me when I’m in the shower… it’s quite possible this may be the latter, but let’s see where it goes!

For those of you who are either regular readers of this blog, or perhaps even know me in the “walking, talking flesh sacks” world, you will probably have noticed I’m prone to long-form communication; whether it’s writing, or indeed speaking!

Due to many reasons I won’t bore you with today (but maybe later!), life has been spectacularly busy the last few months. This has led to something which I want to correct; missing out the enjoyable act of blogging here!

What’s the plan, Stan?

In response I am going to try a little experiment based on the theory of “little and often”.

In addition to my traditional “epic saga” posts, I will be producing a new post series I’m calling #TekBytes. Not quite Twitter-style microblogging, but more regular, bite-sized chunks of content. No more than a few paragraphs or a couple of hundred words per post, based on observations and challenges I see day to day in my role as a multi-cloud solutions architect.

That doesn’t mean it will all be cloudy of course, just whatever comes to mind and I can get down into a post in a few minutes, possibly even from my phone! Some of them might even only be questions for you, the readers!

And before you ask… of course there will still be terrible memes! ūüėÄ

terrible-memesThoughts? Feedback? Make yourself heard using the comments below!

Docker – State of the Nation (aka Observations of a Brit)

Docker Logo

It may surprise you to learn that Docker is actually quite old now (at least in Startup terms!), having released the first version of their very cool software in March 2013!

Throughout that time, Docker (the company) have moved at a fairly rapid pace in terms of feature and etween

ug releases, with an average of a point release about every quarter and minor releases every month (or more)!

Whilst sitting here awaiting my flight to VMworld Europe 2017, where there are MANY MANY MANY (MANY!) sessions on Docker, Photon, Kubernetes, etc on the session schedule, I am prompted to consider Docker’s rise to popularity, and finish off a post I begun a few months back after Tech Field Day 12!

Well¬†come on Galbraith… get on with it then!

My experience in UK IT industry over the last (nearly) 15 years has taught me a few things, one of which is that whenever new technologies begin serious adoption in the US, it usually becomes popular in the UK within 2-3 years. That said, this number has been squeezed down a little in the past few years as companies move towards more agile development and deployment methods. Fail fast is becoming the mantra of many more organisations, though some people I speak to still wake up with night sweats at even the thought!

The first time a customer asked me about Docker in the UK was over 3 years ago, yet in all that time, people I talk to outside of the social media bubble many of us live in have been virtually silent about it; that is until now. Docker is becoming a weekly conversation topic now with a lot of organisations I talk to, with a many people wanting to jump on board the band wagon. The switch from an operating system-centric view of the world, to a more application and service-oriented (or should that be microservice-oriented) view of the world is becoming far more prevalent in my experience.Docker Swarm

Drivers to Docker Adoption

So what is it about this Docker stuff which seems to be catching the attention of people I talk to? A few common themes I hear are:

Automation of code deployment pipelines (CI/CD) to increase business agility
I think this is probably the number one driver to Docker adoption for people I talk to. Automation of CI/CD pipelines has become so common now, it is almost becoming the norm. Yes, it is tricky to do this with more traditional applications, but it certainly isn’t impossible. Using containers as the delivery mechanism for your application provides very consistent and repeatable outcomes. I mean, you can even get Oracle DB in a container now?!?!

That said, once you dockerise your applications there are many further challenges you will run into, including something as simple as how to apply your current security tooling, policies and proceedures to these new environments.

Maturity of the platform
The Docker code base and third party ecosystem has finally reached a point of maturity where many of the networking and storage issues of the past are beginning to reduce to within acceptable risk boundaries.

Improved cross-industry support
Following this maturity model, a swathe of vendors have put their names behind the Docker ecosystem; from VMware to Openstack, AWS to Azure, Google to Cloud Foundary, everyone is getting on board! You no longer have to buy support direct from Docker (the company), but can instead get it from your cloud vendor, along with a managed orchestration tier too, such as Docker Swarm, Kubernetes or Mesos!

Because Cloud
Yes, you can Dockers your existing applications for use on premises, but many organisations I speak to are using Docker as a method to allow their developers to write code on premises, test in their dev environments on prom or in the cloud, then deploy in a consistent fashion to their brand spanking new Production cloud platforms. PaaS solutions such as Azure WebApps and AWS Elastic Beanstalk are becoming a good option for customers who just want to write code, but for those who want that little bit more control, Docker gives them flexibility and consistency.to the cloud

CIO/CTO CV Padding
I hate to play the cynic, but I think there is definitely a significant¬†percentage of CIOs/CTOs who are doing “digital transformations through containerisation and cloud” specifically to pad out their CVs and help them get a better gig.

This is otherwise known as a¬†“Resume-driven IT Strategy”!

I am aware of one CIO who deliberately went to a cloud platform, even though it was significantly more expensive than a traditional managed hosting solution of a similar spec, when their business case and steady workload drew few, if any discernible benefits from the use of cloud.
CIO CV Padding When I hear people refer to technologies such as VMware vSphere as “Legacy” it really drives home to me the shift we are all going through, yet again, in the industry. This is another reason though which CIOs/CTOs/Heads of IT tell me they want cloud and containers. That said, I still struggle to find a single person who doesn’t have at least one physical server in their infrastructure, so just like the mainframe before it, I don’t think the hypervisor is going away any time yet!

The Tekhead Take

As expected the lag of a couple of years from the US to the UK in adoption of containers was apparent, but now is most definitely the time! Despite both positive and negative reasons for integrating it, Docker has become the part of the information technology zeitgeist in the UK…

Want to Know More?

I was fortunate enough to meet with the product team from Docker at Tech Field Day 12 towards the end of last year. It was a really interesting session which covered many of the enterprise networking and security features recently introduced to the platform, along with Docker’s new support offerings. I highly recommend checking it out!

Docker Presents at Tech Field Day 12

Some of the other TFD12 delegates had their own thoughts on the session and Docker as a whole. You can find them here:

Disclaimer/Disclosure: My flights, accommodation, meals, etc, at Tech Field Day 12 were provided by Tech Field Day / Gestalt IT, but there was no expectation or request for me to write about any of the vendors products or services and I was not compensated in any way for my time at the event.

Amazon AWS Tips and Gotchas ‚Äď Part 10 ‚Äď EFS (Elastic File System)

Continuing in this series of blog posts taking a bit of a ‚Äúwarts and all‚ÄĚ view of a few Amazon AWS features, below are a handful more tips and gotchas when designing and implementing solutions on Amazon AWS. This week, we talk about the latest feature of AWS, EFS (aka Elastic File System).

For the first post in this series with a bit of background on where it all originated from, see here:
Amazon #AWS Tips and Gotchas ‚Äď Part 1

For more posts in this series, see here:
Index of AWS Tips and Gotchas

20. Amazon AWS Tips and Gotchas ‚Äď Part 10 – EFS¬†(Elastic File System)

A big challenge when designing highly available web infrastructures is historically how to provide a centralised content store for static content without wasting resources.

A classic model for this is a pair of web / file servers with either rsync or Gluster to replicate the content between them. In Windows world, this would be something like either a WSFC (failover cluster) or perhaps something evil like a DFS replicated share. This means that not only are you wasting money on multiple virtual machines / instances just to serve file content, but you also add significant risk and complexity in the replication and failover between these machines.

Enter, AWS EFS!AWS EFSAt a simple level, EFS is basically an NFS (v4.1) share within the AWS cloud, which is replicated across all AZs in any one region. No need for managing and replicating between instances, or indeed paying for EC2 instances just to create file shares! Great!

As this is still a relatively immature product, there are still a few “features” to be aware of:

  1. There is no native EFS backup solution (yet!). I’m sure this will come very soon. As we have Re:invent coming up, it wouldn’t surprise me if something came out then. In the meantime, your main methods would be either to use Data Pipeline to backup to another EFS store¬†or potentially mount EFS and backup through an EC2 instance using your own tools or scripts. I would be concerned about backing up EFS to EFS (if in the same region), as this is putting all your eggs in one basket. Hopefully, AWS will provide other target options in the¬†future.
  2. There is no native encryption of EFS data as yet. If you need this right now, you could achieve it by simply pre-encrypting the data in your application first, before it is written to EFS. Alternatively, just hold your breath as AWS have already stated that:
    “Amazon EFS does not currently provide the option to encrypt data at rest, but we will offer this option soon”.AWS EFS Meme
  3. If you have less than about 100GB, then due to the way the performance burst credits work you may not get the performance you need. The more you buy, the more performance you get, so don’t short change your app for the sake of a few dollars!

    ‚ÄúAmazon EFS uses a credit system to determine when file systems can burst. Each file system earns credits over time at a baseline rate that is determined by the size of the file system, and uses credits whenever it reads or writes data‚ÄĚ

    In early testing, it has been seen that very small filesystems can lead to IO starvation and performance issues. I would recommend you start with 100GB as a minimum (subject to your workload requirements of course!). This is still pretty cheap at only about $30-33 a month; a lot less than even a pair of EC2 instances, never mind the complexity reduction benefits. KISS!

    Of course, the more caching you can do on that content, e.g. using CloudFront as a CDN, the lower the IO requirements on your EFS store.

    For more info on performance see here:
    Amazon EFS Performance

    kiss - Keep it simple stupid EFS

  4. And finally… being NFS based, this is obviously primarily aimed at Linux solutions. It would be nice to think that AWS will release an SMB version in the future… we can but hope!

Thanks to my learned colleague Tom Ellis for the tip! As he says, “The size needs to be determined by the throughput needs, and not the storage capacity needs. “

Find more posts in this series here:
Index of AWS Tips and Gotchas

%d bloggers like this: