Making Automation More Effective with Dynamic Environments

Chris Grabosky | chris.g@quali.com
Principal Solutions Architect @ Quali

Risk of Catastrophic Failure is Real

Bug dog stories aside… the risks are real. Amazon Crash – Feb 2017 May 2017 Cairns Hospital – Security patch gone wrong May 2017 - British Airways – Disruption for 75,000 passengers And this just highlights the level of exposure we have today with broad, global digital systems… Why is this? One of the main reasons is that applications are becoming more complex. And complex and complicated are not the same thing. Complicated is kind of like an engine… monolithic with lots of moving parts. But complex is another level. It is distributed, made up of lots of small components – often ones not even written or handled by you. The goal was to make independent pieces simple. However the end effect is that the overall solution is quite complex. And at the same time, in today’s hypercompetitive and changing landscape, we’re being told to release as fast as possible. We’re being told that we need to automate and get release out the door as fast as possible.

Real World Story

$1B Retail Paper Supply Company

Traditional Enterprise undergoing digital transformation
Adopting DevOps and cloud with production and dev/test in Azure
Distributed Global R&D
Enterprise Scale Dev/Test (100+ developers & testers)

So we all see those fail stories in the news, but lets look at a simpler situation affecting one company. However I’m sure that their story is probably something you have heard before. This, believe it or not, is a paper company. And they would hate me if they heard me say that. No, they are a software company. Right? UPS isn’t a shipping company. No. Their differentiator in the market is their logistics, and the same way here, this paper company’s differentiator is their software that provides efficient and JIT service. When it works. See, they are undergoing a transformation with DevOps. I explained where they want to be, but they need help getting there. The biggest issues were on the right here on this slide. So they now have a global R&D presence with over 100 developers and testers, so they have truly scaled to make this happen. And happen securely. Lots of turn over with contracts. Lots of huge Azure bills. Pain points: So first – their practice of using static environments wouldn’t allow them to scale. It was taking them way too long to get environments to developers and testers. They also had quality issues - bugs were often not being found until pushing into production They also had $millions in cloud consumption costs because these static environments were “always on” and end users have no incentive (actually negative incentive) to releasing; out of environments So they realized that they really need to enforce standards and access to environments.

Dev/Test/Team Lead

"I need access to a pre-configured environment ASAP"

"Oh, and five contractors just left"

DevOps Team

"I need to improve the productivity of my dev/test teams"

Case Study – Dynamic Environments

Turbo-charged Agility > Improved Reliability > Reduced Costs

Remove dependency on static environments and the overhead of maintaining them with standard blueprints
Environment orchestration flow using a building block orchestration platform.
Integrate environments with other ecosystem tools

So breaking this down into two phases: Blueprint and standardize environments for three main use cases relevant to SCRUM teams: Dev Environments Feature validation environments for testers and automation tools. CI Nightly Environments for CI tools / nightly build So remove dependency on static environments and different environments and the overhead of maintaining environments and as a result showed higher velocity as well as reliability and cost savings. Then phase 2 was all about defining the building blocks to integrate with the overall CI/CD tool chain So we didn’t want to displace the exiting CI tool set the customer already had… which was Team City; Instead we used our open source modeling templates to integrate them directly into the blueprinting framework. So team city was used as their CI tool – which would automatically deploy dynamic test environments. Octopus used for production release We defined the orchestration workflow using building block based orchestration platform.

Challenges

Based on Quali 2016 survey of 2045 respondents

The data says differently: 26% over a month to deliver infra; 59% over a week. Is that suprising? Maybe not – especially if you’re a more traditional enterprise… 68% complexity impedes agility 46% -Lack of self service People know who Jez Humble is? Author and researcher on SLDC: He said He said the best proxy measure, for him, of developer productivity, is lead time. Lead time would be measured by, “how quickly can we go from code committed through the test cycle were everything is running in a production-like environment and being deployed into production?” Looks like the piece where the software runs on is still the slowest part

How do we get there?

Ad Hoc
Waterfall Processes
Piles of Scripts
Manual Processes

-->

Continuous Integration
Continuous Test/Delivery
Continuous Load Test
Continuous Security Test
Continuous Perf Test
Continuous ____ Test

SO, how do we do automation in a way that gets us from here to ci, CT, and CD? That's the question. How do we make automation effective to get us from here… to here With all of the complexity here… Ephemeral testing environments, containers/Microservices. security/networking/data store, Service Oriented Architectures, N-tier… The behavior is not as predictable the way a large monolithic application is. That’s complexity. So we’re just not penetrating automation to the level of complexity that our applications are moving toward. We need to automate this - We have to automate this if we want to get here…. So the answer may be simple. Just build out a test bed and write unit tests for each component.

Unit Test and Static Environments Fantasy Land

One approach is to pretend there’s not actually a complex software system that needs to be addressed in the continuous test process. I call this “Unit Test Fantasy Land”. This is real. It’s very tempting to live in an imaginary land and ignore the fact that today’s software is very complex and that we need to account for the whole system. And the problem is that Unit Test Fantasy land kind of works... Until it doesn’t That’s the problem with unit test – doesn’t account for the whole, complex system. Gives you a false sense of assurance. Another very common approach is to just build up environments statically. So a developer or tester... Or environments for feature releases... Built statically in kind of a manual or semi-manual way. Resources are pre-allocated, and assumed your environment will not change. Doesn't scale... Developer environment for 100 developers needs one change... Inefficient. It’s also very Costly. There’s also significant Configuration drift... Developer or tester kind of tweaking environment over time, unreliable. Leads to WOMM (works on my machine). Static environments are also not very repeatable as you’re gonna miss a piece when it rolls to production

Effective Automation: Dynamic Environments

So, what we’ve found to be the best approach to making automation effective at really bringing an organization to true continuous test and continuous delivery : dynamic environments. So what’s the best practice for doing this? In our experience of helping our customers adopt devops and cicd we've come to identify 3 best practices to get dynamic environments in place for more effective automation STATIC Setup once (manually) Resources are pre-allocated WOMM Not repeatable Fixed configuration DYNAMIC (Sandboxes) Resources allocated on demand and reclaimed automatically Can select what’s saved between sessions Repeatable Flexible configuration On-demand orchestration

Best Practice 1

Create Modular Automation Components

First - build higher order modular automation components that are reusable. We’ve found that you can break down automation assets into key components: So first we have the infra – what tools and mechanisms you use for infra provisioning.. On-prem, cloud, physical… Look at common processes: going to Azure? AWS? VMWare? OpenStack? Docker? We can still break down that infrastructure deployment to the steps to deploy the app or VM and steps to handle the networking or communications. Doesn’t matter where it goes. Then what about state? Tons of great tools out there already, whether that’s Ansible or just a shell script. Don’t re-invent the wheel – USE WHAT YOU HAVE , integrate it with a step called “configure” And repeat this process of every step of the lifecycle of the sandbox TOSCA is great for this. So the way we do this is by defining those components as Tosca based models – and they’re open sourced and very easy to extend. So we create standard templates that allow you to start modelling all those existing components that you have in place.

Best Practice 2

Build Repeatable Self-Service Blueprints

Second – Now put this in reusable blueprints Use that same blueprint or set of building blocks for every step of the CI/CD process – Developers, testers, QA, security, automated testing… all use the same blueprint This ensure standardization and uniformity and means the orchestration is now part of the process and a first class citizen. No tribal knowledge. No creep of out of date CM So you build this catalog of blueprints and what’s better than seeing it and pressing a button to get it on demand? Make it available via API. What is your API strategy? Here, everything should be consumed via this one REST API. So whether a user asks for that blueprint via a UI or Jenkins does it via the REST API, you are getting the same thing . It scales Its secure and is driven by policy Consumption is managed

Best Practice 3

Allow Live & Historical Environment Context

Fourth – The blueprint provided a great insight. Keep this context of the environment When we make these live in CloudShell, we call it a sandbox. I’ve actually seen this terminology pick up in recent years. So regardless of how you implement this, the context of this is most important. Again, what is your API strategy? There are tons of things in the sandbox now. The VMs, containers, test or traffic tools, complex networking…. Let your automation platform using those reusable functional components we mentioned in practice 1 handle that. So any other system externally touching this sandbox, just has to deal with the contect of the sandbox… No need to reach out directly to those APIs for VMWare or whatever. You handle it all through the sandbox. One of the key concept of the sandbox is it statefulness. In other words you get access to the metadata during your test cycle and you get a fast feedback loop to adjust your code. Let’s assume that the test fail, then I need to do some troubleshooting with a specific resource in the sandbox. In this case, that means I get a direct link to the sandbox resource log or console relevant to my error (event correlation). After I resolved the issue I can create a blueprint from my updated sandbox (optionally) and rerun the test against the new blueprint. Fourth –Record and correlate everything! This is important for business people paying the bills… what were our public cloud costs? What did it come from? Who was doing it in which sandbox? Did that find any bugs? What was the bug ID in Jira? Which build ID from TFS was it? Pull all these different data sources together through the context of the sandbox.

How Dynamic Environments Help

Now I want to pull this together and take a look at a typical CI/CD pipeline and see where dynamic environments fit in. How many tools are in this thing? What is your strategy for controlling that? What is your strategy for handling the different taxonomies for each tool?Cloudformation… ARM template… Inventory YAML… and did each team agree to use the same tool? How do you glue it all together? Take a step back and think in the uniformity that blueprints bring and offer as dynamic environments. It actually simplifies it doesn’t it? Write a wrapper (we call them shells) for the tool, expose a common interface for the API and UI, and put it in the context of a blueprint. Each phase then just tackles that one API using that common blueprint and can swap in and out components. So for Functional testing I would have a suacelabs service Load testing I’d use a blazemetere service Security a whitehat secutiry service… All using the same blueprint. And at the bottom you still have lower level infrastructure that’s going to be consumed by the blueprints, but these are components that you can choose and define ahead of time.