Guide to Building DIY Cloud Solutions

The rise of cloud computing changed the face of information technology forever. Before the cloud, infrastructure access required upfront capital investment and months-long lead times. After the launch of cloud computing, infrastructure was available in mere minutes at a cost measured in pennies per hour.
In turn, this infrastructure revolution spawned Web 2.0 – a range of companies born in the cloud. These innovators launched new offerings that substituted cheap digital technologies for lethargic analog processes, which resulted in wrenching disruption to large companies with storied histories. New verbs sprang up to characterize this disruption: Ubered. Or Airbnbed. Or Netflixed.
Of course, the incumbent players in these industries were unwilling to cede them to the upstarts. So they followed the disrupters into the cloud. But a curious thing happened to most of these efforts: they didn’t pay off.
Enterprise after enterprise launched big cloud initiatives, but found no acceleration in their IT processes and an inability to move at the speed of the disrupters.
It turns out that cloud adoption isn’t enough to move at the speed of Web 2.0. Fast infrastructure needs to be married to fast software practices. Put another way, the software path to production needs to operate at the speed of the underlying infrastructure resources.
Web 2.0 companies have typically constructed highly customized toolchains to squeeze the time required for a software change to go from keyboard to production. Common across Web 2.0 is the heavy use of open source components wired together with company-specific integration components to create a custom DIY solution.
In response, many enterprises have started their own DIY initiatives. But all too many of them never result in a workable solution, leaving the companies hindered in their efforts to move at Web 2.0 speed. The obvious question is why: what prevents so many enterprises from successfully creating their own DIY path to production?
I’ve observed many of these failed DIY initiatives, and would like to share the common issues I’ve observed – and offer suggestions to help tech professionals to create a streamlined path to production.
Also see: Top Cloud Service Providers and Companies
Beware the Downsides of Cloud DIY
What are the issues I’ve seen arise in large enterprise shops that attempt to build a DIY cloud computing path to production?
Here are a few of the reasons these DIY efforts fall short:
Component curation
There is such a dizzying array of open source components available that address various parts of the path to production that just selecting a good set to integrate into the tool chain is a major job.
The eyechart that is the CNCF landscape poses a challenging task just to list the options for different parts of the toolchain, not to mention evaluating and selecting a good one. Of course, there is an alternative – allowing an employee to choose one based on reputation or what someone he or she knows used – but that seems haphazard at best and high-risk at worst.
Underestimated integration costs
Some components have built-in integrations for complementary components so that the integration task would seem to be simple.
However, not all components that might be selected have integrations to other components an organization might choose. And, in any case, in my experience, assuming built-in integrations work well in all use cases and can operate at scale is misplaced confidence. The truth is, what seems like simple integration “glue” software is typically far more work that expected, requiring more labor and longer schedules than initially budgeted.
Failure to address the entire application lifecycle
DIY toolchain efforts commonly get started in one group that wants to “scratch its itch.” The group sets off to automate its work tasks, reducing operational cost and accelerating throughput. That’s fine as far as it goes.
Unfortunately, there is often no work to connect an automated workflow into the upstream and downstream work teams that also contribute to the path to production. So the transfer of one task to another ends up being a manual handoff. Quite often the individual automation work results in no overall speedup, since the majority of time in the path to production is spent between tasks, not within tasks.
Staffing shortages
The flip side of underestimating the difficulty of work and complexity of end-to-end automation is that insufficient staff is assigned to the project. These projects often start as skunkwork-type efforts, siphoning off part-time work from a few people in a work group; this avoids the overhead of “real” project creation like budgeting, scheduling, and so on.
As the real scope of the project becomes clear when other groups that participate in the path to production put forward their requirements, the skunkwork staffing capability is rapidly outstripped. Organizations then need to confront unpalatable choices: abandon the project – and thereby remain mired in a long-duration manual process – or devote real dollars to it, which requires documentation and budget competition.
Talent diversion
Every technology organization only has so many talented engineers. If those engineers are assigned to improving an internal process, they’re not contributing to the larger goals of the parent company.
Very few companies are willing to consciously decide to devote precious, scarce talent to an internal project, worthy though it might be. So the internal DIY project is starved for talent, resulting in delays or incomplete functionality.
Value delay
The beginning of this article pointed out the primary motivation for enterprises to adopt cloud-native practices: responding to market disruption caused by born-in-the-cloud competitors.
Taking on an expensive and complex DIY project that will be plagued by talent and budget shortfalls is a recipe for late delivery and delay in value. While a self-developed software toolchain provides pride of ownership, taking 24 to 36 months to get it into production prevents the parent company from responding to dangerous competition in the here-and-now.
Also see: Top Digital Transformation Companie
Guide to DIY Cloud: Doing DIY the Right Way
“Aha!,” you think. I want to avoid all those problems in creating my company’s automated path to production. But how?
When I talk with large enterprises embarking on a DIY path to a cloud production project, here are the factors I suggest they ensure their plan addresses:
Understand your why
What is the business reason a home-grown system is important? All too many times a DIY project is a way for someone to grow a fiefdom, which serves their interest, but not necessarily the company’s.
So understand what you’re trying to achieve by building your own path to production. And don’t imagine you’re a snowflake, unique in the world, that requires a one-off solution – or at least understand why you’re a snowflake to better gauge the importance of a home-grown system.
Most companies, if they’re honest with themselves, can be perfectly well-served with a generic solution for their path to production.
Incorporate all the siloes
As mentioned before, all too many DIY projects start within a single group and never address the needs of other application lifecycle constituencies. The old cliche is measure twice, saw once; in other words, plan before execution.
Before you start building, ensure the solution helps every group that touches code on its way to production automate its part of the process – or that the resulting system can easily incorporate those other groups’ own automated systems.
Otherwise, at best you run the risk of no speed improvement in your software practices; at worst, you run the risk of organizational strife as those other groups put forward roadblocks and refuse to participate.
Plan for ongoing maintenance
Recognize that you’re building a platform, and every platform requires ongoing work to extend functionality, incorporate bug fixes, and improve operations and resilience. This is a downstream budget commitment and it’s important to plan for it. Otherwise, you end up with an obsolete system, unsatisfactory to users, limping along.
Create a budget
I hope this article has underlined how an important system that supports your path to production is a long-term commitment and not something that can be slapped together by a few staff working part-time.
As you create your cloud project plan, assign headcount and plan for staff costs. Also calculate the infrastructure you’ll need to operate a platform; this means you’ll need development, staging, and production environments to allow platform improvements and pre-production testing.
Calculate your resource opportunity cost
The staff you assign to building your platform could be working on other things. What other things? What value could you achieve by assigning them to work on other tasks?
Every tech organization has an inexhaustible list of to-dos – are there to-dos with greater benefit to the organization or the company at large?
Perform a buy vs. build analysis
As I noted at the beginning of this piece, most Web 2.0 companies build their own custom path to production systems. It makes perfect financial sense for them as they are digital enterprises at their core and run very large volumes of code changes and releases through their systems.
Amortizing the cost of the technical resources and infrastructure across such volume means a home-grown system makes financial sense for them. The question is, does it make financial sense for you?
There are commercial products available that provide end-to-end path to production functionality. You can take your budget and opportunity cost numbers and compare them against the cost of a commercial solution and figure out which approach makes sense for your situation.
Also see: Cloud Native Winners and Losers
Key Takeaways: Building DIY Cloud Solutions
What are the most important points you should take away from this article?

Software competence is a key prerequisite in the age of cloud computing. Grafting legacy software development practices onto cloud infrastructure results in no increase in application velocity. Worse, it may result in higher costs than leaving a legacy application undisturbed. To compete in tomorrow’s economy, you need software chops.
You need to put a modern path to production system into place ASAP. Industry disruption typified by Web 2.0 companies shows no sign of abating. Failing to begin immediately poses a risk of being left in the dust.
Carefully consider your approach. DIY is tempting. It seems easy to get started and low cost via the use of open source components. It’s vital you understand the full implications of a home-grown system in terms of cost, staff commitment, and time to value.
Is your plan comprehensive? Ensure your ultimate solution addresses the entire path to production and encompasses all the tasks that a code update passes through on its way into production.

Keep these items foremost in your mind and you increase your odds of success enormously.
The post Guide to Building DIY Cloud Solutions appeared first on eWEEK.