Due to vast data volumes, AIOps becomes essential to understand how the production environment is performing.
For many years there has been demands placed on IT teams, and DevOps in particular, to be more proactive and flexible so that businesses can quickly accommodate and embrace new challenges in order to remain competitive. In effect, this has resulted in more processes being automated as manual intervention has been deemed too slow and cumbersome. This sea change in strategy has commonly been referred to as the digitalization of IT.
All of this began in the late 90s when the Internet became more of a commercial entity. However, in 2007, following the economic crash, the focus on automation sped up and it became a driving force to enable innovation and support revenue generation. Now, automation sits at the heart of many IT strategies and is undoubtedly a mainstream IT concern. Without automation, companies struggle to be competitive and as a result fail.
Therefore, the crash was very consequential for the relationship between business and IT. For the last eleven years there has been a demand from business for IT to be much more responsive and able to change much more quickly. It was the development teams which took up the challenge to change the dynamics and role of the IT department – no longer could it be seen as just a back-end function, it now needed to be proactive and revenue generating. There have been three significant changes in DevOps that are worth highlighting.
Firstly, there was a widespread adoption of agile development practices which is primarily about how to develop software and how to speed up the delivery of creation changes into the production environment. Secondly, DevOps incorporated automation into the process of change delivery. So, you have much faster development, complemented by the automation of the delivery.
Thirdly, if you accept the previous two changes, the next logical step is that development and operations teams should not be so segregated. Historically, operations team took care of the environment and DevOps managed development, meaning they worked independently of one another. It was assumed that changes were relatively rare and predictable and delivered discretely into production environments. But, with the advent of agility and automation, it was no longer possible to keep these two groups separate. They have been brought together in terms of practice, interaction and even in terms of sharing technology.
Was there resistance to this coming together? Absolutely. IT operations resented the close interaction with development and to this day there is a failure in DevOps to fully understand the difficulty in managing the production environment. DevOps believe it to be a relatively straightforward, low-level task. Also, the DevOps perspective on infrastructure is very myopic, they tend to be only interested in the specific patches of infrastructure that is relevant to any particular project they are working on.
This logic does not take into account that no application is isolated. In reality, the application is living in an environment of shared resources. You can’t just worry about an individual applications effectiveness of consuming resource, you have to query the impact that application has on other applications and services running in that production environment.
So, for development teams in general we are dealing with a degree of cultural resistance and naivety, which culminated in operations focussing on the protection of its own infrastructure against the incursion of the DevOps teams. The bottom line is that because of mutual misunderstandings, DevOps remains to this day largely siloed – there is not much Ops in DevOps. The linkage that needs to happen hasn’t in truth been forged.
Therefore, you need to have a technology in place which will rapidly analyze what the presence of this element means, and to quickly provide diagnostic back to production or IT operations on how to deal with negative repercussions.
How does automation feed into this relationship? As we know there are many steps that need to be undertaken for a new module or application to be moved from the development environment into the real-world production environment. Historically, these steps have largely been done manually, today automation takes the human out of the process. And here is the opportunity for AI or AIOps technologies like Moogsoft. The automation path from development to production can be executed in a far smarter way.
For example, if you have algorithms that are able to take data from the production environment and understand the current disposition of resources within that environment, you can ensure that the new application or change being delivered has sufficient resources to support itself, whilst not starving other applications of resource. That can be achieved via automation.
From my perspective there are two stages to the automation process. Firstly, there is the automation of the path from development to production which includes activities like pattern discovery, anomaly detection, causal analysis – all are features of the AIOps toolkit, but in this case applied primarily to resource allocation and timing decisions around the delivery of new components into the production environment.
Secondly, what happens when the new element is actually in the production environment? Let’s consider that we have vastly accelerated the number of changes being delivered into the environment, it is no longer a matter of three changes a week it’s more like three thousand a week. In addition, we have increased modularity, ephemeralness, and IT systems are now more distributed, so it becomes nigh on impossible to predict with a high degree of certainty what the impact of a new change on a production environment will be. Therefore, you need to have a technology in place which will rapidly analyze what the presence of this element means, and to quickly provide diagnostic back to production or IT operations on how to deal with negative repercussions.
Due to vast data volumes, development and operations teams are finding it virtually impossible to understand how the production environment is performing, unless they use some kind of analytical or diagnostic tool. Without a smart filter, the development cycle becomes ponderous and teams find it challenging to action events. This is when AIOps becomes essential – if you don’t deploy this type of technology the consequences can be severe. I started out outlining why companies initially chose to enforce an adaptable IT strategy, I will finish by saying that without AI assistance this strategy will collapse and companies will become increasingly blind to system performance.
About the author
Will studied math and philosophy at university, has been involved in the IT industry for over 30 years, and for most of his professional life has focused on both AI and IT operations management technology and practises. As an analyst at Gartner he is widely credited for having been the first to define the AIOps market and has recently joined Moogsoft as CTO, EMEA and VP of Product Strategy. In his spare time, he dabbles in ancient languages.