Splunk : The Big Data Campaign Trail

Today Splunk announced the results of a Public Sector IT research project, The Big Data Campaign Trail - Every Data Point Matters for Better IT Operations.

Having been in IT Operations for a couple of decades now, I found the insights very interesting because I believe agencies and educational institutions are going through a fundamental digital transformation which is bringing them to a crossroads. As an architect who works with our public sector customers' daily I want to share my thoughts on the results over a series of three blogs. This first one will highlight the major findings, while the following two will offer some insights based on my experiences that I hope you find helpful and actionable.

The major revelation was that 71% of respondents felt that insights from IT operations data are extremely or very important to their organization's overall mission. But 67% said they either don't or don't know if they have holistic visibility across IT systems and can't foresee issues ahead of time.

51% of respondents believed that new IT paradigms are adding to the organization's complexity. Agencies and educational institutions are seeing the effects of new technologies first hand - cloud-first, BYOD, DevOps, virtualization - they are fundamentally transforming their organizations and while they offer great benefits, they are challenging the day-to-day practices of IT shops. And this transformation is not expected to slow down - 69% claim they expect use of COTS products to increase next year while 66% plan to either expand or start using private cloud solutions and 58%, the same with the public cloud.

Some other major insights include:

• The perennial resource shortage drove 44% of responds to claim it as the biggest risk to their organization over the next year
• 58% claimed their troubleshooting processes were manual and ad-hoc
• 50% said that different formats and types of data and the difficulty in normalizing it was the biggest challenge to using it to diagnose IT issues while 40% viewed data ingestion and normalization to be cumbersome and tedious

Over decades, organizations have built silos of operations - extended with heterogeneous technologies over distributed locales and with teams that managed them as such without much collaboration. So it is interesting to see that IT operational personnel are waking up to the need for insights and end-to-end visibility. This signifies that there is a fundamental shift in thinking on the strategic approach to managing operations. Silos don't work, we need holistic visibility and need to collaborate. And in an increasingly complex environment, this becomes an imperative so you can have a better handle on what is transpiring across the organization.

But we clearly have a dichotomy here. Monitoring and troubleshooting in an increasingly changing IT landscape requires agility. Staffing alone will not solve the problem - we know of many organizations where people are available but lack the skills and know-how to operate what they have effectively. What organizational leaders should be looking for is to bring in solutions that can help alleviate some of the low level yet time-consuming tasks - data ingestion, normalization, filtering, etc. - so valuable analyst time can be spent on gaining situational awareness in real-time, prioritizing the most important issues, pinpointing problems and root cause and making decisions on response. IT operations need to find the best way to troubleshoot without being overwhelmed with technology, be able to embrace innovations that can benefit the organization and ensure resilient operations so the enterprise can deliver the best experience to its customers.

[Attachment]

Look for my next two blogs in the coming weeks where I will explore specific results and offer thoughts on how to navigate the complex IT landscape.

Cheers,

Bill Babilon

IT Operations Architect