Big Data represents immense data volumes with huge economic significance. It represents a major opportunity for almost every market and sector. As such, it also represents a major change in how we do business today and in the future.
Enterprise management companies such as IBM estimate that we are creating 2.5 quintillion bytes of data – so much that 90 percent of the data in the world today has been created in the last two years alone. According to a June 27, 2011 press release issued by Gartner Inc., worldwide information volume is growing at a minimum rate of 59 percent annually.
We all dream about the ability to answer questions that previously were beyond reach. And most of us realize the upside to dealing with Big Data volume challenges is the immense opportunity to proactively anticipate issues, gain new insight into emerging types of applications, make faster business decisions, and to gain deeper knowledge of consumer behavior patterns.
But managing Big Data is not just about planning for massive volumes of content – it is also about timely access to information and the ease in which you can establish analytical context. This means quickly pinpointing the physical origins of data, understanding the performance of the applications involved, and mapping out the relationship “structure” between individual data exchanges, and with consumer actions.
From a management perspective, Gartner defines “Big Data” challenges as 3-dimensional:
Volume (amount of data): The increase in data volumes and information within enterprise systems is caused by transaction volumes and other traditional data types, as well as by new types of data. Too much volume is a storage issue, but too much data is also a massive analysis issue.
Variety (range of data types, sources): IT leaders have always had an issue translating large volumes of transactional information into decisions — now there are more types of structured and unstructured information to analyze — mainly coming from social media and mobile (context-aware). Variety includes tabular data (databases), hierarchical data, documents, e-mail, metering data, video, still images, audio, stock ticker data, financial transactions and more.
Velocity (speed of data in/out): This involves time sensitive streams of data, structured record creation, and availability for access and delivery. Velocity means both how fast data is being produced and how fast the data must be processed to meet demand.
While volume is well appreciated, the other challenges represent real-time aspects of place, time, and circumstance that are more difficult to grapple with. But they also represent much of the value.
Simply put – Big Data has context. It is generated within specific environments, at a specific time, by specific sources, under specific conditions, and with respect to other data. The value of Big Data is semantically dependent – information means something to us in the context of its production. Some context is captured within data itself – for example the cost, the name of the consumer, or the item purchased. But much of it derives from relationships between the data and other circumstances – such as time of day, geographical location, customer history, device used, and social trends. Managed out of context, Big Data can lose that value. It becomes just so many numbers, words, and blobs of data.
Data context may include:
Delivery mechanisms and infrastructure – how data is generated, collected, and delivered can affect its significance. Was critical data lost by over-loaded networks? Was excessive delay in response due to the network, application, or the end-user? Was any part of the system unavailable?
Application design, implementation, and integration – how systems that generate data are designed and constructed affect what and how it is collected. What are the dependencies between the different services within the system? Are 3rd parties meeting service level requirements? Were all aspects of the system meeting various compliance requirements?
End-user experience – data often relates to an experience that an end-user was having. How much of the data reflects a positive end-user experience versus a poor one? What parts of the data did the end-user provide as input? Can you tell from the data what the end-user was doing?
Federation and business dependencies – data may reflect the impact of different stakeholders. What businesses were involved or responsible? Who has dependencies on the data and the outcomes of related processes? Were stakeholders actively responding to the data or simply monitoring?
Business process – data often directly relates to business objectives and processes. Are the relationships between data and objectives observable and can the data subsequently be analyzed in terms of business performance? Can the data related to a specific process be distinguished from other processes? Can the parts of a business process not present in the data be accounted for?
So can we manage and exploit this huge potential? Can the benefits associated with Big Data be reaped without blowing your entire 2012 IT software and resource support budgets? The simple answer is, YES.
Big Data is certainly big. And tackling the volume issue is central to the opportunity. However capturing and analyzing the context of Big Data is also critical to trending and analytics – otherwise the full value may be lost. Does that simply mean Big MetaData? Perhaps, if off-line processing is the best practice for a given situation.
But the trend for Big Data is to manage and monitor in real-time, with “now-casting” displacing traditional fore-casting, and to drill into the resulting analysis from multiple transaction perspectives. Real-time, easy-to-deploy business transaction management (BTM) solutions are already available that will help you do just that, without blowing your entire 2012 IT software and resource budgets. By applying real-time transaction profiling and analytics, you can manage all three of dimensions of the Big Data opportunity while achieving all of the business objectives. The focus shifts then from the data itself to the means of gathering and analyzing it – and to the frameworks that enable contextual analysis to be imbedded in the systems that generate it.
Big Data is not just more data – it is better data. Contextual data. Semantic data. Data with meaning. For an example of using a business transaction management solution (BTM) to model Big Data within financial transaction environments, watch:
[youtube]http://www.youtube.com/watch?v=B9_QsgGO0bg&list=UU4_ik_RVWz0HkTeaqz7d7EQ&index=2&feature=plcp[/youtube]