Big Data (Photo credit: Kevin Krejci)

I love technology. That’s why I study and write about it. I’ve spent my entire professional life around data storage systems. I’ve sold them; I’ve designed them; I’ve resold them; I’ve deployed, managed and used them; and I’ve analyzed the vendors who make them. Leading market research firm, IDC, recognized me by hiring me to lead the worldwide research of storage systems; and I am grateful for my experience there.

IDC also gave me the opportunity to step up and initiate and lead the worldwide research for Big Data. I think that job was singularly one of the most challenging in my career. At least at IDC, I had the benefit of having subject matter experts who could help me. I might have had to provide input into how technology adjacencies would come together, and in my role I was also responsible for bringing together a cohesive and definitive position for the company, which made the job both challenging and exhilarating.

Now that I run my own firm covering Big Data, I realize how much morechallenging being a Big Data analyst is. Big Data is not a technology. It isn’t even a process; but an umbrella moniker that encompasses the people, process and technologies used as well as the inputs collected and the outcomes generated by these people, process and technologies.

To understand Big Data fully, it requires an understanding of the hardware, the software, the statistical and analytic techniques, the various types of data inputs, and how to ultimately present or visualize the outcomes.

During Strata 2013, I had nearly 30 vendor meetings in three days. I can’t thank my assistant Gretchen enough for scheduling the meetings and keeping me sane! But the meetings I had, brought home for me, what Big Data truly involves – which is, a lot of moving parts. But it also demonstrated to me that the whole is most definitely greater than the sum of the parts.

Here is a list of the types of vendors with which I met:

• Hadoop distribution vendors;
• Data integration vendors (for example, bringing SQL to Hadoop, and/or integrating relational with non-relational data sets);
• Statistical programming language library vendors;
• Application performance management vendors;
• Database vendors;
• Business intelligence and analytics vendors;
• Infrastructure vendors;
• Data-as-a-Service vendors;
• Consumer marketing analytics vendors; and
• Publishers of books and manuals that teach you about all the above.

This is list by no means exhaustive. In fact, these are only the vendors I can remember at 4am in the morning!

In full disclosure, I have to admit that some of the things that these vendors are doing are way beyond my expertise. I haven’t written code since I left high school; and even then, my programming can best be described as “messy”.

My experience at Strata can only be described as overwhelming. I can now truly understand why there is such buzz around Big Data. Information Technology (IT) has evolved to a point where we are performing discovery and analytical tasks that we once thought were not possible.

We used to call it Artificial Intelligence (AI). Movies were made about the remote possibilities of what our minds could only imagine. Today, we don’t refer to it as AI; we call it machine learning. We can, (if you are one of the smarty-pants who understands the analytics) program a computer, an affordable computer, to test permutations and prove hypotheses that no human could ever possibly do in a lifetime. These programs can be written in a very short amount of time, using only a small slice of a modern computer’s capabilities.

Nearly 3,000 attendees were at Strata. The demographics of the attendees does not fit any normal bell curve. My estimates indicate nearly 30% of the attendees were women. There were a good mix of business and technical attendees. Not all of the technical attendees were your typical long haired, t-shirt and flip-flop wearing types. (Actually, I don’t think I spotted a single pair of flip-flops!)

The giveaways at the booths were not ground breaking. There were t-shirts, a few USB thumb drives, a bag, buttons, and stickers (there was one vendor who gave away green elephant plush toys). Apart from Microsoft, who raffled away an Xbox, the prizes for raffles were quite modest. – I don’t recall any iPads being given away. (That said, there wasn’t a single attendee whom I observed who didn’t have a tablet of some sort!) Attendees went to each booth with a genuine desire to understand each vendor’s offerings; and many attendees took the time to visit most of the booths.

The caucus of attendees, like me, were all in awe of the various technologies being presented. Now, obviously the attendees at Strata are die-hards. But these are also the professionals who are going to influence their colleagues when they return to their day jobs and inspire their coworkers, their bosses and their organizations to imagine the impossible and to make it happen.

I’m

not normally this nostalgic about technology conferences. But I leave this conference appreciating my ignorance, satisfied with my new-found learning, and hopeful that I can and will contribute to the innovation that Big Data will bring to each of us – financially and socially.