It’s not every day that you see the CEO of Pivotal and the CEO of Cloudera on stage together. Today Paul and Tom discussed where they see the future of Hadoop and how Isilon is helping them move Hadoop into the centre of the enterprise.

Paul Maritz’s comments were around the idea that Hadoop is now moving to be a critical platform for enterprises, hence the need for Isilon’s ability to make Hadoop enterprise ready.

Revenue for EMC Isilon in the Hadoop space is growing at 259% per annum.

Customers at the same session (FedEx and Adobe) talked about how Isilon drives down the Hadoop TCO by eliminating the “triple replicate penalty”, while improving availability through business continuity and eliminating data loading by being the central data hub.

Paul said it would take a while, but he can see a point where 50% of all workloads in the world are on HDFS.

The CEO of Cloudera, Tom Reilly, said that Intel’s view is that the #1 workload user of Intel CPUs today is ERP, but eventually it will be Hadoop.

A central theme was the religious debate of whether Hadoop should be used as purely a staging platform, with data being moved into warehouses for manipulation, or if Hadoop should be the manipulation engine. Clearly the view from Pivotal and Cloudera is the latter.

…..and why it is all going to change very soon.

Australia is a long way from a lot of things. This creates a unique environment where we tend to alternate between leading and following technology trends. Take cloud and virtualisation for example. Australia has had one of the fastest adoptions of virtualization technology in the world with over 60% of physical servers now virtualized. This is well above of the global average which is somewhere around 30-40%. (Interestingly NZ is upward of 90% virtualized).

The same is true for Cloud. When I benchmark the Australian Telcos and Cloud providers against those of US and European providers there is a definite early adoption of cloud locally. The shift towards new consumption models in Australia has been extremely fast.

I often hear the term from colleagues in the US that we are “the canary in the gold mine” meaning that technology trends tend to evolve early and quickly in our market place. Leaders in the US generally look to Australia to see how IT will evolve for them.

The same is not true however for big data. None of my US or Asian colleagues have ever said to me “tell me how your Australian customers have become so innovative with Big Data and Analytics”. Usually it is me asking them.

The early adopter scenario that exists with Cloud and Virtualization doesn’t extend to Big Data, data analytics and other trends that mean gathering and combining lots of internal and external data sources and doing something smart with them.

Maybe it is that Australian’s tend to be more skeptical, more risk averse or maybe we haven’t had enough exposure to the potential of the technology that we haven’t adopted it yet. Whatever it is, we are definitely lagging.

Why do I say this?

There’s a few common threads that I am seeing.

– Firstly there are plenty of projects going on in Europe, Asia and the US across many different industries and segments. Telco and retail organizations have been particularly fast to adopt new ideas around Big Data.

– our Australian based “Big Data Scientists” are spending more of their time offshore due to skills shortages there, with few projects going on here

– while some big data projects in Australia were implemented by various companies in 2012-2013, it is difficult to find examples in Australia where these have had major impact to revenue or market share

– one Australian organization that I spoke to that was doing some pretty cool stuff around big data was doing their R&D in the US “because that is where the skills are”

Recently I had a conversation with one of our Data Scientists who said that when he does do work here, he usually refers to himself as “a statistician or analyst, because Australian’s are skeptical when they hear the term big data”.

So that is the current situation, but in my opinion this is all about to change. In the US, companies from Walmart to Netflix to Target to ebay (successfully) use Analytics and data mining to drive customer recommendations. This is driving Australian business to rethink their adoption of the technology.

Towards the back half of 2013 we saw Australian retailers start to ramp up their investment and capabilities in big data (e.g. Woolworths investment in Quantium). In the finance sector a number of banks starting outlining their big data strategies, highlighting how better use of customer information and improved risk management will become a competitive weapon. See one cool example of big data project here from UBank: http://www.peoplelikeu.com.au. From a telecommunications perspective, organizations here are now looking at how they can use platforms like Hadoop to reduce call drop outs, improve network quality and raise customer NPS.

While Australia might be behind, one thing I have found is that we are quick to catch up. 2014 will be an interesting year as these strategies unfold and Australian companies either innovate or face increasing competitive pressures from major offshore rivals, many of which will come from “cashed up” social networking organizations who decide to expand into new markets rather than traditional competitors. This will be a catalyst to driving big data strategies and a greater adoption of new technologies that can process larger pools of information with Australian companies.