The growing world of Big Data
73% of executives have invested or plan to invest in Big Data projects in the next two years.
Data scientists are in huge demand, not least because of Big Data. Big Data refers to collecting, storing and analyzing large data sets to optimize business processes and gain insights into customer behavior.
Data analytics and business intelligence from big data is being used in a variety of industries, most notably in customer relationship management, retail, finance, marketing and security. Companies that do business over the internet are furthest along in Big Data.
According to a survey of 1600 executives, the most common of all big data projects were Customer Analytics (48%), Experience Analytics (45%), and Risk Analysis (37%). In 2014 a Gartner survey found that 73% of executives have invested or plan to invest in Big Data projects in the following 24 months.
The value of Big Data
As a result of Big Data Projects 80% of companies improved their decision-making process. The average return on investment was 46%.
Those who already deployed are reaping many organizational and market gains. Logistics and finance functions are expecting the greatest ROI (even though marketing and sales are currently the biggest investors in Big Data). According to a TCS study, the median per-company spending was $10 million (or about 0.14% of revenue) in 2012. As a result 80% have improved their decision-making process. The average return on investment was 46% (or about $900,000 return after subtracting costs), with Asia-Pacific and Latin American companies reporting greater returns than US and Europe.
Yet, most companies are still in the phase of strategy selection, as only 14% of big data projects were actually deployed in 2014 (up 6% from 2013). Looking forward into 2015 and beyond, this means that companies will start wrapping up their planning stage soon, and more big data projects will come to fruition.
That’s going to make the search for the perfect data scientist even tougher.
Who are data scientists?
By 2020, there will be more than a 100,000 data analyst shortage.
Data scientists can design and analyze data initiatives, communicate the results, and suggest what actions should be taken as a result. Data scientists are expected to have a variety of skills, ranging from technical analytics to business acumen and communication. They have been described as “part artist, part analyst”, “Renaissance individuals”, and even “unicorns”.
Unsurprisingly, there are few people out there that possess all the required skills, and even fewer education institution teaching these skills. While there are a number of programs that have been recently launched to remedy the lack of qualified individuals, the numbers of qualified data scientists is projected to be dangerously low compared to the demand.
For example, Gartner predicts that there will be more than a 100,000 analyst shortage through 2020, while McKinsey predicts a shortage between 140,000-190,000 analytics experts, and an additional shortage of 1.5 million managers capable of making decisions based on data.
Shortage or no shortage?
Many executives are skeptical whether one person can have all the skills required to be a data scientist.
Based on predictions, the industry has accepted (and is prepping for) a huge labor shortage. Yet, companies are increasingly launching big data projects, despite the predicted labor and skill shortage. While the rate of deployment remains low – only 14% in 2014 – it is growing (in 2013 only 8% of new big data projects were deployed). Moreover, the recent survey on executive big decisions suggests that the biggest reason for not launching new projects is not because of talent shortage, but because executives believe it is hard to generate data insights from their products and services. Yet, resource scarcity still has a high influence on big decisions.
This means that companies, facing an apparent lack of qualified workforce in the marketplace, are
(1) taking the time to develop their strategies before launching projects, and
(2) are trying to work around the problem of resource scarcity.
Instead of searching for the perfect unicorn, ehh data scientist, executives choose to either:
- 1. Train workers internallyTraining workers internally seems to be a much more viable option to hiring someone new who doesn’t know anything about the company. It’s easier to teach someone how to use Hadoop, rather than how to run the business. It is precisely because of the profound knowledge of the business space required that executives choose to find and train workers from within the company.
For example, GE is currently looking for 400+ data scientists to collect and analyze the data collected from its new sensor-equipped hardware. Thus far, they have recruited about half of that from within the company. They weren’t all data scientists, but GE created a special training program for their employees to teach them how to analyze the data. The result is a team who knows the company, knows its goals, and who is developing the skills to gain the necessary insight.
- 2. Create big data teams Sometimes hiring internally isn’t an option. Still, many executives are skeptical whether one person can have all the skills required to be a data scientist. Moreover, finding and keeping that person (or two or three) and taking the full advantage of their talent might not be a possibility for the organization anyway. Not to mention, data scientists are expected to have a very deep understanding of the business as well as analytics. So if companies cannot find ONE person that has all the right characteristics, then why not create a team of people who would complement each other’s skills and achieve the same results as one data scientist would?
LinkedIn, for example, launched their first projects with a team of 2 engineers and 5 analysts, who were put at the very center of product development (and they went on to add new features to LinkedIn profiles such as Who’s Viewed My Profile, Career Explorer and other tools that now seem to be engrained in the interface). As the team expanded, it wasn’t all mathematicians and analysts. It included designers, web developers, product marketing, and operations.
Instead of searching for the scarce data scientists with experience and knowledge of your industry, building analytics teams, whether internally or with external help, is the best way to get going with your big data project.
Images via Pixabay.
