A shortage of data analysts is the main challenge of Big Data

News by Dan Raywood

There is a major shortage of skills in data analysis that is not helping businesses meet the Big Data challenge.

There is a major shortage of skills in data analysis that is not helping businesses meet the Big Data challenge.

Speaking to SC Magazine, DJ Skillman, director of technical services EMEA at Splunk, said that no one is able to "chew through a terabyte of data in one day". He said: “Data analysis is a skill set that the world is short on, as people spend time buying expertise and experience, but we need smart people to ask smart questions.

“This is a challenge people will run into, managing more data. A lot of developers know how to develop, but people are coming up with an algorithm and asking the questions correctly, but the process is wrong. One of the challenges we come up against with Big Data is knowing how to process.”

James Murray, vice president and general manager EMEA at Splunk, said that businesses are correlating all sorts of data and the new talent is doing correlation for the next security threat and that is where data scientists are key. “Data exists but you need someone to manage it,” he said.

They claimed that as well as meeting the challenge of the skills shortage, there are also technologies that can aid the indexing and correlation to enable data analysts to focus on the issues within the Big Data, rather than trying to process it all manually, in order that they can get the most out of whatever technology they're using.

Terry Neal, CEO of InfoSec Skills, who recently bemoaned a lack of take-up of free training courses offered as part of the Cyber Security Challenge, said: “From a security standpoint, Big Data analytics is key to be able to uncover hidden patterns and unknown correlations in huge volumes of unstructured data, i.e. web server logs, across thousands of nodes handling thousands of terabytes of data.

“We need a new breed of security analyst - a data scientist, who can integrate emerging frameworks to data mining and predictive analytics disciplines to maintain the battle front against cyber crime. The problem is that organisations may not have the internal analytics skills and the cost of hiring experienced analytics professionals is high.”

Ionut Ionescu, European advisory board member at (ISC)2, told SC Magazine that people need analysts whether the data is big or small.

He said: “You need to have an analyst's mindset on what to look at and understand the life of the data. We could do with more skills in industry and organisations such as (ISC)2 can help, but you cannot create a professional out of nothing, as you need to know what to discard, as with 2.3TB of data created each day and zillions of words said, it is the skill of the analyst to know what to discard.

“It is about the doing analysis but on a bigger field. This is not a new problem, there are skills and applications to get into information security but you need training, as you cannot analyse data without context.”


Find this article useful?

Get more great articles like this in your inbox every lunchtime

Video and interviews