The problem of 'big data' can be solved providing the right techniques are used to manage and search the 'digital landfill'.

At a roundtable hosted by Sourcefire, EMEA managing director Graham Welch claimed there are interesting techniques for searching data to make it more effective, but the challenge is often about finding the needle in a haystack.

He said: “We want a picture of what is happening from a security perspective and if security is crucial then you can find out what is happening. Attackers are using age-old techniques such as phishing, and if you have the ability to put data into a larger dataset, then you have the ability to get into the data. I see it happening and it will develop over time.”

Roger Ellis, treasurer of the Information Security Systems Association (ISSA), said that among its members, the biggest issue usually is "the cost of ratcheting up systems", so security can be written off due to its cost. He also criticised government for a lack of marketing on security.

David Norris, analyst at Bloor Research, said technologies exist to find answers and the concern is around choosing the correct analytical tool to read data. He also claimed that until there is an event that makes companies take 'big data' seriously, there will not be resources put into this area and companies will have to live with it.

“Big data gives us the opportunity to find a pattern, and that is where technology helps as you can follow a digital trail,” he said.

Welch said: “Not a lot has changed, but this is a new world of techniques. It comes down to four elements: analysing risk; educating users; designing and implementing policies; and applying technologies.

“As technology evolves, big data techniques will be used, and in terms of visibility and taking action on events, I believe it will happen at a national level first.”