5 Keys To Big Data Success5 Keys To Big Data Success

Big data success isn't just a matter of choosing the right business problems, data, technologies, and techniques. According to a study by Nemertes Research, big data success also depends on how you fund your projects, manage your data, and engage with internal constituents, and how your company approaches tech spending.

Nemertes' study, "2013-2014 Enterprise Technology Benchmark -- Big Data, Analytics and Virtualization," reveals strong correlations between practices around big data and success. Based on interviews with more than 200 IT leaders, the research measured success along the dimensions of decreased costs, increased revenues or the creation of new streams of revenue, as reported by the survey respondents.

Here's a closer look at five factors that the most successful practitioners had in common:

Separate budgets. There's a strong correlation between having a dedicated big data budget and having success with big data, according to Nemertes. In its earliest surveys (in 2011), the research firm found that many companies were trying to fund big data initiatives out of their BI, data warehousing or marketing budgets. The latest survey finds that 60% of firms have separate big data budgets and another 5% are headed in that direction. The remaining 35% do not have separate big data budgets.

"Those with a separate budget are about 20% more successful those without separate budgets," said John Burke, principle research analyst at Nemertes, at last month's Big Data Conference in Chicago. "That makes sense because you're no longer competing with the app-development folks or the BI and data warehousing folks for how the money gets spent."

[ Here's the flip side: Data science evils to avoid. Read 5 Data Science Sins To Beware. ]

Complete data lifecycle management. The full lifecycle of data management addresses everything from acquisition, classification, and management to analysis, visualization, and end-of-life disposal, with security and compliance safeguards at every step along the way. The most successful big data practitioners address the complete lifecycle of big data. Classifying the data streams up front positions you to make the right decisions and to apply consistent policies and processes throughout the lifecycle.

Analysis and data visualization gets most of the attention today while end-of-life planning is the step that big data practitioners most often ignore. But that can lead to bad decisions handled in a haphazard way. There has to be a standardized approach for determining what to keep live, what to archive, how to archive (making provisions for discoverability if it might be subject to legal or regulatory scrutiny), and what to delete, Nemertes advises. And deletion requires a process all its own.

"You must have a process and you have to follow it, especially for information subject to compliance or [legal] discovery," Burke said. "You have to document that you made the deletion decision in a standardized way, you have to document that you actually deleted the information, and you have to document that you documented all of that."

IT engagement. Involving many people across IT makes good sense because big data initiatives are likely to touch on all parts of the IT infrastructure and organization -- data processing, database management, reporting, storage, networks, app development and so on. "The more successful companies are involving more of the IT teams in decision making around big data," Burke said.

Line-of-business engagement. Initial big data projects should have a tight focus and scope to help ensure success, but long-range big data planning and strategy development should cast a wide net across multiple business areas. The more successful respondents in Nemertes' research engaged business leaders in big data decision making and guidance. One approach is to develop an outward-facing steering committee that brings line-of-business leaders in on setting the big-picture big data strategy and priorities.

"The business lines are the origin of the value that you drive, so the more often and more broadly you involve these people, the more likely you are to be successful," Burke said.

Shared spending model. The more successful big data practitioners report that their firms have higher levels of technology spending outside of IT's control -- about 28% on average, according to the research. Only about 4% of tech spending was handled outside of IT among firms self-reporting as less-successful with big data. The non-IT spending is often tied to software-as-a-service applications in sales and marketing or infrastructure-as-a-service offerings that might be used for analytic exploration.

"We don't have exact stats here, but we anecdotally picked up that it's standard practice at these organizations for business departments such as marketing operations to launch skunk works projects, often using external resources," Burke said, noting that once projects prove value they're often brought in house with IT operationalizing the capability. "The fact that they had budget enough to do that and freedom enough to do that contributed to their success with big data."

Correlations are not necessarily definitive, scientific conclusions, but Johna Till Johnson, the president and founder of Nemertes, said the research is solid enough to merit the following advice: Have a separate budget for big data projects, manage the entire lifecycle of data, engage IT and business groups at every turn, and be open to tech spending outside of IT's control.

"If you launch a big data initiative you have to fund it, and having a line item means it will be taken seriously," Johnson said. "It's also a good idea to share the investment across IT and the lines of business because if you have more funding from outside IT, you're going to be more successful."

There's no single migration path to the next generation of enterprise communications and collaboration systems and services, and Enterprise Connect delivers what you need to evaluate all the options. Register today and learn about the full range of platforms, services and applications that comprise modern communications and collaboration systems. Register with code MPIWK and save $200 on the entire event and Tuesday-Thursday conference passes or for a Free Expo pass. It happens in Orlando, Fla., March 17-19, 2014.