Snowflake state of the data warehouse survey report by Snowflake

THE STATE OF THE DATA WAREHOUSEA SURVEY OF DATA PROFESSIONALS

March 2015

Sponsored by


Dimensional Research | March 2015

www.dimensionalresearch.com © 2015 Dimensional Research.All Rights Reserved.

IntroductionAs the volume and types of business data have increased at a phenomenal pace, and the cost to store that data has plummeted, businesses have looked to data analytics to gain new insights into their customers and operations. The buzz about Big Data, including the use of technologies like Hadoop, has dominated the news. But what does that mean to the future of data warehousing? Are companies abandoning their data warehouses in favor of new technologies? And what is the impact of new paradigms created by cloud computing?

The following report, sponsored by Snowflake Computing, is based on a survey of 319 individuals with responsibility for data initiatives. The goal of the survey was to understand experiences, challenges and trends in data warehousing and data analytics.

Key Findings• Datawarehousesstillcriticallyimportant,butchallengesremain

- 99% say their data warehouse is important to their business operations - 70% are increasing their investment in data warehousing - 97% face challenges with their current data warehousing solutions - Scalability and cost are top challenges

• PlentyofinterestinBigData,butsofarinvestmentislimited - 91% have considered a Big Data investment - Only 11% have a pilot in place, and only 5% have fully deployed their Big Data initiative - 91% have concerns about Hadoop - 96% say Hadoop will not replace their existing data warehouse - Only 12% have easy access to Hadoop expertise; in contrast, 93% have easy access to SQL expertise

• Cloud-baseddatawarehousingdesirable,especiallyfordatainthecloud - 93% see value in the potential benefits of cloud data warehousing - Scalability and reduced overhead most valuable - 32% have a cloud-based data warehouse today; 16% use only a cloud-based data warehouse - 49% of those with data already in the cloud have a cloud data warehouse - 79% of those with data in the cloud bring it into the data warehouse, although only 9% bring all data in

Sponsored by




Page 3

Detailed FindingsData warehouses still critically important, but challenges remainWhen asked about the importance of their data warehouse to business operations, almost all of those responsible for data initiatives (99%) indicated that it was important, including almost three-quarters (72%) who characterized it as very important. Only a very few (1%) described their data warehouse as not important.

Even with the significant amount of discussion of alternative Big Data platforms, data warehousing investment continues to grow. A large majority of companies (70%) are increasing their data warehousing investment, while only a very small number of companies (4%) are reducing their investment in data warehousing.

n = Accountants

72% 27% 1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

How important is your data warehouse to business opera3ons?

Very important

Important

Not important

Investment is increasing

70%

Investment is decreasing

4%

Not changing 26%

How is the level of investment in your company’s data warehouse changing?




Page 4

That said, data professionals do see room for improvement in their existing data warehouse. The vast majority of data professionals surveyed (97%) face challenges with their current solutions for data warehousing. Top challenges cited included scalability (43%), cost to maintain (38%), cost to purchase (37%), and the difficult learning curve for business users (37%).

Plenty of interest in Big Data, but so far little investment To understand the impact of Big Data initiatives, we first examined the state of those initiatives. We discovered that while most data stakeholders are interested in Big Data – only 9% have not considered a Big Data initiative – most are still investigating or planning to investigate.

Just 16% have made an investment in Big Data, including 5% who are fully deployed and a further 11% that are in a pilot. Most companies, 70%, are interested in Big Data but haven’t yet made any investment. That does include 29% who are currently investigating their options, but the largest number is those who are intrigued, but haven’t yet done any work to find out more (41%).

3%

19%

30%

32%

33%

37%

37%

38%

43%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

We have no challenges with data warehousing

Not able to handle all of the different types of data we have

Takes too long to deploy new projects

Difficult to find experJse to manage

Users complain it is too slow

Challenging for business users to learn

Very expensive upfront purchase costs

Very expensive to maintain and operate

Concerned about ability to scale to handle projected data and user growth

What challenges do you face with your current data warehousing solu4ons?

5%

11%

29%

41%

5% 9%

0% 5%

10% 15% 20% 25% 30% 35% 40% 45%

Fully deployed Have a pilot in place Currently inves<ga<ng We are interested, but haven’t inves<gated yet

Have inves<gated and decided not to pursue

Not being considered

What investment is your company making in Big Data?




Page 5

Hadoop not a practical solution for most enterprisesBig Data is rarely discussed without talking about Hadoop, the most common foundational technology for these initiatives. It appears that concerns about Hadoop may be an important factor in the slow pace of investment in Big Data. The vast majority of data stakeholders (91%) indicated that they have concerns about using Hadoop in their organization, with the biggest concern by far being lack of expertise (71%).

It is not surprising that lack of expertise is a challenge for any new technology like Hadoop. In contrast, SQL expertise is common. The vast majority of data stakeholders surveyed can easily access SQL expertise (93%), while only a few (12%) have easy access to Hadoop expertise.

9%

4%

18%

23%

24%

71%

0% 10% 20% 30% 40% 50% 60% 70% 80%

I have no concerns

Other

Hadoop is too complex

No compelling business need

Don’t have enough data to bother

Lack of experIse with Hadoop

Do you have any concerns about using Hadoop in your organiza6on?

Yes 93%

No 7%

Do you have easy access to SQL exper3se? Yes 12%

No 88%

Do you have easy access to Hadoop exper2se?




Page 6

This difference in access to Hadoop expertise is even more pronounced for smaller companies, whereas SQL expertise is accessible to almost everyone. Among the companies in our study with fewer than 1,000 employees, only 5% have access to Hadoop skills, while most (92%) have easy access to SQL expertise.

When asked directly about opinions on Hadoop as a possible replacement for the data warehouse, only a very small number (4%) reported that they thought Hadoop could eventually replace their existing data warehouses. Most data stakeholders (64%) believed Hadoop would not impact their data warehouse, and any investment would be fully complementary. About a third (32%) did think that Hadoop could replace some but not all of what was done today using a data warehouse.

95% 93% 92%

22% 13%

5%

0%

20%

40%

60%

80%

100%

More than 5,000 1,000 -‐ 5,000 100-‐1,000

Access to SQL or Hadoop exper1se by company size

Have easy access to SQL exper@se

Have easy access to Hadoop exper@se

No, Hadoop will be complementary to our data

warehouses 64%

Hadoop will replace some of what we do with our data warehouse, but not all

32%

Yes, Hadoop will completely replace our need for data

warehouses 4%

Do you consider Hadoop as a possible replacement for exis6ng data warehouses?




Page 7

Cloud-based data warehouse seen as valuableData stakeholders see potential value in having a data warehouse that is in the cloud. The majority (93%) see benefits in cloud data warehousing with the right capabilities. A cloud-based data warehouse with the ability to scale on demand (66%) and deliver lower administrative overhead (59%) would deliver the most benefit.

Because of this potential value, cloud data warehousing is no longer something that only early adopters are using. Almost a third of companies (32%) are already using cloud-based data warehousing, including 14% who have exclusively adopted cloud and do not support an on-premises data warehouse.

7%

27%

47%

47%

53%

59%

66%

0% 10% 20% 30% 40% 50% 60% 70%

None of these would be valuable

Op=mized for structured and semi-‐structured data

Simplify access for business users

Ability to easily share access to data

Reduced cost

Less administra=ve overhead

Ability to scale on demand

Which of these poten-al benefits of a cloud-‐based data warehouse would be valuable to your organiza-on?

32%

86%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Cloud-‐based data warehouse

On-‐premises data warehouse

Which of the following are currently in use in your organiza4on?




Page 8

What concerns have kept the rest from using a cloud data warehouse? When we asked those data stakeholders who did not have a cloud data warehouse, the top reason was that they had concerns about security (64%).

It is worth noting that some of these who do not have a cloud-based data warehouse today do plan to adopt the technology in the near future. This includes 16% who indicated that they want a cloud-based data warehouse but haven’t gotten around to it yet, as well as multiple participants who used the “other” option to indicate they were actively in process with a cloud-based data warehouse initiative. who wrote in as an “other” option saying that they were actively in process with a cloud-based data warehouse initiative.

Cloud data warehouse a natural fit for cloud dataCloud services have traditionally caused challenges for data and business intelligence professionals because cloud data needed to be brought back into the datacenter to be available for general use. However, this has clearly changed.

Almost two-thirds of participants in this survey (62%) have data in the cloud – either in SaaS applications like Salesforce.com or in public infrastructure clouds like Amazon AWS. These participants were asked how that data fit into their data warehouse initiative.

Most companies with data in the cloud are bringing at least some of their cloud data into a data warehouse (79%). Only a small portion of those (9%) are bringing all cloud data into the data warehouse. Most (55%) have less than half of their cloud data in their data warehouse.

10%

11%

16%

19%

26%

64%

0% 10% 20% 30% 40% 50% 60% 70%

Other

Have never thought about it

We’d like to but haven’t go?en around to it

Not saBsfied with current cloud-‐based data warehouse offerings

Technically complicated to move our data

Concerned about security of data in the cloud

Why doesn’t your company use a cloud-‐based data warehouse?

n = do not have a cloud data warehouse

n = have data in the cloud

21%

33%

22%

15%

9%

0%

5%

10%

15%

20%

25%

30%

35%

No data Less than 25% 25-‐50% More than half All data

Percentage of cloud data brought into a data warehouse




Page 9

Interestingly, there is a clear correlation between companies who have data in the cloud, and those who have a cloud-based data warehouse. When we consider only companies who have data in the cloud, almost half have a cloud data warehouse (49%), while very few companies who don’t use data in SaaS applications or public infrastructure clouds have adopted cloud for data warehousing (4%)

Survey Methodology and Participant DemographicsIn February 2015, technology and analytics stakeholders with responsibility for corporate data initiatives were invited to participate in an online survey on the topic of data analytics and warehousing. Participants were asked a series of questions about their role in data initiatives as well as specific questions on experiences, challenges, and trends with data warehousing, Big Data, Hadoop, and cloud.

A total of 319 individuals completed the survey. All had professional responsibility for data initiatives. Participants represented a wide range of geographies, company sizes, role, and vertical industries. All participants worked at companies that used data warehousing.

8%

11%

27%

46%

51%

0% 10% 20% 30% 40% 50% 60%

BI Business Owner

BI or Analy;cs Technology Owner

IT execu;ve with data ini;a;ves in my porFolio

IT staff responsible for implemen;ng and opera;ng data infrastructure

IT manager responsible for delivering data ini;a;ves

Role in data ini+a+ves

100 – 1,000 42%

1,000 – 5,000 37%

More than 5,000 21%

Company Size

5% 1%

1% 1%

2% 2%

2% 2%

4% 4%

4% 8%

9% 12%

13% 14%

15%

0% 2% 4% 6% 8% 10% 12% 14% 16%

Other Pharmaceu6cal

Media and Adver6sing Transporta6on

Telecommunica6ons Hospitality and Entertainment

Food and Beverage Retail

Energy and U6li6es Services

Non-‐Profit Government

Educa6on Healthcare

Manufacturing Financial Services

Technology

Ver$cal Industry

Yes 62%

No 38%

Have business data in the cloud

4%

49%

0%

20%

40%

60%

No cloud Cloud data

Have cloud data warehouse




Page 10

About Dimensional ResearchDimensional Research® provides practical market research to help technology companies make their customers more successful. Our researchers are experts in the people, processes, and technology of corporate IT and understand how corporate IT organizations operate. We partner with our clients to deliver actionable information that reduces risks, increases customer satisfaction, and grows the business. For more information visit www.dimensionalresearch.com.

About Snowflake ComputingSnowflake Computing, the cloud data warehousing company, was founded in 2012 by a team of experts who hold over 120 patents in the fields of database architecture, data warehouses, query optimization and parallelization. Snowflake was created with the vision to reinvent the data warehouse, bringing together all users, all data and all workloads in a single repository. The company is backed by leading investors including Redpoint Ventures, Sutter Hill Ventures and Wing Ventures. Snowflake is headquartered in Silicon Valley and can be found online at www.snowflake.net.

Date post:	14-Jan-2017
Category:	Technology
Upload:	muzaffer-yontem
View:	333 times
Download:	0 times