CUSTOMER PROFILE With about 8,000 employees in more than 135 offices worldwide, NetApp creates innovative storage and data management solutions that accelerate business break- throughs and deliver outstanding cost efficiency. It’s a company with a passion for helping other companies around the world go further, faster. THE CHALLENGE It takes good teamwork to help other com- panies accelerate results. Good teamwork depends on informed decision-making, and informed decisions require fresh and accu- rate data. NetApp wanted to provide global access to more current data and improve reporting efficiency by addressing three areas: 1. Sourcing data from transactional systems Frequently extracting data from transactional systems was a problem at NetApp—a problem that can be found at virtually every other large global enterprise. Data warehouses for multibillion-dollar corporations quickly become unwieldy: NetApp’s data warehouse is more than 50TB (raw storage size). Extracting data from the source system would disrupt the source system operation. “Extracting, transforming, and loading [ETL] data from our source system to generate reports can take 8 to 10 hours,” says Mano Manoharan, IT director, Enterprise Data Warehouse at NetApp. “And it’s the nature of our business that volumes of data are very high in the last two weeks of the quar- ter. That’s when dependence and stress on the production system are at their highest. And it’s also when management wants information, and production operation teams don’t want to consume system resources for extracting data. So the challenge was: How can we reduce the extraction footprint on the production transaction systems?” 2. Conflicting ETL (write) and reporting (read) requirements There were added challenges. Extracting data is write intensive, but reporting on that data with an enterprise business intelligence application (eBI) is read intensive. “To have a single system configured optimally for both of these requirements is a challenge by itself,” says Manoj Chauhan, database architect at NetApp. Meanwhile, decision makers want informa- tion in near real time, and they couldn’t get it in NetApp’s prior eBI environment. “When ETL was running on the same database used for reporting and analytics, performance was very poor,” Chauhan points out. “Whenever we ran an ETL at night here in California, it was morning in our Europe-Middle East- Africa [EMEA] region, and people there were not able to run their reports.” KEY HIGHLIGHTS Industry Technology The challenge Deliver near-real-time business intel- ligence from a global data warehouse with high availability worldwide. The solution Use NetApp ® SnapMirror ® , FlexClone ® , and Snapshot ™ technology to quickly replicate data and space-efficiently clone a 50TB data warehouse. Benefits • Near-real-time data with no disrup- tion to transactional production systems (OLTP) • 24x7 availability eliminates up to 15 hours of downtime per patch • $3 million in disk space saved • 90% productivity gain equivalent to 7700 hours per year in performing database refresh NetApp Business Intelligence Architecture Saves $3 Million Success Story
Transcript
CUSTOMER PROFILE
With about 8,000 employees in more than 135 offices worldwide, NetApp creates innovative storage and data management solutions that accelerate business break-throughs and deliver outstanding cost efficiency. It’s a company with a passion for helping other companies around the world go further, faster.
THE CHALLENGE
It takes good teamwork to help other com-panies accelerate results. Good teamwork depends on informed decision-making, and informed decisions require fresh and accu-rate data. NetApp wanted to provide global access to more current data and improve reporting efficiency by addressing three areas:
1. Sourcing data from transactional systemsFrequently extracting data from transactional systems was a problem at NetApp—a problem that can be found at virtually every other large global enterprise. Data warehouses for multibillion-dollar corporations quickly become unwieldy: NetApp’s data warehouse is more than 50TB (raw storage size).
Extracting data from the source system would disrupt the source system operation. “Extracting, transforming, and loading [ETL] data from our source system to generate reports can take 8 to 10 hours,” says Mano
Manoharan, IT director, Enterprise Data Warehouse at NetApp. “And it’s the nature of our business that volumes of data are very high in the last two weeks of the quar-ter. That’s when dependence and stress on the production system are at their highest. And it’s also when management wants information, and production operation teams don’t want to consume system resources for extracting data. So the challenge was: How can we reduce the extraction footprint on the production transaction systems?”
2. Conflicting ETL (write) and reporting (read) requirementsThere were added challenges. Extracting data is write intensive, but reporting on that data with an enterprise business intelligence application (eBI) is read intensive. “To have a single system configured optimally for both of these requirements is a challenge by itself,” says Manoj Chauhan, database architect at NetApp.
Meanwhile, decision makers want informa-tion in near real time, and they couldn’t get it in NetApp’s prior eBI environment. “When ETL was running on the same database used for reporting and analytics, performance was very poor,” Chauhan points out. “Whenever we ran an ETL at night here in California, it was morning in our Europe-Middle East-Africa [EMEA] region, and people there were not able to run their reports.”
KEY HIGHLIGHTS
IndustryTechnology
The challengeDeliver near-real-time business intel-ligence from a global data warehouse with high availability worldwide.
The solutionUse NetApp® SnapMirror®, FlexClone®, and Snapshot™ technology to quickly replicate data and space-efficiently clone a 50TB data warehouse.
Benefits• Near-real-timedatawithnodisrup-
tion to transactional production systems (OLTP)
• 24x7availabilityeliminatesupto15 hours of downtime per patch
“By refreshing the NetApp data warehouse twice a day, North America can start its business day with current information, and so can EMEA. That’s a big win in terms of being able to run the business more effectively.”Dongyan Wang Senior IT Director, Enterprise Business Intelligence, NetApp
3. Storage efficiencyImproving the eBI system was difficult because developers couldn’t get complete, up-to-date copies of the data warehouse in time to run tests on. “The eBI system is so important to NetApp that we’re always running upto20majorandminoreBIdevelopmentprojects,” Manoharan notes. “All the projects require a production copy. Since our ware-houseis50TB,20copieswouldconsumea petabyte of storage. That can generate upwardsof$3millioninadditionalstoragecosts to support our nonproduction environment.”
Many enterprises solve these challenges by creating data marts, or subsets of the data warehouse made with varying business purposes in mind. But because data marts are only partial copies, they have limited uses and are quickly out of date, resulting in inconsistent data delivered to various functional groups.
THE SOLUTION
To solve these challenges, NetApp turned to its own technology.
For data sourcing, the IT team now uses NetApp SnapMirror software to replicate block-level changes in the source system data. To quickly create writable source system clones, the team uses NetApp FlexClone software.
The team optimally configured the system for both ETL and reporting by creating two separate database instances to isolate ETL and reporting requirements. The data load occurs on one database instance and, once the data refresh is complete, the data is cloned to another database instance, which is used for reporting, without interrupting the application users.
NetApp SnapMirror is used to replicate data quickly from the ETL database instance (master) to the reporting volume (slave A or slave B) after the data load is completed. One of the slaves actively serves data to the application while the other receives changes to synchronize with the master data. The team then uses NetApp SnapDrive® and NetApp Snapshot technology to take a point-in-time backup copy of the database. A FlexClone volume is then created from the Snapshot backup. Finally, an Oracle® TAF configuration is used to fail over between slave A and slave B alternately, after the data refresh.
“We call it A/B switching,” explains Manoharan. “The application server sits on top of a slave environment, and we alternate between slave A and B. By doing this, we’re providing consistent performance and uninterrupted data availability to our end users.”
To improve storage efficiency and reduce the storage requirements for the eBI system from 1PB to 100TB, NetApp again used its own technology. NetApp QSM (qtree SnapMirror) is used to quickly replicate the data from production to development and staging environments in a couple of hours. From that single replicated volume, the team usesNetAppFlexClonetocreate10–12clones without splitting the volumes. Each clone is used to create separate databases for development and testing purposes. Because the clones within the replicated volume share common storage, additional storage is required only for changed blocks.
BUSINESS BENEFITS
There are a number of benefits, and they impressed an independent panel of industry judges enough to name the new NetApp eBIarchitectureawinnerofthe2009OracleInnovation Awards in the Business Intelligence/Enterprise Performance Management cat-egory. Said the press release announcing the award: “The new eBI solution transformed how NetApp uses information and achieved many of the industry’s long-standing BI goals.” In addition, InfoWorld named the NetApp solution one of the “Top 100 ITProjectsfor2009.”
Near-real-time data with no disruption“To get recent information, we refresh our data warehouse twice a day,” explains Manoharan. “To do that in our legacy envi-ronment would require extracting, transform-ing,andloadingdatanearly20hoursaday—making reporting impossible or slow. But with NetApp FlexClone and SnapMirror technology, we can clone 5–10TB of trans-actional source systems and a 50TB data warehouse in just 30 minutes. We can do this twice a day, and extract data from the cloned environment. The result is near-real-time data availability with no interruption of services to application users. We’ve reduced ourETLfootprintby95%ontheproductiontransaction source systems.”
Global access to more current dataWhen EMEA comes online in the middle of the North American night, its time lag in data reporting is now at most 8 hours, comparedto48hourspreviously.“Byrefreshing the NetApp data warehouse twice a day, North America can start its business day with current information, and so can EMEA. That’s a big win in terms of being able to run the business more effectively,” points out Dongyan Wang, senior IT director, Enterprise Business Intelligence, NetApp.
New 24x7 availabilityUsers want a highly available system. Previously, the system would go down for 4to6hoursforanupgrade,andupto15 hours for patching. “Now, by switching to a cloned environment, we can make the systemavailabletousers24x7anddoupgrades and patching on an offline envi-ronment before switching it back online,” Manoharan notes. “And we can upgrade or patch during the business day instead of in the middle of the night on the weekend.”
Single version of the truth unifies teamworkPreviously, NetApp teams around the world drew from eight separate data marts to solve different reporting and other business needs.
This caused a number of drawbacks, Chauhan points out. “A data mart cannot provide near-real-time reporting,” he observes. “There is a time lag in data when it’s being made, and, after deployment, it is expensive to maintain. What’s more, having multiple data marts causes different people to end up with different versions of what is only a partial copy of data.” In contrast, he observes, “Business decision makers at NetApp can now draw on a single version of the truth.”
Saving $3 million in disk spaceSpace-efficient data warehouse clones acceleratedevelopment.“Wecanhave20to30 development teams working on enterprise business intelligence projects, and, with NetApp FlexClone technology, each can have their own space-efficient clone of the complete data warehouse,” Manoharan notes. “Instead of a petabyte in our develop-mentenvironment,weconsume120TBtotal,savingNetApp$3millioninadditionalstorage costs.”
Troubleshooting fivefold fasterIn the previous environment, it took about 5 to 10 days for the data warehouse team to troubleshoot priority-one business intel-ligence issues. “Now, with the ability to work with a cloned data warehouse using QSM to refresh the nonproduction environment, we have demonstrated that we can solve any priority-oneissuewithin24hours,whichwe couldn’t do before,” Chauhan observes.
90% productivity gain in database refreshThe team developed UNIX® shell scripts to automate the cloning of the data warehouse. “Now we don’t have to spend time doing
www.netapp.com
NetApp creates innovative storage and data management solutions that deliver outstanding cost efficiency and accelerate performance breakthroughs. Discover our passion for helping companies around the world go further, faster at www.netapp.com.
things manually,” Manoharan points out. “Wejustspend700hoursinayearfordatabase refresh using NetApp cloning technologycomparedto8400hoursittakes using the traditional approach.”
“With NetApp FlexClone and SnapMirror technology, we can clone 5–10TB of transactional source systems and a 50TB data warehouse in just 30 minutes. The result is near-real-time data. We’ve reducedourETLfootprintby95%ontheproduction transaction source systems.”Mano Manoharan IT Director, Enterprise Data Warehouse, NetApp
SOLUTION COMPONENTS
NetApp productsNetAppFAS6080storagesystems
NetApp FlexClone software
NetApp SnapDrive for UNIX software
NetApp SnapMirror software
ProtocolsFCP
NFS
CIFS
EnvironmentApplications: Oracle Business Intelligence Enterprise Edition (10.1.3.4.1),Siebel(7.5.3),SAP® CRM (5.2),OracleERP(11.5.9)
