Microsoft SQL Server Customer Solution Case Study Utilities ISV Scales to Meet Customer Needs for Storage and Analysis of Big Data Overview Country or Region: Belgium Industry: Utilities—Power utilities Website: www.mecoms.com Customer Profile Ferranti Computer Systems was founded in 1976 in Antwerp, Belgium, to improve the business processes of its customers. The company offers the MECOMS™ solution to energy and utilities companies. Business Situation To help its customers improve their services, Ferranti needed its MECOMS™ solution to quickly store, verify, and process large data volumes of data sent by millions of smart meters. Solution Ferranti and Microsoft used In-Memory OLTP and HDInsight high-performance database technology in their Big Data solution. Benefits Increased the database write speed Enhanced business opportunities “With In-Memory OLTP and HDInsight, we have the best technology on which to base the current and future design of our architecture.” Guido Van de Velde, Director of MECOMS™ Product Organization, Ferranti Ferranti Computer Systems, the developer and distributor of MECOMS™—a business support system for the energy and utility industry based on Microsoft Dynamics AX—needed to help its customers solve an emerging Big Data business problem. Increasing adoption of smart meter technology has created very large volumes of structured data that must be quickly and continuously processed by the energy and utility companies to support business processes. To solve this problem, Ferranti teamed with Microsoft to incorporate the In-Memory OLTP, Reactive Extensions, and HDInsight technologies into its MECOMS™ solution. MECOMS™ can now process a continuous data flow of up to 200 million measurement channels, making the system fully capable of meeting the demands of smart meter technology. The two companies continue to work together to make MECOMS™ capable of processing the larger data volumes they expect in the future.
Transcript
Microsoft SQL ServerCustomer Solution Case Study
Utilities ISV Scales to Meet Customer Needs for Storage and Analysis of Big Data
OverviewCountry or Region: BelgiumIndustry: Utilities—Power utilitiesWebsite: www.mecoms.com
Customer ProfileFerranti Computer Systems was founded in 1976 in Antwerp, Belgium, to improve the business processes of its customers. The company offers the MECOMS™ solution to energy and utilities companies.
Business SituationTo help its customers improve their services, Ferranti needed its MECOMS™ solution to quickly store, verify, and process large data volumes of data sent by millions of smart meters.
SolutionFerranti and Microsoft used In-Memory OLTP and HDInsight high-performance database technology in their Big Data solution.
Benefits Increased the database write speed Enhanced business opportunities
“With In-Memory OLTP and HDInsight, we have the best technology on which to base the current and future design of our architecture.”
Guido Van de Velde, Director of MECOMS™ Product Organization, Ferranti
Ferranti Computer Systems, the developer and distributor of MECOMS™—a business support system for the energy and utility industry based on Microsoft Dynamics AX—needed to help its customers solve an emerging Big Data business problem. Increasing adoption of smart meter technology has created very large volumes of structured data that must be quickly and continuously processed by the energy and utility companies to support business processes. To solve this problem, Ferranti teamed with Microsoft to incorporate the In-Memory OLTP, Reactive Extensions, and HDInsight technologies into its MECOMS™ solution. MECOMS™ can now process a continuous data flow of up to 200 million measurement channels, making the system fully capable of meeting the demands of smart meter technology. The two companies continue to work together to make MECOMS™ capable of processing the larger data volumes they expect in the future.
SituationFerranti Computer Systems is a global independent software vendor (ISV) and a Microsoft partner. Ferranti has developed its MECOMS™ solution for the energy and utility industry at its headquarters in Antwerp, Belgium, and has implemented the solution for more than 50 customers in Europe, America, and Asia. The software helps utility companies manage their “meter-to-cash” and related processes. The company has worked closely with Microsoft since 2007 to improve the performance and flexibility of MECOMS™, which is based on Microsoft Dynamics AX and SQL Server 2012. SQL Server performs data analysis and storage and Microsoft Dynamics AX performs business processes such as ERP and CRM. This partnership has been a “win-win” for Ferranti and Microsoft, resulting in substantial improvements in performance and functionality to the three software products.
In recent years, the energy and utility industry has incorporated smart-meter technology at an increasing rate. Smart meters produce a large amount of structured data that not only include the power usage information produced by mechanical meters, but also detailed power-quality data such as the number of outages and surges over a period of time. While a utility company worker may check a mechanical meter once every month, a smart meter can provide continuous power consumption data for one customer every 15 minutes. The measurements are more frequent and there are more kinds of measurements. All these measurements generate very large volumes of data that are being sent to the utility companies. Guido Van de Velde, Director of the
MECOMS™ product organization at Ferranti, explains, “Instead of 5 million measurements per year, it’s now 500 million measurements per day. Our customers are experiencing an explosion of data, and this is all driven by the implementation of smart energy solutions such as smart meters.”
MECOMS™ processes and analyzes the data in these very large volumes to determine whether it is valid, and then it stores the validated data in a SQL Server database. The speed of the analysis and storage must be at least equal to the rate at which new data arrives to avoid falling behind. However, as the volume of data has exponentially increased with the number of smart meters, the amount of time available to complete the validation and storage has decreased at the same rate.
Ferranti found that its implementation of SQL Server could not keep up with these new demands, so the company needed to find a way to make MECOMS™ much faster at processing incoming data. Ian Bruyninckx, Solution Architect for MECOMS™ at Ferranti, says, “There was no doubt that we needed to make MECOMS™ Big-Data-ready. The question was how.”
SolutionBecause MECOMS™ is based on Microsoft technology, Ferranti enlisted the help of Microsoft engineers to meet its goal. As a participant in the Microsoft Technology Adoption Program, Ferranti could work closely with Microsoft engineers and gain early access to new technologies.
In late 2011, Ferranti and Microsoft began work on the new implementation of
25
“There was no doubt we needed to make MECOMS™ Big-Data-ready. The question was how.”
Ian Bruyninckx,Solution Architect for MECOMS™,
Ferranti
MECOMS™, concentrating on optimizing the database lookups and data analysis performed by Microsoft Dynamics AX. During the summer of 2012, the MECOMS™ team decided to evaluate two Microsoft technologies that would prove key to solving Ferranti’s Big Data problem.The first was In-Memory OLTP, a memory-optimized, online transaction processing (OLTP) solution developed by Microsoft in SQL Server 2014. The In-Memory OLTP engine speeds up SQL Server processing by optimizing the T-SQL procedure code and with new table structures called memory-optimized tables. Ferranti implemented memory-optimized tables in the MECOMS system as a “shock absorber,” where bursts of incoming smart-meter data are quickly stored, verified, and aggregated before it is moved to disk storage.
The second technology was the Reactive Extensions (Rx) framework, which provides for event-based data stream processing. The team recognized that data verification and aggregation could be made much faster if it were performed in an event-driven manner. They implemented Rx to verify and process the incoming raw data, and then to send the aggregated data to SQL Server for quick storage in memory-optimized tables. SQL Server also analyzes the aggregated data, and sends the results of the analysis to Microsoft Dynamics AX for demand-side business processes such as scheduling service calls, terminating service, and invoicing.
During testing, the team found that In-Memory OLTP and Rx made it possible for MECOMS™ to analyze and store the data as quickly as the smart meters sent the data. However, processing the unusually large
volume of smart meter data—on the order of 500 million rows per day—impeded the performance of the SQL Server relational store that Microsoft Dynamics AX is built on.
To solve this problem, Ferranti and Microsoft are designing a solution that uses Windows Azure HDInsight Service and nonrelational technologies to perform these searches and provide the information to the business processes in MECOMS™ and Microsoft Dynamics AX. HDInsight is the Microsoft Big Data offering of an Apache-compatible Hadoop distribution, and it is designed to process very large quantities of structured and unstructured data in a distributed manner. Searches of the memory-optimized tables are distributed between groups of computers, called clusters, which are managed by HDInsight. Other benefits that HDInsight offers are full compatibility with Microsoft business intelligence technology such as SQL Server 2012 Analysis Services and SQL Server 2012 Reporting Services, as well as the ability to scale up processing power by adding more computers to the cluster.
Although the evaluation of HDInsight is still in progress, Ferranti and Microsoft have been pleased with its performance. They envision expanding their use of the technology in MECOMS™ beyond performing searches of the In-Memory OLTP database. Says Van de Velde, “We are mainly using SQL Server for data aggregation now, but in the near future we may be looking at using HDInsight to take over that aggregation in a distributed way.”
35
“We are mainly using SQL Server for data aggregation now, but in the near future we may be looking at using HDInsight to take over that aggregation in a distributed way.”
Guido Van de Velde,Director of MECOMS™ Product
Organization, Ferranti
BenefitsFerranti has seen substantial improvements in MECOMS™ performance, including increased database write speed, reduced data analysis time, and enhanced business opportunities for the company and its customers.
Increased Sustained Database Write Speed to 200 Million Rows in 15 MinutesTests performed by Ferranti and Microsoft demonstrate that approximately 200 million rows can be written to an In-Memory OLTP test database in 15 minutes. Although SQL Server without In-Memory OLTP is capable of this rate during short durations, it does not maintain the rate as the aggregate data volume increases. With In-Memory OLTP, the rate can be sustained regardless of the amount of data consumed.
“We proved that we can handle the data flood from over 200 million smart metering channels, which represents more than the number of smart meters currently installed worldwide. Our tests were performed on standard, commercially available hardware, which makes it very promising,” says Van de Velde.
New Business Opportunities The solutions that Ferranti and Microsoft are developing for the challenges of accumulating and querying very large volumes of data are also developing new business opportunities for Ferranti and the utilities industry. Says Van de Velde, “As we discover ways to access and analyze more of the data generated by the smart meters, we find that we can do a lot more with it. We have seen new ways that the data can be used by us and our customers.”
One of the uses that Ferranti has identified is adaptive rate adjustment of services. Utility companies can bill customers in different geographical areas at different rates for different times of the day, to encourage or discourage consumption patterns. Another is the manufacture of home appliances designed to automatically adjust their operation according to the smart meter data it receives. An example of this would be a dishwasher that runs only during the times of the day when energy is cheapest. “One of the attractive things about the new types of data provided by the smart meters is that the more you look at it, the more business possibilities you see in it. It sparks people’s imagination,” says Bruyninckx.
In-Memory OLTP and HDInsight have been the breakthrough technologies that have made it possible for Ferranti to solve its Big Data challenges and better serve the needs of its customers. Because of Ferranti’s success with these technologies, the company plans to expand their use within MECOMS™ to create new MECOMS™ functionality for its customers. Van de Velde concludes, “With In-Memory OLTP and HDInsight, we have the best technology on which to base the current and future design of our architecture.”
45
Microsoft Server Product PortfolioFor more information about the Microsoft server product portfolio, go to:www.microsoft.com/servers
55
For More InformationFor more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:www.microsoft.com
For more information about Ferranti Computer Systems products and services, call (32) (3) 540 49 11 or visit the website at: www.mecoms.com
This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.
Document published June 2013
Software and Services Microsoft Server Product Portfolio− Microsoft Dynamics AX− Microsoft SQL Server 2012− Microsoft SQL Server 2014
