IBM TM1 ServerFaster than the speed of light and more flexible than a Yogi

Hubert HeijkersSTSM, Chief Architect TM1 Server and OData Evangelisthubert.heijkers@nl.ibm.com

IBM Cloud University 2017 | October

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.

The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming inthe user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

TM1 ServerPlanning Analytics, powered by TM1 Server

Recent TM1 Server Features and Improvements

§ Hierarchies!

§ Multi-Threaded Feeders and Cube Load

§ Transitory View and Set creation in TI Processes

§ Locking, Contention, Performance and other Improvements

§ Encryption for 'Data at Rest'

§ Sandbox Compare

§ TI Debugger

6

Hierarchies

§ Introducing the concept of having multiple distinct hierarchies as part of one dimension– All hierarchies sharing the overall set of leaf level elements– Whereas each hierarchy has its own unique set of consolidated elements

§ Alternate hierarchies typically based off attribute values associated to the leaf elements – describing an aggregation structure for the dimension’s leaf elements

§ Use of alternate hierarchies diminishes the need for extra dimensions resulting in:– Smaller cubes – less memory consumption– Better performance!

§ Alternate hierarchies can, in an analysis, be used as just any other dimension

§ Hierarchies can be added and removed from dimensions ‘on the fly’

7

Hierarchies – The important things

§ Do I have to use Hierarchies?– You do not, they are opt-in and disabled by default, but really why wouldn’t you?

§ What clients can use and or see Hierarchies?– Our more modern clients. Planning Analytics Workspace, Planning Analytics for Excel

§ What will old clients see?– They will only see the same named hierarchy, hence it will look like it always did

§ What about Cognos Analytics?– Cognos Analytics, as of version 11.0.6, supports our REST API and supports hierarchies

§ If I add a new hierarchy will my reports (Perspectives/BI) break?– No, we’ve been really conscious of not breaking backwards compatibility. The old

reports will reference the same named hierarchy and continue to work as expected

8

Hierarchies

9

Query Performance

RAM Savings

Flexibility Standards Conformance

Primary Benefits

Primary Use Cases

Single Dimension Multiple AxesIntroduce

Hierarchies to Queries as Needed Dimension Versioning

MDX

Unique Levels per Hierarchy

Cubes with Less Dimensions

Hierarchies as ‘Virtual Dimensions’

Adopting Hierarchies

§ Upon Upgrade, a ‘Same Named Hierarchy’ is created

§ All Dimension Members are transitioned to that Hierarchy

§ Everything Operates the Same

§ Creation of the ‘next’ Hierarchy triggers new Hierarchy Behavior

= Dimension = Hierarchy

All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs

EquipmentFielding GlovesSticksNetsShoulder Pads

All SportsSpring

BaseballBaseballsFielding Gloves

LacrosseLacrosse ballsSticks

FallFootball

FootballsShoulder Pads

SoccerSoccer BallsNets

All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs

EquipmentFielding GlovesSticksNetsShoulder Pads

All SportsSpring

BaseballBaseballsFielding Gloves

LacrosseLacrosse ballsSticks

FallFootball

FootballsShoulder Pads

SoccerSoccer BallsNets

Adopting HierarchiesAll Products

BallsBaseballsLacrosse ballsSoccer ballsFootballs

EquipmentFielding GlovesSticksNetsShoulder Pads

All SportsSpring

BaseballBaseballsFielding Gloves

LacrosseLacrosse ballsSticks

FallFootball

FootballsShoulder Pads

SoccerSoccer BallsNets

All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs

EquipmentFielding GlovesSticksNetsShoulder Pads

All SportsSpring

BaseballBaseballsFielding Gloves

LacrosseLacrosse ballsSticks

FallFootball

FootballsShoulder Pads

SoccerSoccer BallsNets

All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs

EquipmentFielding GlovesSticksNetsShoulder Pads

All ProductsSpring

BaseballBaseballsFielding Gloves

LacrosseLacrosse ballsSticks

FallFootball

FootballsShoulder Pads

SoccerSoccer BallsNets

Levels

0 – All1 – Product Type2 - Product

Levels

0 – All1 – Season2 – Sport3 - Product

Adopting Hierarchies

All LeavesBaseballsLacrosse ballsSoccer ballsFootballsFielding GlovesSticksNetsShoulder Pads

All ProductsBins

BaseballsLacrosse balls

BoxesSoccer BallsFootballsNetsShoulder Pads

LooseFielding GlovesSticks

Levels

0 – All1 – Product

Levels

0 – All1 – Package Type2 – Product

All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs

EquipmentFielding GlovesSticksNetsShoulder Pads

All ProductsSpring

BaseballBaseballsFielding Gloves

LacrosseLacrosse ballsSticks

FallFootball

FootballsShoulder Pads

SoccerSoccer BallsNets

Levels

0 – All1 – Product Type2 - Product

Levels

0 – All1 – Season2 – Sport3 - Product

Leaves Category Seasonal Packaging

Adopting Hierarchies with Planning Analytics

Intersect multiple hierarchies from the SAME dimension in a single ‘view’

13

Product Category: Balls

Time: Quarters

Seasonal: Spring Packaging: Bins

Q1 Q2 Q3 Q4

BaseballsLacrosse Balls

1

2 3Dim X Dim Y

More Precise Planning

Hierarchies – Business Logic

§ You can add Hierarchies to any Dimension– and Rules for those Cubes will continue to work

§ Unique Consolidations are fine. Same named consolidations must be qualified

§ All member references can be qualified to a Hierarchy in Rules

§ DB Statements can be references with more then one Hierarchy for a Dimension

§ New TIs to create/maintain Hierarchies

§ CellPut / CellGet can write to any Hierarchy combo Cell

14

Alternate Hierarchies

Two hierarchies ofthe same dimension

Multiple Alternate Hierarchies in the Same View

16

4 Hierarchies from the sameDimension

Multiple Alternate Hierarchies in the Same View

MTx – Multi-Threaded Feeders and Cube Load

§ Applies the MTQ framework to cube loads and feeder creation

§ Multi-Threaded Feeders are triggered– at startup,– by changing feeder definitions of cube rules, – when executing the CubeProcessFeeders() TI function

§ Single Cube, many cores!

§ Configured using new parameters in the server configuration– MTCL only kicking in over a, defined, minimum cub file size (MTCubeLoad.MinFileSize)

§ Be aware: these might (read: will) take up to more memory during processing

Caution: Issues have been identified with conditional feeders and MTCL, check support!17

Transitory objects TI Processes

§ Allows for Transitory Views and Sets to be created in TI Processes

§ No registration typically implies less lock contention

§ No need to clean up, they get deleted at the end of the process/chore

§ Simple to implement– SubsetCreate( ‘DimName’, ‘SubName’, 1 );– SubsetCreateByMDX( ‘SubName’, ‘MDX’, 1 );– ViewCreate( ‘CubeName’, ‘ViewName’, 1 );

Be aware: They exist for the duration of the ‘transaction scope’ but their use is scoped to the process they are created in so in some cases you might want to clean them up anyway.

18

Locking and Contention Improvements

§ PullInvalidationSubsets– Remove contentious RO dimension lock from creation/deletion– Also eliminates private subset login contention– Open and work with views when running dim updates

§ VersionedListControlDimensions– Remove contention on }Cubes, }Dimensions, }Groups, }Clients, etc…– Allows creation of new objects without IX locking the dimension

§ LoadPrivateSubsetsOnStartup– Avoid contention between dimension edits and client logins– Usually not a significant memory consumer

19

Locking and Contention Improvements

§ ReduceCubeLockingOnDimensionUpdate– Avoid cube IX locks when dimension edits don’t affect existing rule

§ Adding a new element that’s not referenced by any rules– Allows simultaneous edits to multiple dimensions sharing a cube– Allows writing to the cube during dimension edits

§ CreateNewCAMClients– Don’t like the automatic creation of clients at first login?– Have a process in place that assures that all CAM users with access rights are

represented as clients in your TM1 Server’s security?– You can now manage when locks associated to the creation of these clients occur

20

MTQ Improved

§ Multi-Threaded Queries (MTQ)– Dramatic performance improvement– Using, when available, many threads instead of one

§ MTQ View Datasource in TI (view extract)– MTQQuery=T– DisableMTQViewConstruct() in prolog disables for that process

21

Rule Profiling

§ Are rule slowing you down?

§ What rule line is slowest?

§ How the heck do I find that out?

22

Turbo Integrator related Improvements

§ Tm1runti.exe chore support-chore ‘ChoreName’

§ CubeDimensionCountGet( )No more looping using TabDim

§ SubsetElementGetIndex( ‘Dim’, ‘Subset’, ‘ElementName’, [StartIndex] )

§ LogOutput( Severity, Message )– Write directly to message log– Control output with tm1s-log.properties

23

Retries on File Operations

§ Introduced in 10.2.2.FP7

§ Help improve stability in cases where the file system occasionally is ‘unavailable’

§ Configured through these configuration settings:– FileRetry.FileSpec

§ Specifies the directory path(s) of the affected log files.§ String value, default value is <empty>

– FileRetry.Count§ Specifies the number of retry attempts.§ Integer value, default value: 5

– FileRetry.Delay§ Specifies the delay (in milliseconds) between retry attempts.§ Integer value, default value: 2000

24

Cancelled Thread Logger

§ Thread cancellation logger– TM1.Context.Cancel = DEBUG

Traditional:3512 [3] DEBUG 2015-10-23 15:08:24.175 TM1.Context.Cancel Cancelling function "ViewArrayConstruct" with parameters:3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel Argument #0: Object - Major(0000000020000006) : Minor(0000000000000000) : Type(0000000000000009) - Name "NULL"3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel Beginning View Array Construction for view: NULL (abstraction 0x0000000033C27E58, index 0x20000006)3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel cube: [SalesCube]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel user: [Admin]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel There are 3 title elements:3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel [actvsbud].[Actual]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel [region].[Argentina]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel [model].[S Series 1.8 L Sedan]3512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel There is 1 row subset:3512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel [account1].[NULL], element count: 53512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel There is 1 column subset:3512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel [month].[NULL], element count: 12

REST:10896 [101] DEBUG 2015-10-23 17:09:25.966 TM1.Context.Cancel Cancelling REST request: Method "POST" URI "/api/v1/Processes('foo')/tm1.Execute"

Lock Exception Logger

§ Lock exception logger– TM1.Lock.Exception = DEBUG

Encryption for 'Data at Rest'

§ What’s more to say…

§ Oh, you won’t be able to do fiddle around with your files on disk any longer

§ Once encrypted always encrypted (otherwise that would defeat the purpose)

§ (Data)Administrator can ask the server to ‘save’ an unencrypted ‘snap-shot’ somewhere

§ Look for an EncryptData option in your tm1s.cfg

27

Sandbox Compare

§ Implicit ‘virtual’ dimension

§ One element per available sandbox

§ Side by side comparison of sandboxes

§ Calculations across sandboxes

28

TI Debugger

§ A TI Debugger? The TI Debugger? What/Where?

§ Well, we added support for TI Debugging in the Server’s REST API, allowing anybody and everybody to build their own TI Debugger for their favorite development IDE

§ And, just for fun, we’ve provided a stand-alone one, downloadable from DeveloperWorks– https://developer.ibm.com/recipes/tutorials/ibm-tm1-turbointegrator-debugger/

§ Configure your TM1 Server (tm1s.cfg) by:– Making sure the HTTP port for the REST API is enabled i.e.:HTTPPortNumber=8000

– Enable TI Debugging:EnableTIDebugging=True

§ Requires TM1 Server v11 or the, more recent, v10.2.2.FP7

29

TI Debugger

30

Prolog,Metadata,Data,andEpilogtabsidenticaltothatofArchitect

ßAlleditableTIprocesses

Listofbreakpoints

VariablesusedinexecutionObjectlocksontheTM1serverStepOver,StepIn,StepOutcapability

What’s next in TM1 Server?TM1 Server features to be expected in future versions

Attribute Based, alternate, Hierarchies

§ Now that we have support for having, potentially many, alternate hierarchies

§ Hierarchy structures, more often then not, being driven by attribute values

§ Allow the system to auto maintain a hierarchy based on an, attribute based, specification

§ Resulting in hierarchy updates whenever a source attribute value is being update

§ Attribute Based Hierarchy specification would:– Refer to the attributes on which the hierarchy is build– Specify how to order members– How to create unique names for members– How to deal with null values– Maybe introducing the notion of skipping levels and raggedness

32

Sandbox sharing

§ Wouldn’t sandbox compare be even more powerful if one could share each others work?

§ Building on the new sandbox compare feature

§ Extend the list of sandboxes available to the user with those shared with him/her

§ While still allowing side by side comparison of any and all sandboxes

§ And allowing calculations across sandboxes

33

Feeder-less Rules, yes, you heard it right;-!

§ Brand new calc engine that spans both rules and MDX member expressions

§ Eliminates the need for feeders altogether

§ Feeder-less rule computation over sparse data– driven solely by the rule definition– feeder markers are not required

§ Faster– batch computation of rule results (as in stargate vs. per cell calculations)– incremental result set cache

§ What this is NOT is an engine that generates feeders, we are NOT writing feeders under the covers, feeders go away completely and forever.

34

Could live without feeders be any better?

§ No more memory used to store feeders– If the feeder doesn’t exist, nor does the

memory space it used to occupy

§ Real time modelling– Model changes don’t require lengthy

feeder updates

§ Simpler Modeling Experience– No need for feeder section in rules

§ Less model errors– Feeder definitions are error prone

§ Faster cell updates– No need to trigger feeder value

updates

§ Faster server load times– No need to load/generate feeders

§ And that’s not all!

35

Yes, it can get even better then that!

§ Better support for Virtual Cubes– Without feeders no ‘underfed’ tricks need to be applied to make ‘virtual’ cubes

perform, ignoring the restrictions those tricks came with

§ Slices/Partitions can come from other servers– Now that we minimize the ‘queries’ for data from other cubes we could envision,

especially in virtual cubes, data to come from different server instances that might reside on different machines (self-serve horizontal scalability?)

§ ‘Time’ dependent hierarchies– Not depending on feeders makes it much more feasible to accomplish this

36

Feeder-less Rules, where are we at?

§ Seriously, you wonder where we are at?– 99% of TM1 10.2.2 rule functionality covered without need of feeders

§ What we need to do?– Support for alternate hierarchies!– Additional functional coverage primarily focussing on supporting MDX natively– Expand test cases– Identify required improvements– Make some hard decisions as to how/what/where/when

37

What’s brewing in the lab?Interested to know what’s cooking wrt TM1 Server in the labs?

What’s brewing in the lab?

Well, that’s beyond the scope of this presentation but…

Don’t hesitate to reach out to me if you want to know more about a next generation, deployable as-a-service, highly available TM1 server, and want to know more about:

§ Source Control Integration

§ Containerization

§ Clustering, High Availability and Fail-Over

Or ideas around

§ Model/modeling improvement

§ A new scripting language to ‘replace’ TI perhaps

39

Questions

?Think of any more questions later?Don’t hesitate to contact me or send an e-mail to hubert.heijkers@nl.ibm.com

Thank you!

IBM Cloud University 2017 | October

Notices and disclaimersCopyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.”

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and

the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actionsthe customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

IBM Cloud University 2017 | October

Notices and disclaimers continuedInformation concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services®,Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.