SPDE engine usage made easy with the SASﬁ 9.1 … · SPDE engine usage made easy with the SASﬁ...

Copyright © 2003, SAS Institute Inc. All rights reserved.

SPDE engine usage made easy with the SAS® 9.1metadata architectureJochen KirstenStorage Technology StrategistSAS EMEA Headquarters


"Everything should be made as simple

as possible, but not simpler."Albert Einstein, 1879-1955


SAS 9.1 I/O-Engines

! Base

! SPDE

! Scalable Performance Data Server


When To Use The SAS Base Engine?

! Single CPU / single disk computer� Single-threaded

! Small data sets

! Catalogues

! Views

! Integrity constraints

! Default engine for SAS 9.1


When To Use The SPDE Engine?

! Multi-CPU / multi-hard disk computer / server� Multi-threaded

! Scalability, performance, throughput

! Support for more than 2G rows / 32K columns

! On-the fly sorting with BY-statement

! Hybrid indexing technology� Fast by multi-threaded evaluation and creation� Small footprint


When To Use SASScalable Performance Data Server?! All the above for SPDE with the exception of

Linux and OS/390

! Client / Server environment

! Concurrent multi-user update of large amountsof data


libname scale spde 'c:\disk01' /* primary path */metapath=('c:\disk01') /* metadata path */datapath=('d:\disk02' /* 1st data path */

'e:\disk03' /* 2nd data path */ 'f:\disk04' /* 3rd data path */ 'g:\disk05') /* 4th data path */

indexpath=('i:\disk06'); /* index path */

An SPDE libref is quite complex to code when scalability is desired...

partitionpartition 11disk0disk022

partitionpartition 55


metadatametadata

disk0disk011

partition 5partitionpartition 22disk0disk033

partitionpartition 66partitionpartition 33disk0disk044


partitionpartition 88 indexindex

disk0disk066


... and even more complex when full feature control is required

libname scale spde 'c:\disk01' /* primary path */metapath=('c:\disk01') /* metadata path */datapath=('d:\disk02' /* 1st data path */

'e:\disk03' /* 2nd data path */ 'f:\disk04' /* 3rd data path */ 'g:\disk05') /* 4th data path */

indexpath=('i:\disk06') /* index path */partsize=64 /* partition size*/bysort=NO /* no engine sort*/startobs=10 /* first observ. */endobs=100 /* last observ. */temp=YES; /* temporary lib */


Unfortunately this wouldn't work as desired

libname noscale spde 'c:\disk01'; /* primary path */

disk0disk011

indexindex

partitionspartitions

metadatametadata


But this would ...libname scale META libid='AT00008D';

Or even better, this

libname scale META liburi="SASLibrary?@name='SPDElib'";

metadatametadata

disk0disk011

indexindex

disk0disk066

partition 5patition 1 partitionpartition 33

disk0disk044


partitionpartition 66 partitionpartition 88partitionpartition 44disk0disk055







libname scale META library='SPDElib';

metadatametadata

disk0disk011

indexindex

disk0disk066


disk0disk044







... when an OPTIONS statement like this

options metaserver=localhost

metaport=9990

metaprotocol=BRIDGE /* default */

metarepository="repository_one"

metauser="omruser"

metapass="omruser123";


... or a SASV9.CFG file with these invocation options is in place

/* OMR Server Name */-METASERVER localhost

/* Port on which OMR Server is running */-METAPORT 9990

/* Protocol used for accessing OMR Server */-METAPROTOCOL BRIDGE

/* Name of metadata repository */-METAREPOSITORY repository_one

/* Installation ID -- DO NOT MODIFY MANUALLY */-METAID 42389444-20B6-447D-8A23-35D926039974


Metadata LIBNAME Engine

! Retrieves information about target SAS data library from metadata objects in specified OMR.

! Provides a consistent method for accessing many different data sources.

! Hides engine-specific options from end-user.

! Constructs a data source-specific LIBNAMEstatement.

! Production status in SAS 9.1


Metadata Engine Process






?













C:\V9demo\SPDE\spde_data\primary




('C:\V9demo\SPDE\spde_data\meta')

('C:\V9demo\SPDE\spde_data\index')

('C:\V9demo\SPDE\spde_data\data_1' 'C:\V9demo\SPDE\spde_data\data_2')





When To Use The SPDE Engine?

! Multi-CPU / multi-hard disk computer / server� Multi-threaded

! Scalability, performance, throughput

! Support for more than 2G rows / 32K columns

! On-the fly sorting with BY-statement

! Hybrid indexing technology� Fast by multi-threaded evaluation and creation� Small footprint



























How to use the LIBNAME object?! Now that we have defined the SPDE LIBNAME

object in the metadata repository, we want to use it in an interactive SAS session.

! For that purpose we start SAS 9.1 making sure we either have the mentioned OPTIONSstatement in the AUTOEXEC.SAS orSASV9.CFG files.


Start SAS 9.1





metadatametadata

disk0disk011

indexindex

disk0disk066


disk0disk044










?





















Start SAS 9.1







Testing the Libref

Now try submitting a simple Data test.test;Do i=1 to 1000000;X=1; output;End;Run;


Testing the Libref

And check whether the files in the primary pathand the first data path look like SPDE tablecomponent files.

Primary path:

First data path:


test.mdf.0.0.0.spds9

C:\V9demo\SPDE\spde_data\data_1

test.dpf.v9demo_SPDE_spde_data_primary.0.1.spds9


In order to see the newly created TEST table a re-start of the SAS MC might be required.





Benefit for the End-User

! All the end-user has to code in order to get scalability with the SPDE engine is

! As opposed to

libname scale META libid='AT00008D';


libname scale spde �c:\disk01� /* primary path */metapath=(�c:\disk01�) /* metadata path */datapath=(�d:\disk02� /* 1st data path */

�e:\disk03� /* 2nd data path */ �f:\disk04� /* 3rd data path */ �g:\disk05�) /* 4th data path */

indexpath=(�i:\disk06�) /* index path */partsize=64 /* partition size*/bysort=NO /* no engine sort*/startobs=10 /* first observ. */endobs=100 /* last observ. */temp=YES; /* temorary lib */


Benefit for the End-User

! All the end-user has to code in order to get scalability with the SPDE engine is

! As opposed to

libname scale META libid='AT00008D';

libname scale META library='SPDElib';

libname scale spde �c:\disk01� /* primary path */metapath=(�c:\disk01�) /* metadata path */datapath=(�d:\disk02� /* 1st data path */

�e:\disk03� /* 2nd data path */ �f:\disk04� /* 3rd data path */ �g:\disk05�) /* 4th data path */

indexpath=(�i:\disk06�) /* index path */partsize=64 /* partition size*/bysort=NO /* no engine sort*/startobs=10 /* first observ. */endobs=100 /* last observ. */temp=YES; /* temorary lib */


Benefit for the Admin

! All members being created using this libref willautomatically be registered in the OMR.

! Access permissions and security can be granted centrally via SAS Management Console.

! If the data source changes, the end-user application is not at all affected.


Applies to

Any data source either requiring

! specific knowledge and understanding of its internals, hence complex coding, or

! access control to users.

E.g.

! SPD Engine / SPD Server

! SAS/Access engines

! Remote engine

Copyright © 2003, SAS Institute Inc. All rights reserved.Copyright © 2003, SAS Institute Inc. All rights reserved. 99

Date post:	28-Jul-2018
Category:	Documents
Upload:	lymien
View:	234 times
Download:	0 times

SPDE engine usage made easy with the SASﬁ 9.1 … · SPDE engine usage made easy with the SASﬁ...

Documents