Date post: | 20-Aug-2018 |
Category: |
Documents |
Upload: | vuongnguyet |
View: | 221 times |
Download: | 0 times |
w w w . i o m a r k . o r g
08Fall
IOmark-VM
TintriT5060StorageSystemTestReport:VM-160826-a
TestReportDate:26,August2016
Copyright©2010-2016EvaluatorGroup,Inc.Allrightsreserved.IOmark-VM,IOmark-VDI,VDI-IOmark,andIOmarkaretrademarksofEvaluatorGroup,Inc.intheUnitedStatesandotherjurisdictions.
2 IOmark-VMTestReport
IOmark-VM:TestIdentifier:VM-160826-a ReportDate:26,August2016
TableofContentsExecutiveSummary.....................................................................................................................................3VendorSuppliedProductDescription.........................................................................................................3IOmark-VMTestSummary..........................................................................................................................4IOmark-VMResults......................................................................................................................................5TestedConfigurationDetails.......................................................................................................................6
HypervisorConfigurationforIOmark-VMWorkload...............................................................................6StorageConfigurationforIOmark-VMWorkload....................................................................................7ConfigurationDiagram.............................................................................................................................8Connectivity.............................................................................................................................................9TestedConfigurationPricing..................................................................................................................10DetailedResults.....................................................................................................................................11IOmark-VMMeasurementsandUse.....................................................................................................12HowIOmark-VMOperates....................................................................................................................13BenchmarkApplicationWorkloadSet...................................................................................................13
WorkloadDetails...............................................................................................................................14UnderstandingResults...........................................................................................................................14
BenchmarkCriteria............................................................................................................................15MoreInformationaboutIOmark-VM....................................................................................................15AboutEvaluatorGroup..........................................................................................................................15
IOmark-VMTestReport 3
IOmark-VMTestIdentifier:VM-160826-a ReportDate:26,August2016
ExecutiveSummaryThisdocumentistheofficialbenchmarkreportforthetestedconfigurationwithaTintriT5060all-flashstoragesystemrunningtheIOmark-VMbenchmark.IOmarkisastoragespecificworkloadandbenchmarkdesignedtoteststoragesystemsperformanceusingavarietyofrealworld,applicationcentricworkloads.TheIOmark-VMbenchmarkisaspecificworkload,whichmeasuresServerVirtualizationworkloads(VMs)runagainststoragesystems.ResultsarepublishedafterauditandcertifiedapprovalbyIOmarkauthorizedauditors.IOmark-VMisabenchmarkthatcertifiesstoragesystemsforvirtualserverworkloads.Themeasurementcriterionisstorageperformance,withtherestrictionthatallstorageworkloadsmustbesupportedbythetestedsystem.AlthoughthereareCPUandmemoryconsiderations,theseaspectsarenottestedbytheIOmark-VMworkload.
TheresultsachievedbytheTintrisystemrunningIOmark-VMareasfollows:• TheT5060supported480IOmark-VM’satacostof$468.75perVM
Afulldescriptionoftheconfigurationstestedalongwithpricinginformationisprovidedinthisdocument.Thecriteriaandperformancerequirementsareasfollows:
• Forallapplicationworkloads:o Allworkloadsmustresideentirelyonthetestedsystemo Workloadsarescaledinsetsof8workloadso 70%ofresponsetimesforI/O’smustnotexceed20mso Theaverageresponsetimeforeachapplicationtypemustnotexceed20mso Thereplaytimemustcompletewithin1hourand15secondsforeach1hourworkload
• Forhypervisoroperations:o Clone,deploy,boot,softwareupgrade,VMdeletiono Storagemigration(akaStoragevMotion)betweenstoragevolumes
VendorSuppliedProductDescription
Tintrisimplystoresvirtualizedworkloads—offeringafullyintegratedVM-AwareStorage(VAS)systemforvirtualizedenterprisesandcloud.Balanceyourprojectsandworkloadsacrosshigh-performanceAll-Flashandaward-winningHybrid-Flash.Encrypteverythingwithasingleclick.ReplicateindividualVMstoremotelocations.SyncchildVMswithmasterVMstospeeddevelopmentcycles.AndmanageitALLfromasinglepaneofglass.That’showyoukeepstoragesimple.
Systemdesign,including:
• TintriVMstoreUI,drilldowntoVMandvirtualdisk,inTintriweb-consoleorHypervisorplugin• Supportformultiple,concurrentHypervisors(VMware,Hyper-V,KVMandXenServer)• Hypervisor-embeddedstorageforeaseofmanagementanddeploymentusingexistingtools• Consistentperformanceandlatencywithall-flashT5000series• OperationsmanagedperVM,includingQoS,snapshots,clonesandreplication
4 IOmark-VMTestReport
IOmark-VM:TestIdentifier:VM-160826-a ReportDate:26,August2016
Tintriall-flashT5000seriessystemsprovidethefollowingenterprisestoragefeatures:
• Capacityfrom6TBto92TBraw,and17-308TBeffectivecapacitywithdatareduction• Tintriall-flasharraysassignI/OonaperVMbasis• Qualityofservice(QoS)settingsonindividualVMstoguaranteeperformance• Datareductionincludesdeduplication,compression,thinprovisioning,andzerocopy
optimization
IOmark-VMTestSummaryForthetestedconfiguration,thefollowingdataisprovided
Item Value
TestingIdentifier: VM-160826-a
Product(s): TintriT5060
TestSponsor: Tintri
Auditor: EvaluatorGroupInc.
Table1:TestIdentifierInformation
Item Value
IOmark-VMVersion: Version:IOmark-VM3.8
TestingCompleted: July2016
EquipmentAvailability: August2015
AuditCertificationDate: 26,August2016
ReportDate: 26,August2016
Table2:TestRevisionandDates
IOmark-VMTestReport 5
IOmark-VMTestIdentifier:VM-160826-a ReportDate:26,August2016
IOmark-VMResultsShownbelowaretheIOmark-VMresultsforthesystemundertest.ThedefinitionandworkloadcharacteristicsofthebenchmarkareprovidedinAppendixA.PriceinformationprovidedbelowisexplainedindetailinTable8inthisreport.Table3belowshowsanoverviewoftheIOmark-VMresults.
Config IOmark-VMTotalVM’s
IOmark-VMResponseAvg.
TestedCapacity
TestedRAIDLevel
TotalPrice IOmark-VM$/VM
T5060 480 4.33ms 6.6TiB(7.2TB)
TintriDefault
$225,000.00 $468.75
Table3:IOmark-VMResultDetails
ThetotalnumberofIOmark-VMvirtualmachinessupportedisshownaboveinTable3,basedontheIOmark-VMworkloadsetsshowninTable4below.Eachapplicationsetconsistsof8virtualmachines,thus60applicationsetsyields480IOmark-VM’sreported.TheVMwarevCenterServer™operationvaluesarealsoshownbelow,withtwocomponentsbeingreported.The“CloneandDeploy”portionoftheworkloadcreatesaclonefromaspecificVMtemplate,startstheVMrunningandthenupgradesitsversionofVMwaretoolsinstalled.Thereportedvalueindicateshowmanyoperationcycleswerecompletedduringthe1-hourtestrun.Similarly,thestoragevMotionvaluereportedindicateshowmanymigrationcycleswerecompletedduringthe1-hourtestrun.Acombinedscoreiscalculated,knownasthe“HypervisorWorkloadScore,”whichistheratioofreportedresultstotheminimumrequiredresults.TheminimumnumbersofvCenteroperationsforpassingthetestare6cloneanddeployand3storagevMotionoperationsrespectivelyforconfigurationssupporting21IOmark-VMsetsormore.DetailsofpassingresultsshownbelowinTable4:
Config IOmark-VMSets
ReadResp.
Average
WriteResp.
Average
#vCenterCloneandDeploy
#vCenterstoragevMotion
HypervisorWorkload
Score(1-inf.)
T5060 60 3.04ms 4.44ms 8 7 5.47
Table4:IOmark-VMPassingResultDetails
6 IOmark-VMTestReport
IOmark-VM:TestIdentifier:VM-160826-a ReportDate:26,August2016
TestedConfigurationDetailsThissectioncoverstheconnectivity,configurationandpricinginformationforthesystemundertest.
HypervisorConfigurationforIOmark-VMWorkload• AsingleTintriT5060systemwasused• Multiplemountpointswerecreatedtothetwohostsusedfortesting• AvirtualdiskwascreatedforeachofthereportedIOmark-VM’scertified(480)• Datareductionwasenabled,bygeneratingdatathatis2:1compressible
o RAIDlevelutilizedisnotuserselectable,usedTintridefaultsettingsDetailedconfigurationparametersforthesystemundertest,includingconnectivityareprovidedbelowinTable6.
StorageSystemParameter Value
Hypervisor VMwarevSphere™ESXi6
Numberofinterfacestothestoragesystem: 4PerNode(2/controller=4total)
Connectivitytostoragesystem: 2@10GbEthernet/controller
Hypervisorstorageprotocolused: NFSv3
Hypervisorversion: VMwareESXi6.0U2(3620759)
Thinprovisioning: UtilizedinTintridatastore
HypervisorStorageAccess: NFSdatastore
DatastoreFilesystem: NFS-filesystemaccesstoTintri
VAAI: Yes,usingTintriVAAIplugindrivers
SATP: N/A
PSP: N/A
TotalcapacityofsystemallocatedtoIOmark-VM: 6.6TiBusable(7.2TB)
Table6:HypervisorConfigurationParameters
NOTE:PerIOmarkrequirements,a“write-only”workloadisrunpriortotheactualworkload.Thispre-writesdatatoallstoragelocationsreferencedduringtesting.Bypre-writingdatapriortoactualworkloadtesting,thereisnowriteallocationpenaltyassociatedwiththinprovisioning.Thisalsoensuresthatwhenreadsareperformedthestoragesystemreadsthemedia,ratherthanreturningzero’sforunallocatedaddresses.
IOmark-VMTestReport 7
IOmark-VMTestIdentifier:VM-160826-a ReportDate:26,August2016
StorageConfigurationforIOmark-VMWorkload• AsingleTintriT5060providedthepooledcapacityacrossbothofthehosttestnodes• 6datastoreswerecreatedoneachhostfortesting.(12total)• EachVM’sVMDKwasallocatedusing“thinprovisioning”perTintriandNFSdefault
DetailedStorageSystemconfigurationparametersforthestoragesystemundertest,includingconnectivityisprovidedbelowinTable7.
StorageSystemParameter Value
StorageSystemfirmware Tintri4.2.0.6
HighAvailabilityAccess Yes(active/standby)
Totalrawcapacityofsystemundertest(SUT) 12TB
Totalusablecapacityofsystemundertest(SUT) 6.6TiB(7.2TB)usable,upto35+TBeffectivewithdatareduction
Datastores Totalof6datastores/host(NoLUNsrequired)
Thinprovisioning: Yes
RAIDLevel(s) Tintridefault,RAID6
TotalCacheCapacity: N/A
ReadCache: N/A
WriteCache: N/A
VAAIFeaturesEnabled: Yes
-NFSFullClone Yes
-NFSExtendedStats Yes
-NFSReserveSpace Yes
Automatedtieringwithinthestoragesystem: N/A(T5060isall-flash)
Deduplicationorcompressionofdata: Yes,bothin-lineandalwayson
Storagesystemclones/writeablesnapshots: Yes,utilizedstorageclonesduringtesting
Typeofstoragesystemclone: Tintrinativeclones
StorageMediaUtilized: -
-SSD’s 24x480GB(includesspares)
-15KRPM NA
-10KRPM NA
-7.2KRPM NA
Table7:StorageSystemConfigurationParameters
8 IOmark-VMTestReport
IOmark-VM:TestIdentifier:VM-160826-a ReportDate:26,August2016
ConfigurationDiagramThelogicaldatalayoutofthetestconfigurationisshownbelowinFigure1.SinceTintriisanNFSdatastore,blockLUNsorvolumeswerenotutilized.Instead,individualvirtualdiskswereassignedtoVM’sasrequired,distributedacrossthe12logicalmountpoints.TheVMDK’sforeachVM’sworkloadwereallallocatedfromthesameTintriDatastorecapacitypoolcreatedbyTintriacrossthecomputenodes.
Figure1:LogicalSystemConfiguration
Note:AlthoughasingleNFSdatastorecouldhavebeenused,itwasdeterminedthatutilizingmoremountpointsimprovedperformance.Atotalof12mountpointswereused,whichequatesto12hostsystemsmountingasingleTintrishareddatastore.
TintriStoragePool–(12TBRawCapacity)
PooledCapacityfrom
TintriT5060Storage
IOmark-VM
WorkloadNodes
NFSMountPoints(6/Host)
45GB 80GB
DS2DB
VirtualDisks
OlioWeb
VirtualDisks
Exchange
VirtualDisks
OlioDB
VirtualDisks
DS2Web
VirtualDisks
vCenterOp.
VirtualDisks
80GB 40GB14 3
3
40@ 40@40@ 40@ 40@ 3@3GB
3
IOmark-VMTestReport 9
IOmark-VMTestIdentifier:VM-160826-a ReportDate:26,August2016
ConnectivityStorageconnectivityusedwas10GbE,usingLACPtobondthetwonetworkinterfacespercontroller.Eachtesthostused210GbElinkstoa10GbEthernetswitch,foratotalof8connectionsacrossboththetwohostsandthedualcontrollerT5060.TestingdidnotutilizearedundantHAconfiguration,althoughproductiondeploymentassumesconnectivitytoaHAnetworkinfrastructure.ThetestedconfigurationconnectivitydiagramisshownbelowinFigure2.
Figure2:PhysicalSystemConnectivity
10 IOmark-VMTestReport
IOmark-VM:TestIdentifier:VM-160826-a ReportDate:26,August2016
TestedConfigurationPricing
Item Description Qty. Ext.ListPrice
1 TintriT5060(Includesbasesoftware) 1system $225,000.00
2 Tintri3yearsmaintenance Included N/A
Total HW+SW+3yearservice&support $225,000.00
Table8:IOmark-VMPriceInformation(4NodeConfiguration)
Note:Supportincludedforallhardwareandsoftware
IOmark-VMTestReport 11
IOmark-VMTestIdentifier:VM-160826-a ReportDate:26,August2016
DetailedResultsIOmark-VMperformanceresultsaremeasuredbyapplicationworkload.Thecumulativeresponsetimesofall480applicationsthatcomprisedthetestworkloadareshownbelowinFigure3,withaverageresponsetimesreportedperapplicationtypebelowinTable9.
Figure3:PercentageofTotalResponseTimesatMeasuredValue
FromFigure3above,theprimaryresponsetime(s)ofinterestare:
• Nearly85%ofresponsetimeswerelessthan5ms.• 96.7%ofresponsetimeswerelessthan20ms.(exceedingrequirements)
ApplicationWorkload AverageResponseTimes
DVDStoreDB 3.34ms
ExchangeMailServer 4.27ms
OlioDatabaseServer 3.93ms
OlioWebServer 5.38ms
DVDStoreWebApp1 4.57ms
DVDStoreWebApp2 4.57ms
DVDStoreWebApp3 4.57ms
WindowsStandby 4.57ms
Table9:ApplicationResponseTimesbyWorkloadType
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
2 5 10 20 30 60 100
Percen
tageofV
alue
s
ResponseTimeinms.
CumulafveResponseTimes
Tintri5060
12 IOmark-VMTestReport
IOmark-VM:TestIdentifier:VM-160826-a ReportDate:26,August2016
AppendixA-IOmark-VMOverviewTheabilitytorecreateaknownworkloadisimportantforcomparingasystemagainstpotentialalternatives.EstablishingareferenceorbenchmarkworkloadenablessystemvendorsaswellasresellersandITuserstocompareseveralsystemsutilizingaknownworkload.Specifically,theIOmark-VMbenchmarkrecreatesastorageworkloadthattypicallyoccursinavirtualinfrastructureenvironment.Theworkloadisnon-syntheticandrecreatesseveralapplicationsthatarecommonlyfoundinvirtualizedserverenvironments.
Figure3:IOmark-VMConceptualOverview
IOmark-VMMeasurementsandUseDatacentersrunningapplicationsinavirtualinfrastructurecontainmultipleworkloadsrunningonavirtualizationplatform.OftenmultiplephysicalserverssharetheresourcesofasinglestoragesystemprovidingprimarystorageforbothvirtualmachineOSandapplications.Currently,severalbenchmarkshavebeendevelopedthatfocusontheserveraspectsofinfrastructure,includingtheCPU,memoryandI/Obandwidthcapabilitiesoftheinfrastructure.However,therehasbeennocorrespondingdevelopmentofstandardizedworkloadsdesignedtodrivestorageworkloadsfortheseapplicationenvironments.ByestablishingasetofstandardapplicationsandcapturingtheirI/Ostreams,itispossibletorecreateapplicationbasedstorageworkloadsforthesecomplexenvironments.IOmark-VMisdesignedutilizingtheseconcepts,andassuchisthefirstbenchmarkdesignedtoaccuratelygenerateapplicationworkloadsforstoragesystems,enablingdirectcomparisonofstoragesystemconfigurationsandtheirabilitytosupportaspecificnumberofapplications.
Hypervisor*
Linux*64b*
DVD*Store**
Web*Server*
Linux*64b*
DVD*Store**
Web*Server*
Linux*64b*
DVD*Store**
Web*Server*
Linux*64b*
DVD*Store**
Database*
Linux*64b*
Olio**
Web*Server*
Win*2008*
Standby***
Win*2008*
Mail*Server**
Web*Server*
Linux*64b*
Olio**
Database*
Storage(System((***(System*Under*Test)*
IOmark-VMTestReport 13
IOmark-VMTestIdentifier:VM-160826-a ReportDate:26,August2016
Additionally,IOmark-VMrealizesthatasignificantimpactonstoragemayoccurfromadministrativefunctionscommoninvirtualinfrastructures.Forthisreason,severalhypervisor-basedfunctionsareapartoftheIOmark-VMworkload.Theseadditionaloperationsinclude;cloningavirtualmachine,bootingaVMandupdatingsoftware,whilealsomigratingavirtualmachinefromonestoragevolumetoanother.
HowIOmark-VMOperatesIOmark-VMusestheconceptofworkloadreplay.I/Ostreamsarecapturedfromactualrunningapplicationsandthen“replayed”sothattheexactsequenceandI/Ocommandsareissued.Thisallowsthecreationofaworkloadthatisindistinguishablefromanactualworkloadtothesystemundertest,whilebeingreproducibleandrequiringfewerresources.Additionally,thetestenvironmentislessexpensive,easierandfastertocreatesinceactualapplicationsarenotrequired.BecauseCPUandmemoryarenotconsumedrunningapplications,amuchhigherI/Oworkloadmaybegeneratedwithasetofserverresourcesthanispossibleusingnativeapplications.Thisratioistypically10:1,butmayvary.InFigure3onthepreviouspage,asinglesetofapplicationsisdepictedrunningonasinglephysicalhostinavirtualinfrastructure.Inordertoscaleuptheworkloadonastoragesystem,additionalapplicationssetsmaybeaddedtothesame,ortootherphysicalhosts.Theonlylimitationtothescaleofthetestisthephysicalinfrastructuresupportingtheworkload.Sufficient,CPU,memoryandI/Ocapabilitiesmustbeavailabletorunadditionalworkloadsets.Unlikeartificialworkloadgenerationtools,IOmark-VMrecreatesaccuratereadvs.writeandrandomvs.sequentialI/Orequests.AnothermeasurementofIOmark-VMisthatitcreatesaccurateaccesspatterns,thusenablingstoragecachealgorithmstoworkproperly.Finally,IOmark-VMmaintainsanaccurateratioofperformancetocapacityasworkloadsarescaled,ensuringthatstorageperformanceismeasuredwithrespecttostoragecapacityaccurately.Asaresult,IOmark-VMmaintainsanaccurateratioofI/Otocapacity,producingresultsapplicabletoITusers.
BenchmarkApplicationWorkloadSetAconceptutilizedfortestingmultipleapplicationsisthatof“Applicationsets”,alsoknownas“tiles.”Asetof8applicationsisruntogether,alongwithseveralcommonhypervisorinfrastructureoperations.Inordertoscaletheworkloadupandplaceahigherloadonthestoragesystem,additionalapplicationsetsarerun.Applicationsetsarealwaysruntogetherforofficialbenchmarkresults,alongwithadefinedsetofinfrastructureoperations.
14 IOmark-VMTestReport
IOmark-VM:TestIdentifier:VM-160826-a ReportDate:26,August2016
ThespecificapplicationscomprisingaworkloadsetaredetailedbelowinTable10.
Application GuestOS StorageCapacity/Instance
MicrosoftExchange2007 MicrosoftWindowsServer2008,Enterprise,64bit
80GB
OlioDatabase SuSELinuxEnterpriseServer11,64bit
14GB
OlioWebserver SuSELinuxEnterprise11,64bit 80GB
IdleWindowsServer MicrosoftWindowsServer2003SP2EnterpriseEdition,32-bit
10GB
DVDStoreDatabase SuSELinuxEnterprise11,64bit 45GB
DVDStoreWebServer1 SuSELinuxEnterprise11,64bit 10GB
DVDStoreWebServer2 SuSELinuxEnterprise11,64bit 10GB
DVDStoreWebServer3 SuSELinuxEnterprise11,64bit 10GB
HypervisorClone&Deploy N/A-VMwarevCenterrequired 15GB
HypervisorStorageMigration N/A-VMwarevCenterrequired 30GB
-- -- Total=305GB
Table10:IOmark-VMApplicationOverview
Thetotalcapacityrequiredforeachsetofapplicationsisapproximately305GBofcapacity.Eachadditionalworkloadsetrequiresanadditional305GBofcapacity.
WorkloadDetailsTheOlioapplicationconsistsofbothadatabaseserver,andawebclientrunningondifferentvirtualmachineswithapre-loadeddataset.FormoredetailsonOliosee:http://incubator.apache.org/olio/TheDVDapplicationconsistsofasingledatabaseserveralongwiththreewebclients,eachrunningonadifferentvirtualmachineusingpredefinedworkloadanddataset.FormoredetailsonthepubliclyavailableDVDdatabaseapplicationsee:http://linux.dell.com/dvdstore/TheExchangeserverisaMicrosoftmessagingandemailserver.OnlytheserverportionofExchangeisrecreatedinthisworkloadset,withtheclientworkloadsnotbeingapartoftheI/O,onlyindirectlythroughtheirrequeststothemessagingserver.ThetwohypervisorworkloadsarebasedoncommonoperationsperformedinvirtualinfrastructureenvironmentsandrequiretheavailabilityofaVMwarevCenterservertoperformtheoperations.
UnderstandingResultsIOmark-VMproducesresultsindicatingtheresponsetimeofastoragesystemgivenaparticularworkload.Basedonestablishedcriteria,theseresultsinturndictatehowmanytotalvirtualmachinesetsaresupportedbyaspecificstorageconfigurationandtheaverageresponsetime.ThereportisauditedforaccuracyandissuedbyEvaluatorGroup,Inc.,anindependentstorageanalystfirm.Note:IOmark-VMresponsetimescannotbedirectlycomparedtoVMmarkresponsetimes.IOmarkmeasuresresponsetimesofindividualI/Orequests,whereasVMmarkmeasurestransactionresponsetimes,consistingofmultipleI/Ooperationsalongwithdatacalculations.
IOmark-VMTestReport 15
IOmark-VMTestIdentifier:VM-160826-a ReportDate:26,August2016
BenchmarkCriteriaIOmarkhasestablishedthebenchmarkcriteriafortheIOmark-VMworkload.Theperformancerequirementsareestablishedasfollows:
• Forallapplicationworkloads:o Workloadsarescaledinsetsof8workloadso 70%ofresponsetimesforI/O’smustnotexceed20mso Theaverageresponsetimeforeachapplicationmustnotexceed20mso Allstoragemustresideonthestoragesystemundertesto Thereplaytimemustcompletewithin1hourand15secondsforeach1hourworkload
• Forhypervisoroperations:o Clone,deploy,boot,softwareupgrade,VMdeletiono Storagemigration(akaStoragevMotion)betweenstoragevolumes
MoreInformationaboutIOmark-VMFormoreinformationabouttheIOmarkbenchmark,atheoryofoperationsguide,publishedresultsandmore,visittheofficialwebsiteathttp://www.iomark.org.Somecontentisrestrictedtoregisteredusers,sopleaseregisteronthesitetoobtainallavailableinformationandthelatestresults.
AboutEvaluatorGroupEvaluatorGroupInc.isatechnologyresearchandadvisorycompanycoveringInformationManagement,StorageandSystems.ExecutivesandITManagersuseusdailytomakeinformeddecisionstoarchitectandpurchasesystemssupportingtheirdigitaldata.Wegetbeyondthetechnologylandscapebydefiningrequirementsandknowingtheproductsin-depthalongwiththeintricaciesthatdictatelong-termsuccessfulstrategies.www.evaluatorgroup.com@evaluator_group