Deploying SAS®9 on SolarisTM 10: Containing Cost, Consolidation, &
Capacity Challenges with Virtual Simplicity
A SAS R&D Case Study
Maureen ChewStaff EngineerSun MicrosystemsMarch [email protected]
2 Sun Microsystems, Inc.
Table of Contents
Many IT Challenges – Complex and Diverse.......................................................................... .........................3
Solaris 10 Containers – Addressing Cost, Consolidation and Complexity..................................................3
Lessons Learned and Best Practice Considerations.......................................... ...........................................4
● Third Party Application and Version Considerations.................................... ...........................................5
● Application and Performance Considerations.................................................... .....................................5
● Whole vs. Sparse Root Zones.......................................................................... ......................................6
● Zone Root Location................................................................................................................................. 7
● Shared Binaries................................................................................................. .....................................8
● Memory Requirements............................................................................................................................ 8
● Resource Management........................................................................................................................... 8
● The Role of ZFS.................................................................................................................................... 10
● Solaris Fingerprint Database (sfpDB)................................................... ................................................10
Summary............................................................................................................... ...........................................11
Acknowledgments................................................................................................................. ..........................11
References.......................................................................................................... .............................................11
Appendix A – Introduction to Solaris 10 Containers .............................................. .....................................12
Appendix B – Container Creation Cookbook................................................................. ...............................14
3 Many IT Challenges – Complex and Diverse
Many IT Challenges – Complex and DiverseAt SAS headquarters in Cary, NC, R&D staff responsible for both 9.1.3 and 9.2 product development, testing,
and coordination with global development teams faced major challenges. Solution development for 9.1.3 was
continuing at a rapid pace while efforts towards the next generation 9.2 platform was ramping up. Not only
were there two major development release trains, but as with typical, large software development efforts, there
were nightly and weekly builds for each development track. On the testing front, there was the obvious need
for unit and integration testing. SAS R&D realized that production enterprise deployments differed from
dev/test deployments and required modeling various production, enterprise class environments. Consequently,
the need to support enterprise class, customer centric test deployments became an added requirement.
Because of the convergence of 2 major simultaneous development efforts, a plan needed to be put into place
to address the complex and diverse needs of the various development and test communities. Faced with the
typical challenges of today's IT data centers, the cost and management complexity for multiple compute and
data servers could easily have grown out of bounds. The needs and requirement challenges could be
summarized as:
● Architect and validate realistic, enterprise ready SAS deployments
○ Discover and document system management best practices
● Create Test/Production Environments for both 9.1.3 and 9.2 releases
○ Provide ability for rapid provisioning of test environments
● Support global development teams who collectively require follow the sun support
● Foster cross product collaboration
● Create common test data repositories for testing and debugging
● Regression Testing
● Migration Path Testing
○ Inter-release, Intra-release
● Consolidation of Administrative resources
○ Reduce server and storage requirements
○ Reduce software installation duplication
Solaris 10 Containers – Addressing Consolidation, Cost, and ComplexityAfter careful evaluation of various alternatives, a project proposal was built around the use of Solaris 10
Containers to meet the urgent and critical needs of this diverse community. (If the reader is not familiar with
Solaris 10 Containers, the concepts of whole vs sparse root zones, and zone roots, see Appendix A - An
Introduction to Solaris 10 Containers). A Container is a superset of the combination of Zones and Resource
Management capabilities. In this paper, the terms Containers and Zones are
(unofficially) used interchangeably. Additionally, SAS R&D felt that Solaris provided a
rich set of tools such as dtrace(1), truss(1M), pfiles(1) that could help debug
problems should they arise.
The SAS R&D proposal was to create a Shared Production and Test Environment.
Solaris 10 Containers would be used to create virtual instances of different
deployment options. After the proposal was given the green light, the project (code
named: PRDSHARE) went forward quickly and staged on the following configuration:
● SunFire E2900
4 Solaris 10 Containers – Addressing Consolidation, Cost, and Complexity
○ 8X1.5 GHz dual core UltraSPARC IV+ processors
○ 128GB RAM
● Sun StorageTek 6540 - 6+TB
The initial deployment consisted of over 20 Containers; half for 9.2 testing and the other half dedicated to 9.1.3
testing. Figure 1 below is a logical diagram of the containers used to support 9.2 testing. A similar set for 9.1.3
was also configured.
Figure 1: PRDSHARE 9.2 Container Deployment (7 non-global containers shown)
Lessons Learned and Best Practice ConsiderationsPRDSHARE is a sophisticated environment with over 20 zones in which each zone is supporting its own
complex configuration of applications. We outline some lessons learned and best practice considerations:
● Third Party Application and Version Considerations
● Application and Performance Considerations
● Whole vs. Sparse Root Zones
● Zone Root Location
● Shared Binaries
● Memory Requirements
5 Lessons Learned and Best Practice Considerations
● Resource Management
● The Role of ZFS
● Solaris Fingerprint Database (sfpDB)
Third Party Application and Version Considerations - A sample of some of the 3rd party components
for each major SAS release is listed below. Check the SAS site below for the official current and
comprehensive support listing:
http://support.sas.com/resources/thirdpartysupport/v913sp4/index.html
Third Party Software Support 9.1.3 SP4 9.2
Java Runtime Environment (JRE) 1.4.2_09 Java 5, Update 12
BEA Weblogic 8.1 SP4, SP6 9.2
IBM Websphere 6.0.2.15, 6.0.2.19 6.1.0.2
Xythos Webdav 4.2.34.4, 4.2.35.3 4.2.34.4
At the current time, there have been 4 update releases to Solaris 10:
● Solaris 10 8/07 - Update 4
● Solaris 10 11/06 - Update 3
● Solaris 10 6/06 - Update 2
● Solaris 10 1/06 - Update 1
● Solaris 10 Initial General Availability (3/05)
When non-global zones are installed, the upgrade process to move from one Solaris 10 Update release to the
next requires careful planning, especially as the number of zones increases. It is highly recommended that any
new Solaris installation begin with the latest available Update release, today Solaris 10 8/07. The PRDSHARE
project launched with Solaris 10 1/06 and was subsequently upgraded to both Solaris 10 11/06 and Solaris 10
6/06 as those releases became available. The length of time to complete the upgrade increases incrementally
with each additional zone hosted on the platform. With the introduction of Solaris 10 8/07, Live Upgrade
support of systems hosting zones was added. Live Upgrade continues to be the recommended method to both
patch and upgrade Solaris systems, especially those hosting Zones. For further information, visit:
“Upgrading a Solaris 10 System That Has Installed Non-Global Zones”
http://docs.sun.com/app/docs/doc/817-1592/6mhahup0m?l=en&a=view
Application and Performance Considerations – From an application perspective, what are the
differences between running in a non-global zone and running in a global zone? Zones provide a secure and
isolated environment which cannot compromise processes in either other non-global zones or the global zone.
As such, there are a few restrictions that a user or user application may run into. First and foremost, from a
non-global zone, processes do not have access to any other processes outside of their zone. prstat(1) will only
show information about processes local to the zone running the command. Additionally, applications or utilities
that need to read kernel memory or that install kernel device drivers will not work in a non-global zone. Many
of the standard system monitoring utilities (i.e.: prstat(1), vmstat(1), zfs iostat(1), etc) can be used at the non-
global zone level but for the ones that can't, they can be run, unmodified, at the global zone level given proper
authorization.
6 Lessons Learned and Best Practice Considerations
DTrace(1) has three distinct privilege sets; dtrace_kernel, dtrace_user and dtrace_proc. While dtrace_kernel
operations cannot be initiated from within a non-global zone, beginning with Solaris 10 8/07 (Update 4)
dtrace_user and dtrace_proc privileges can be assigned to a zone. This allows the opportunity for developers
and application owners within zones to leverage DTrace. Of course, in any of the Solaris 10 releases, the
ability existis for privileged administrators within the global zone to use all DTrace capabilities to
comprehensively observe all zone operations. Subsequently, DTrace can be leveraged to effectively observe
kernel, user and process interaction of all ( SAS or non SAS) applications both within and across zones.
At this time, there are no known issues with running SAS applications in zones. A few SAS binaries such as
elssrv and sasauth require setuid root privileges but that presents no issue because each zone has its own
specific set of root privileges and permissions. For further information about application heuristics in zones,
visit:
“Bringing your Application into the Zone”
http://developers.sun.com/solaris/articles/application_in_zone.html
One item to note is on the common practice of using shared home directories where a given user is presented
the same home directory regardless of the server they log into. This is not specific to running SAS in multiple
zones; its relevant if running SAS across multiple systems as well. A SAS installation requires several different
user ids. It is not good practice for these users to have a common home directory across zones or systems.
SAS installation state is stored in $HOME/vpd.properties and $HOME/.sasprefs and could become corrupted if
overwritten by an alternative installation.
Unlike other virtualization technologies which translate and/or emulate requests to I/O devices, that is not the
case with zones. I/O requests are directly handled by the global zone so there is no performance penalty for
applications that are heavily I/O dependent as is the case with many SAS applications. In general, there is little
to no performance overhead from running in a zone – typically, its less than 1%.
The benefits of containers/zones are many, with few, if any, tradeoffs and constraints. Containers serve to
maximize server utilization, create flexible, agile IT deployment platforms, provide secure, isolated
environments with little to no performance penalty. This provides a seamless foundation for virtualization
strategies.
One of the constraints of container based virtualization is that all the containers have to be, more or less,
running the same version of Solaris. One zone cannot be running Solaris 10 6/06(U2) if the global zone is
running 8/07(U4). Note: BrandZ Containers do allow different versions of the OS to run in a container (i.e.:
Solaris 8 and 9 in a Solaris 10 SPARC Container) and even different OS' (i.e.: Centos in a Solaris 10 x64
Container) but this is not, at this time, a recommended way to run SAS applications.
Early in the architecture phase, PRDSHARE was designed to be oversubscribed in terms of user population.
Similar to the way that airlines routinely overbook flights on the premise that there will be a given number of late
cancellations, no shows, missed connections, etc, PRDSHARE was configured to have many more containers
or zones than what the system could reasonably handle should everything be running concurrently. To be
more precise, Solaris could handle the load just fine, rather, the resulting service level may not be acceptable
7 Lessons Learned and Best Practice Considerations
to the user community. The current system has 16 cores and 21 zones configured. Each zone is running its
own set of SAS services or third party applications. Just considering CPU resources, if the cumulative CPU
utilization for all applications in a given zone utilizes 4 cores on average (some more, some less), the system
would need approximately 80 cores, or roughly 5 times as many as configured to service all of the applications
with minimal CPU wait time (and that isn't counting CPU resources needed for the Solaris Operating
Environment itself). While all the zones are booted with the SAS and application services running, not all of
them are in use at any given time. Suffice it to say that close monitoring is required as user communities start
and stop their testing/usage cycles since this configuration could easily be running at 100% utilization with
relatively few zones in full action.
While PRDSHARE was configured in this oversubscribed fashion, only a given subset of zones were in use at
any given time. The powerful Container resource management features provide insurance, flexibility and agility
to dynamically adjust CPU and memory allocation should the requirement for resource prioritization arisen.
Whole vs. Sparse Root Zones - When zones are configured, one of two categories of configurations exist
- Sparse and Whole Root Zone. A sparse root zone is one where one or more directories are shared from the
global zone via a read-only loopback file system (LOFS) mount. When a zone is installed, the default
configuration will be a sparse root zone where /lib, /platform, /sbin, and /usr are shared or inherited by the non-
global zone from the global zone. Obviously, the fewer directories which are shared or inherited implies that
the level of sparseness will converge towards zero and a sparse root zone will start to more closely resemble
a whole root zone.
Whole root zones are necessary if an application or its installation needs to write into directories that would
otherwise be mounted read only if it were a sparse root zone. For instance, an application might need to
create symbolic or hard links into /usr or requires installation into /usr/local. Whole root zones provide
maximum configurability options since all of the required and any selected optional Solaris packages are
installed into the private file systems of the zone. Whole root zones take on the order of 10 Gigabytes of disk
space while a sparse root zone would consume on the order of 100 Megabytes. Additionally, whole root zones
take longer to provision and upgrade.
In general, creating sparse root zones and sharing installations at a global level is recommended and more
efficient. However, there are very valid, and sometimes mandatory, reasons for choosing whole root zones
instead. Once a zone is installed, you can't switch from a whole root to sparse root or vice versa. So, in the
absence of knowing in advance whether any third party software package will need to write into /usr, the only
option would be to use a whole root zone. If it is not a problem to later re-install application software down the
road, then choose the sparse root option with the possibility that the zone might need to be re-created and
previous application installation and configuration re-done.
In the PRDSHARE environment, 3 out of the 20+ were created as whole root zones; the rest were created as
sparse root zones. Two of the whole root zones were needed because of 3rd party application requirements
(Websphere & Webseal), while the third one was to have a whole root zone available for additional
testing/comparison. Note: Per the “How to Get Started with IBM WebSphere Application Server on Solaris 10
and Zones” paper, it is not necessary to install Websphere in a whole root unless the Websphere web server
plugin needs to be installed. In this case, Websphere needs to create symbolic links in /usr and will require a
8 Lessons Learned and Best Practice Considerations
whole root zone in order for the installation to complete successfully.
Zone Root Location – The zone root directory is the directory root in which all the zone system files are
stored. For a whole root zone, it effectively contains a copy of the Solaris operating environment. PRDSHARE
was initially configured with all of the zone roots located on the primary boot disk (which was mirrored with
Solaris Volume Manager). An obvious concern was the resulting I/O workload just to service all of the zones.
Indeed, this was the case, as I/O contention to these disks surfaced early. Because the default SAS
configuration points SAS WORK to /usr/tmp (which is a symbolic link to /var/tmp), it was quickly determined
that the majority of the zones were unknowingly sending great streams of I/O to this primary boot disk which
also housed all of the Zone ROOTs and was thus creating a major I/O bottleneck. There was an easy and very
effective solution to this problem – move SAS WORK to the faster, more powerful external Sun StorageTek
6540 storage. Once that minor SAS configuration was done, I/O to the primary boot disk was no longer an
issue.
A good disk layout strategy is to install all system and application software to the boot disk and have distinct
storage pools/luns/filesystems for user home directories and other ones for data storage. Separation by some
distinct factor is generally a good idea. This strategy supported the rationale for locating all of the zone roots
on the boot disk. While different enterprise environments may warrant or require spreading out the zone root
locations, there were several over-arching reasons as to the rationale for the PRDSHARE environment:
● Simplicity – the SunFire E2900 had 2 internal disks which were configured as a mirrored pair. External
storage to the Sun StorageTek 6540 was connected via fibre channel connections. In this case, all OS
and application software resided on the internal disks while all data resided on external storage.
● Efficiency – out of the 20+ zones which were configured and installed, only 3 were configured as whole
root zones, the rest were configured as sparse root zones. Additionally, several of the SAS
installations, Java runtime environments, and database clients were installed to the global zone and
inherited by the non-global zones. Thus, application software is shared both logically and physically by
the non-global zones.
● Upgradeability / Availability- A major benefit to having the zones all co-located on the primary boot disk
was to facilitate the upgrade process. Because of the number of configured zones, the upgrade
process was fairly lengthy – for every zone, every patch must be inspected for appropriateness and
install-ability. Live Upgrade was not supported until Update 4 and downtime needed to be minimized
because of the large community of users. Since all the zones and application software were co-
located on the primary boot disk, the disk mirror was taken offline, removed, inserted into another
system and upgraded over the course of 2 days (the upgrade didn't take that long but additional time
was needed to work through peripheral complications which required manual interventions). Once the
disk was upgraded and replaced back, it was booted as an alternate boot disk. Once everything was
deemed satisfactory, a mirror sync was initiated as a background task with the newly upgraded disk as
the primary disk. The end result was that an upgrade was performed, in the absence of Update 4 Live
Upgrade support, with minimal downtime.
In this particular situation, it worked out well that all the zone roots were co-located on the primary boot disk.
Obviously, I/O contention has to be carefully monitored. For this situation, once extraneous (all non application
and system binary services) I/O was relocated to external storage, the I/O workload for application and zone
servicing was well under control.
9 Lessons Learned and Best Practice Considerations
Shared Binaries – As discussed previously in Zone Root Location Efficiencies above, sharing application
installations leads to a reduction in storage footprint and complexity of administration in having to maintain
multiple copies. Its worth noting separately that additional efficiency is gained because text code segments
can be shared at the kernel level if they are loaded from the same source resulting in a smaller kernel memory
footprint.
Memory Requirements - In additional to the cumulative memory requirements for all applications running
in a given zone, each zone will consume roughly 40-64MB of memory for zone management. There is little to
no performance impact for applications running in a zone (<=1%).
Resource Management – Solaris 10 includes a powerful hierarchy of options to provide resource
management at the CPU, memory and network level. Resources can be controlled at the higher level global
zone level as well as within a non-global zone.
From the global zone level, resources can be allocated to non-global zones through dynamic resource pools.
Solaris 10, Update 4(8/07) introduced powerful enhancements to this already rich set of dynamic resource
allocation feature set. Update 4 introduced:
● Exclusive-IP zones (default is shared-IP) which provide full IP-level functionality in which zone
assigned interfaces can be fully plumbed, modified via ifconfig and configured with IP Network
Multipathing (IPMP) and IP routing.
● Zone based memory capping – While previous releases supported memory resource allocation at a
project level (via rcapd daemon running in the non-global zones), rcapd(1M) running at the global
zone level is now supported. rcapstat(1M) is now zone aware and can report attempted and actual
mount of memory paged out. The ability to cap swap space is also new to Update 4.
● Support for locked memory – tagging memory that is not eligible for paging. This attribute affects
Intimate Shared Memory (ISM) as well as mmap(2). SAS primarily uses mmap(2) for memory
allocation.
● For more information on “What's New” in Update 4 non-global zone configuration, visit:
http://docs.sun.com/app/docs/doc/817-1592/6mhahuooj?l=en&a
In summary, at the global zone level, resources can be dynamically allocated to given containers via CPU
poolsets, CPU shares (through the Fair Share Scheduler-FSS) and via memory capping (physical, swap,
locked). Within a container, the same directives can be used as well as imposing user or application CPU time
limits, file or core size limits, or limitating the stack, heap or virtual memory of a process. Fine grained controls
within a non-global zone are supported at a project and task level. For more information, visit:
“Understanding the Basics About Solaris Containers in the Solaris 10 OS”
http://www.sun.com/bigadmin/features/articles/basics_containers.html#controlling
“Configuring Resource Controls and Attributes”
http://docs.sun.com/app/docs/doc/817-1592/6mhahuoiq?a=view
While resource allocation capabilities are very powerful, it is also very easy to configure poorly and make the
system unusable. In general, the Solaris 10 defaults, out of the box, work just fine. It's best to not change
anything unless there is a specific issue that needs to be addressed with some reasonable expectation that the
10 Lessons Learned and Best Practice Considerations
change will address the specific problem at hand. Administrators who work with other UNIX vendors are often
used to having to modify kernel parameters and feel that they are missing something if there are none to
modify. In general, with Solaris 10, it's just not necessary. If changes are made, it's recommended to start with
wide granularity and add small enhancements at a time. For the most part, changes can be made dynamically
without any system disruption, so plan and architect a change management system which allows time for true,
peak load usage to provide system feedback. Don't introduce many changes unless system performance
heuristics are well monitored and understood at the global zone level. For instance, increasing locked memory
for one zone will take away from memory generally available for the global zone and other non-global zones.
In the spirit of 'let the kernel do its job in the absence of anything specific', one example comes to mind for
discussion. CPU processor sets can be assigned to a container through the use of poolcfg(1M). The
specifications allow for primitives which define a range of CPU resources (i.e.: min 2, max 4) or, alternatively,
to dedicate a specific number of CPUs. In the example specification of min 2/max 4, 2 CPUs will be allocated
at a minimum and if utilization warrants, a maximum of could be used. If the container is idle, the 2 CPUs
which are on standby can be utilized elsewhere until called back into action to its originally assigned zone.
Sometimes administrators will want to allocate a fixed number of CPUs to a given container (i.e.: min 6, max 6
for SAS Workspace Server). Unless there are very strict service level agreements, don't restrict the kernel
from utilizing idle CPU resources.
Lastly, if there are system performance issues and a support call to Sun will be placed, if possible, try to run
without any resource controls in place to see whether that helps or hurts the performance situation. Support
personnel will likely want to know whether resource controls are helping or hindering the issue at hand.
The Role of ZFS – The ZFS file system and Zones work well together and allow for great configuration
flexibility. From the global zone, a ZFS file system can be configured for use:
● exclusively specific to a zone
● shared between non-global zones
● shared with the global zone
For more information, visit:
“Using ZFS on a Solaris System With Zones Installed”
http://docs.sun.com/app/docs/doc/819-5461/gbdaw?l=en&a=view&q=zones
ZFS has simple administration interfaces and is a high performance file system. As an anecdotal point, SAS
R&D saw an OLAP cube build time that went from 20 minutes to 3 minutes by moving to a ZFS filesystem.
While other similar performance gains have been realized, be cautious when generalizing or extrapolating the
results as the expression goes “your mileage may vary”.
ZFS provides snapshot/cloning features which can yield tremendous savings in both cost, administrative
hassle, and downtime. ZFS clones and snapshots can be a key architectural component for SAS application
backup and recovery strategies. While many storage HW and SW vendors provide hot backup solutions –
data integrity at the storage, LUN, volume or file system level, relatively few software applications are backup
aware and must be halted or quiesced in order for backups to be taken with guaranteed integrity. SAS 9.1.3
and 9.2 applications fall into this category; SAS services need to be halted in order for application data
backup. SAS R&D has used ZFS snapshots and clones to improve backup down time from 39 hours down to
11 Lessons Learned and Best Practice Considerations
15 minutes. For further information on SAS 9 Backup stratgies and using ZFS snapshots/clones in this
manner, visit:
“Backup and Disaster Recovery: When Disaster Strikes, What Will You Do?” - SAS Global Forum 2008
paper 312, Arthur Hunt, Tanya Kalich, Billy Dickerson, Chuck Antle.
While not related to the PRDSHARE project, its notable to mention how the Center for Disease Control used
ZFS clones to solve a tough IT problem; their nightly SAS SPD server update was taking too long and the data
warehouse was not available for use during this time. Since an SPD Server data store cannot be replicated on
the same system, the only alternative available was a complete server and storage replication – a costly and
very lengthy time-to-solution option. However, using ZFS clones, married to Zones, a solution was devised and
went from idea to proof of concept in a few hours and at no additional HW or SW cost. For further information,
visit:
“Zebra, Zamboni, Zen and the Art of ZFS” – SAS Global Forum 2007 Paper, Presentation
http://www.sas.com/partners/directory/sun/sas9-and-zfs.pdf
http://www.sas.com/partners/directory/sun/zebra-preso.pdf
Solaris Fingerprint Database (sfpDB) - This is not related to virtualization per se but worth a mention
under any kind of Best Practices discussion especially since its not a well known utility/service. The Solaris
Fingerprint Database is a new Sun service offering that enables you to verify the integrity of files distributed
with the Solaris Operating Environment (for example, the /bin/su executable file), Solaris patches, and
unbundled products such as the Sun Studio compilers. You can use Solaris Fingerprint Database to verify that
you are using a true file in an official binary distribution, and not an altered version that compromises system
security and causes other types of problems. SfpDB was very helpful in the PRDSHARE environment – due
to some HW failures in conjunction with many application and patch installs/un-installs, there was uncertainty
as to the integrity of some of the system files. Using sfpDB, the administrators were able to verify that all the
system files were validated to the appropriate release and not corrupted. This provided great confidence that
the current system installation integrity was as expected and that the upgrade process could proceed as
planned. For further information, visit:
“Solaris Fingerprint Database: An Identification Tool for Solaris Software and Files“
http://sunsolve.sun.com/show.do?target=content/content7
SummaryThe project requirements presented a daunting challenge - a large and diverse population of global users with
conflicting and contradictory goals: support for development, test and production environments on two
completely separate release tracks. To that end, the PRDSHARE project has met with great success; complex
appliation environments have been provisioned onto a single system maximizing server utilization. Product
teams and developers worldwide are able to focus on their tasks at hand without having to tackle infrastructure
concerns. The alternative would have required several servers and additional storage to have supported the
requirements. Deployment using Containers is administratively simple and straighforward, poses virtually no
performance overhead yet provides for great configuration flexibility via resource management, performance
monitoring, dynamic modifications, etc. Economies of scale, efficiency, increased utilization, lower costs and
agility are key benefits for a virtualization strategy for SAS deployments using Solaris 10 Containers.
12 Acknowledgments
AcknowledgmentsThanks to SAS R&D Staff responsible for the PRDSHARE architecture, deployment and support: Billy
Dickerson, Chuck Antle, Arthur Hunt, Kim Sherrill, Mark Everett, Gary Hutchison, Tim Carter – for their vision,
persistence, support, discipline and patience. Also, thanks to Phil Kinslow and the many other colleagues at
Sun who developed documentation, case studies, and papers on Solaris 10 Container technologies. Lastly,
thanks to Jason Schroeder, Greg Smith, Margaret Crevar and the many others who have answered questions
and provided feedback and review.
ReferencesUnderstanding the Basics About Solaris Containers in the Solaris 10 OS
http://www.sun.com/bigadmin/features/articles/basics_containers.html
Deploying SAS(R)9 Business Intelligence Server using SolarisTM 10 Containers, Phil Kinslow
System Administration Guide: Solaris Containers- Resource Management and Solaris Zones
http://docs.sun.com/app/docs/doc/817-1592?l=en
Using ZFS on a Solaris System With Zones Installed
http://docs.sun.com/app/docs/doc/819-5461/gbdaw?l=en&a=view&q=zones
Solaris 10 - System Administrator Collection -> IP Services -> IP Quality of Service (IPQoS)
http://docs.sun.com/app/docs/doc/816-4554/ipqos-config-planning-tbl-28?l=en&a=view
Solaris Operating System - Feature FAQs
http://www.sun.com/software/solaris/faq.jsp
Zones and Containers FAQ at OpenSolaris.org
http://opensolaris.org/os/community/zones/faq/
Solaris Operating System - Move a Solaris Containers How To Guide
http://www.sun.com/software/solaris/howtoguides/moving_containers.jsp
Bringing your Application into the Zone
http://developers.sun.com/solaris/articles/application_in_zone.html
Third Party Software Requirements for use with SAS Products
http://support.sas.com/resources/thirdpartysupport/v913sp4/index.html
How to Get Started with IBM WebSphere Application Server on Solaris 10 and Zones
http://www.sun.com/software/whitepapers/solaris10/websphere6_sol10.pdf
WebSphere Application Server V6.0.2 detailed system requirements
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27007256
13 References
8.1 Supported Configurations: Sun Solaris 10 on SPARC
http://e-docs.bea.com/platform/suppconfigs/configs81/solaris10_sparc/81sp4.html
Backup and Disaster Recovery: When Disaster Strikes, What Will You Do? - SAS Global Forum 2008
Arthur Hunt, Tanya Kalich, Billy Dickerson, Chuck Antle
Zebra, Zamboni, Zen and the Art of ZFS – SAS Global Forum 2007 paper, presentation
http://www.sas.com/partners/directory/sun/sas9-and-zfs.pdf
http://www.sas.com/partners/directory/sun/zebra-preso.pdf
Solaris Fingerprint Database: An Identification Tool for Solaris Software and Files
http://sunsolve.sun.com/show.do?target=content/content7
Appendix A - Introduction to Solaris 10 Containers1
In the Solaris 10 OS, Containers are an exciting, new partitioning technology allowing multiple virtualized OS
instances on a single system. Solaris Containers can be built using one or more the following technologies.
These technologies can be combined to create Containers tailored for a specific server consolidation project.
● Solaris Resource Manager, for workload resource management
● Resource Pools, for partitioning
● Zones, for isolation, security and virtualization
While Solaris Zones (“Zones”) provide a security boundary and virtualization of the underlying platform
providing a separate name space, IP addresses, file systems, unique root user, password file, and name
server, it is Solaris Resource Management that establishes the resource consumption boundary and allows the
and allows the system resources, such as processors and memory, to be allocated to each Solaris Zone.
It is important to note the terminology of Solaris Container and Zone. The terms are often used interchangeably
but that's technically incorrect. Solaris Containers are a combination of Resource Management, System
Administration (customization) and Zones. Zones is one of the technology that enables Containers. Zones
technology can be used to create a Container with certain characteristics, such as the isolation provided by the
virtual Solaris environment. But it is also possible to create another Solaris Container, without Zones, using
Resource Pools technology if the required characteristics of that Container can be met with the features
Resource Pools provide. So while a Zone is a Container, a Container is not necessarily a Zone. Note: Solaris
Resource Manager and Resource Pools technologies existed before Solaris 10. Zones are new in Solaris 10.
1 Kumar, Leigh “How to Get Started with IBM Websphere Application Server on Solaris 10 and Zones”
14 Appendix A - Introduction to Solaris 10 Containers1
Figure 2: Conceptual view: Single server using Solaris 10 Containers provides virtual logical environments
Zones are classified as either a global zone or non-global zone. The global zone encompasses the entire
system. Because it is equivalent to a typical Solaris OS instance, the global zone has access to the physical
hardware, and can control all processes. Non-global zones are located inside the global zone, and are isolated
from the physical characteristics of the system. For any given system, there is exactly one global zone but 0, 1
or many non-global zones.
Solaris Zones provide the following features:
● Security—Network services can be run in a zone, limiting the damage that can be done to the system
and other zones in the event of a security violation.
● Isolation—Applications requiring exclusive access to global resources, such as specific usernames or
network ports, can run on the same machine using Solaris Zones. Each zone has its own namespace,
completely separate from other zones. Users in a zone are unable to monitor other zones, such as
viewing network traffic or the activity of processes.
● virtualization—Solaris Zones present a virtualized environment to applications, removing the physical
details of the hardware from view. This eases redeployment of applications on a different physical
machine.
● Granularity—Since Solaris Zones are implemented in software, zones are not limited to the granularity
defined by hardware boundaries. Instead, zones offer sub-CPU granularity. Zones do not require
dedicated CPU resources, dedicated I/O devices such as host bus adapters and network interface
cards, or dedicated physical memory. As a result, even a system with a single processor can be used
to host several zones.
● Transparency—The environment presented to the application in a zone is nearly identical to the
standard Solaris OS environment. There are no new, zone-specific application programming interfaces
15 Appendix A - Introduction to Solaris 10 Containers1
(APIs) or application binary interfaces (ABIs) to which applications must be ported. Some restrictions
do exist due to security and isolation requirements. These restrictions mainly affect applications that
perform privileged operations or need access to physical devices
Appendix B – Container Creation Cookbook2 This appendix contains a cookbook sample to demonstrate the ease of
which Solaris Containers can be setup for a sample 3 tiered SAS
installation.
Solaris Containers and SAS EBI ServerSince the SAS EBI Server is a prototypical three tier architecture,
Solaris Containers allow customers to easily and effectively deploy SAS
BI Server. The SAS BI Server will be deployed with three functional
areas – the SAS Metadata Server, the SAS Web Application Server
(BEA Weblogic, Xythos and SAS Remote Services),
and the SAS Application Server (OLAP Server, Stored Process,
Workspace and Object Spawner). Figure 3 illustrates a sample SAS
BI Server architecture. SAS BI Server's modular architecture lends to
having three separate Solaris Containers to support this sample deployment. In this case, a single server can
be partitioned using Solaris Containers to create a development, test and production environment for SAS BI
Server. Since each Container has its own port address space, each of the multiple SAS deployments can use
the default ports. For example, the SAS Metadata Server for production, test, and development can all use
port 8561.
Creating a Solaris ContainerCreating a Solaris Container is easy. Recall that a Solaris Container is a combination of two technologies -
Resource Management and Solaris Zones, a step-by-step installation is detailed below. Notes: Solaris
Resource Management and Zones are useful independent of each other, but they are particularly powerful and
useful when used in combination. There is no required order to how Sections 1 and 2 are completed; they
could be interchanged.
Section 1: Resource management
The three BI Server components have differing resource requirements. In this section the resources, in this
case processors, will combined into a resource pool. When this resource pool is created, it will be allocated
to the zone (in section 2). The processors added to the metadata pool will only be used by the Container
that is running the SAS Metadata Server. While the Solaris kernel will limit memory allocation for all the
Containers, the initial release of Solaris 10 does not support memory limits for individual Containers. This
feature is scheduled for Solaris 10, Update 1.
Three steps are required to create a resource pool. The steps are: 1) enable, 2) configure, and 3)
instantiate. Table 1 shows these steps. In this case, two of the available twenty-four cores were allocated
for the SAS Metadata Server, four two for the SAS Web Application Server, and six for the SAS Application
Servers.
2 Phil Kinsow, 2006, “Deploying SAS®9 Business Intelligence Server using SolarisTM 10 Containers”
16 Creating a Solaris Container
Enable the resource management subsystembash# pooladm -e
Configure the resource pool for the Container using the contents of file createSASpools.bash# poolcfg -f createSASpools<<file contents of createSASpools>>create system ctclavacreate pset metadata (uint pset.max = 2)create pool metadata (string pool.scheduler="TS")associate pool metadata (pset metadata)create pset midtier (uint pset.max = 4)create pool midtier (string pool.scheduler="TS")associate pool midtier (pset midtier)create pset sasFS (uint pset.max = 6)create pool sasFS (string pool.scheduler="TS")associate pool sasFS (pset sasFS)
Instantiate the configuration and creating the resource poolbash# pooladm -c
Table 1: Limiting resource consumption of the Container by using Solaris Resource Manager
To confirm the creation and to show the configuration of the resource pool, the poolstat command is used. Note the resource pools name, the number of processors in that pool and the pool’s utilization metrics. As Table 2 shows, the system has four resource pools defined – the pool_default, metadata, midtier, and sasFS. As expected, the total of the size column is 24 processors. If the SAS workload demands more resources, more can be added to the pools at any time.
bash# poolstat pset id pool size used load 0 pool_default 12 0.00 0.17 1 metadata 2 0.00 0.00 2 midtier 4 0.00 0.00 3 sasFS 6 0.00 0.00
Table 2: Verifying the creation of the resource pools
Section 2: Solaris Zones
With the resource management complete and the resource pools created, the Zone creation requires only
four steps; they are: 1). configuring, 2). installing, 3) initializing, and 4) booting as represented in Table 3
below. The first step of creating a Zone is the most complicated and requires some planning. For this
exercise, the key Zone specifications are defined as follows:
– set zonepath=/zones/midtier: the "home" of the Container for SAS Middle Tier
– set pool=midtier: assigning the resource pool to the Zone configuration.
– set autoboot=true: the Container is automatically started a system boot time.
– add net: this section adds a network interface and IP address for the Container.
– add fs: this section adds additional file system to the Container. By default, a single a
filesystem is added to the Container. If additional are required, they are added using this
syntax.
17 Creating a Solaris Container
Create the Container with the configuration specifications listed in file midtier-create.bash# zonecfg -z midtier -f midtier-create<<file contents of midtier-create>>createset zonepath=/zones/midtierset pool=midtierset autoboot=trueadd netset physical=ce0set address=192.168.30.29endadd inherit-pkg-dirset dir=/optendadd fsset dir=/apps/sas set type=lofsadd options [rw,nodevices]set special=/san/zones/fs1 endverifycommit
Installing the Container.bash# zoneadm -z midtier installPreparing to install zone <midtier>.Creating list of files to copy from the global zone.Copying <3114> files to the zone.Initializing zone product registry.Determining zone package initialization order.Preparing to initialize <1125> packages on the zone.Initialized <1125> packages on zone.Zone <midtier> is initialized.The file </zones/midtier/root/var/sadm/system/logs/install_log> contains a log of the zone installation.
Initializing the Container.bash# cp sysidcfg /zones/midtier/root/etc/sysidcfgbash# cat sysidcfgsystem_locale=Cterminal=xtermnetwork_interface=primary { hostname=midtier ip_address=192.168.30.29}security_policy=NONEname_service=NONEtimezone=US/Easternroot_password=dVaKf.jNJSeLE
Boot the Container.bash# zoneadm -z midtier boot
Table 3: Configuring, installing, initializing and booting the Container
Validating the ProgressBefore SAS is installed, it is worthwhile to validate the progress so far and review the environment that SAS
will be installed. Table 4 shows the output of the several commands.
List all the Containers on the system.bash# zoneadm list -cv
18 Validating the Progress
ID NAME STATUS PATH 0 global running / 1 sasFS running /zones/sasFS 2 midtier running /zones/midtier 3 metadata running /zones/metadata
Check the resources assigned to each Container (as view from the Global Zone).bash# poolstat pset id pool size used load 0 pool_default 12 0.00 0.17 1 metadata 2 0.00 0.00 2 midtier 4 0.00 0.00 3 sasFS 6 0.00 0.00
Login to a Container.bash# zlogin -C sasFSsasFS console login: rootPassword:******
Check the resources assigned to this Container (as view from inside the sasFS Container).bash# psrinfo0 on-line since 10/17/2005 10:58:261 on-line since 10/17/2005 10:58:262 on-line since 10/17/2005 10:58:263 on-line since 10/17/2005 10:58:264 on-line since 10/17/2005 10:58:265 on-line since 10/17/2005 10:58:26
bash# poolstat pset id pool size used load 3 sasFS 6 0.00 0.00
Table 4: Displaying the status of Solaris Containers
Each container has its own name space, IP addresses, file systems, unique root user, and password file. Additionally, the resource
consumption boundary is also guaranteed to ensure that each part of the SAS BI Server is guaranteed to have access to the system
resources, such as processors and memory, which were assigned to it. Since the sasFS Container is bound to the sasFS pool which
has a defined limit of six processors, the SAS Application Server can only use the six processors assigned to this Container. At this
point, the SAS installation can proceed as planned.