<Insert Picture Here>

Oracle Solaris Innovative VirtualizationZones and NetworkingChenlu ChenChen-lu.chen@oracle.comSolaris Networking development TeamJuly 2012

3

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

4

Zones Overview

5

OS Virtualization● Provides separate virtual environments

● with performance, scale and observability● without virtual machine overhead

● Appearance of many OS instances, not many machines● Single kernel● Isolated namespaces for: IP addresses and port range, process lists

and authentication (file, NIS, LDAP,...), ZFS, NFS services, and many more

● Separate security, resource management, and failure scopes● Can boot, reboot a zone, run SMF services● Customers typically run 2 – 200 zones per machine● Can create a new zone in minutes or seconds if cloning

6

OS Virtualization (2)● Secure

● Excellent track record● Audit available globally and/or per zone● Zones always run with reduced privilege

● Mature, widely adopted, in production at many sites● Part of Solaris 10 FCS, continuously enhanced● Runs everywhere Solaris runs (x86, SPARC sun4u & sun4v)● Deployed on 25% - 33% of Solaris 10 installed base (2009)

● Strong Ecosystem● Oracle Solaris Cluster Support● OEM Ops Center 2.5● Trusted Extensions● Clearcase (in-kernel) Support● Good adoption among management vendors

7

Terminology● Global Zone

● Installed directly on bare metal or into a virtual machine that emulates bare metal.

● Every Solaris 10 and Solaris 11 installation has exactly one global zone.● Non-Global Zone

● Virtualized environment● May be 0 – 1000's of non-global zones running on each global zone● a.k.a. zone, local zone, container

● Branded Zone● A zone that (sometimes) emulates a different operating system

● Solaris 10 Zone● A branded zone used to run Solaris 10's user space on Solaris 11

● Solaris Legacy Containers● Branded zones used to run Solaris 8 and Solaris 9 user space on

Solaris 10

8

Solaris 11 Zones

9

Zones for Solaris 11: Big Ideas

● Rationalized installation, system configuration, and update● Greatly improved p2v● Greater runtime “completeness”: NFS server, lofi, and more● Networking

● Shared Stack → Exclusive Stack● Automatic networking● Network Resource Management

● Immutable Zones● Storage enhancements● Accurate Utilization Monitoring● Solaris 10 Zones

10

Zone Installation

● Installation methods:● “Vanilla”: Accept the defaults● Using an AI manifest and/or SC profile● From a directory● From an archive (cpio, pax, zfs, optionally gz or bz2ed)● Via cloning (zoneadm clone)

●See solaris(5) which explains this in great detail●Zone can be provisioned by AI at initial system install

● Self assembly service svc:/system/zones-install stitches things together at first boot

11

Zone Installation (2)

● Zones support for Automated Install● zoneadm(1m) install uses AI to install zones.● When installing a zone, specify an AI manifest to describe the zone and its

contents via -m:

● Zones support for System Configuration:● When installing, cloning, or attaching a zone, specify a configuration profile via

-c:

# zoneadm -z z1 install -m /data/ai_manifest.xml...

# zoneadm -z z2 clone -c /data/sc_profile.xml z1...

12

Zone Installation via p2v

● p2v supported from bare-metal Solaris 11 and Solaris 10

● New zonep2vchk(1m) p2v migration aid● Designed to help admins with the p2v migration

● Solaris 10 GZ → Solaris 10 "native" NGZ● Solaris 10 GZ → Solaris 11 "solaris10" NGZ● Solaris 11 GZ → Solaris 11 “solaris” NGZ

● Can analyze a systems configuration and workload to evaluate how it will work in a zone

● Can generate a zonecfg(1m) to facilitate migration. This includes auto-generation of anet so datalink and networking configuration can migrated seamlessly

13

Solaris 10 Zones on Solaris 11● solaris10 branded zone

● Similar to existing solaris8 and solaris9 brands on Solaris 10● Adoption and compatibility aid for Solaris 11

● Protect investment in Solaris 10 (infrastructure, training, support)● Leverage new technology in an Solaris 10 context

● e.g. Crossbow for Solaris 10● Avoid required application recertification

● p2v installation process● Also v2v for moving Solaris 10 native zones

● Supports Solaris 10 10/09 or later within the zone

● Legacy Brands (Supported by Solaris 10 only)● solaris8 & solaris9: Part of Oracle Solaris Legacy Containers● lx: Linux 2.4 environment

14

Software Management

● ZFS Based Boot Environments● Speed, safety, simplicity

● New Packaging System (IPS)● Modern, network-centric model● Automatic dependency resolution, rich dependency types● “Live Upgrade”, Patching built in (patches == packages!)● Enterprise functionality and security designed in● No scripting → Safety. Decouples installation & config● SVr4 packaging still available for add-ons● Sane package names● Leads with the best practice, by default

15

Zones & IPS

● Linked Images● Global zone operations keep all zones in sync● Non-global zone pkg operations maintain synchronization● ‘pkg update’ in global zone updates all zones (recursive)● Other operations are not recursive, by synchronization always maintained

● System Repository and Proxy● Zones inherit publishers from global zone: no need to manage publishers or

key material in zones for system publishers● Proxy transport to global zone allows zones to install pkgs regardless of

network configuration

16

Zones and Boot Environments

● BE management via beadm(1m) is supported

● BE layout in NGZ and GZ is the same

● Upgrade procedure for Solaris 11 Express zones

● A non-global zone BE is a known as a ZBE

● Each ZBE has a single global BE parent

● ZBEs associated with inactive GZ BEs are not bootable

solaris-1

solaris-2

zbe-1

zbe-2

zbe-3

zbe-4

zbe-5

redzone

clone

clone

global clone

clone

clone

global clone

solaris-3active

17

Sparse Root Zones

● Bet: Modern systems obviate need for space savings associated with sparse root zones

● Goal: Converge to “whole root” zones while preserving useful properties of “sparse root” zones

● Payoff:● Dramatic reduction in complexity● Reduced customer decision making● ISVs have a single target● Greater customer flexibility inside of the zone

● More on security tradeoffs later

18

Zones & Minimization

● Newly installed zones are minimized using:● Installation of pkg:/group/solaris-small-server (by default)

● Use of group pkg means further minimization possible● Minimization related bugs can be fixed in patches!

● ZFS compression enabled for ZBEs● Core language/locale facets enabled (can be disabled to further minimize)

● User or enterprise tools can add additional software without involving global zone admin

● IPS allows minimization of global zones (relative to their non-global zones) with ease

19

Immutable Zones (aka ROZR)

● Improved security versus any other Solaris runtime● Threats: Attackers, “Cowboy” admins● Creates stronger operational hygiene

● Policy enforcement of pathname white- and black-lists● Configured with file-mac-profile zonecfg(1m) property:

● none: default profile for normal zones● strict: all file systems are read-only● fixed-configuration: limited areas in /var are writable (mainly for logging purposes).● flexible-configuration: like fixed but /etc is also writable

● Zone can be booted writable (-w) for maintenance

# zonecfg -z z1zonecfg:z1> set file-mac-profile=fixed-configuration

20

Runtime Completeness

● Zones as NFS servers● NFSv4/3/2 are all supported● No support for HA-NFS failover, NFS/RDMA or Solaris 10 branded zones (yet)

● Zones Infiniband Support● SDP support, IPoIB, EoIB, Open Fabrics User Verbs● Full support Zones on Exalogic● Trajectory: support for all of IB in zones (RDSv3)

● Zones “clean” shutdown (zoneadm -z myzone shutdown)● Robustness enhancements for more reliable zone destruction (sometimes

called “defunct zones”).

21

Storage Enhancements

● lofiadm(1m) support in a zone● System-wide limit raised from 128 to 65536● zone.max-lofi resource control can be set

● Storage device support● Add block/raw storage devices to a zone without reducing system security● allow-partition: enables format, fdisk● allow-raw-io: enables raw SCSI commands via uscsi(7I)● Documentation presently missing in zonecfg(1m): bug filed

● Create zvols in a zone● ZFS dataset aliasing

● Zones now see only a portion of the dataset hierarchy● Ex: zones/myzone/rpool is visible in zone as rpool● Aliasing is automatic but can be overridden in zonecfg

22

Networking: Exclusive Stacks

● Exclusive stack is the default● Brings ipmp, ipfilter, snoop, ipsec, etcetera to zones

● Added shared-stack style controls● New allowed-address property constrains which IP addresses zone can

use (via in-kernel L2/L3 protection)● defrouter property now supported for exclusive-IP zones

●anet resources greatly simplify exclusive stack configuration● Each zone has one anet by default● Dynamic vnic configuration● Resource management

23

Observability: zonestat(1m)

● CLI and Ops Center integration● Use extended accounting for accuracy● Utilization against configured limits ● Drill down, sort, report in many ways● Networking stats for exclusive stacks only

$ zonestat 5...SUMMARY Cpus/Online: 8/8 PhysMem: 11.9G VirtMem: 28.0G ---CPU---- --PhysMem-- --VirtMem-- --PhysNet-- ZONE USED %PART USED %USED USED %USED PBYTE %PUSE [total] 1.40 17.5% 10.6G 88.9% 12.1G 43.2% 3.8M 3.50% [system] 0.10 1.26% 8546M 69.6% 8639M 30.1% - - global 0.29 4.26% 2356M 19.1% 3715M 12.9% 493K 0.49% z1 0.00 0.06% 14.1M 0.11% 43.8M 0.15% 0 0.00% z2 1.00 100% 24.3M 0.18% 73.2M 0.28% 3.2M 3.00%

24

Introducing zonestat(1m)

● zonestatd Daemon performs monitoring● Allows non-root users and non-global zones to see (some of) the information

● Zonestat can monitor:● virtual-memory, physical-memory, locked-memory, pool-psets, lwps, processes, shm-

memory, network, and more● Limit output to specific zones● Sort by various columns● End-of-run reporting for average, high, total usage.● Drill down by resource type● Machine parsable output mode & Public C API

25

Zonestat examples● Example: Monitor lwps & processes:

$ zonestat -r processes,lwps 5PROCESSES SYSTEM LIMITsystem-limit 292K ZONE USED PCT CAP %CAP [total] 191 0.63% - - [system] 0 0.00% - - global 167 0.55% - - foo 24 0.08% 300 8.00%

LWPS SYSTEM LIMITsystem-limit 2047M ZONE USED PCT CAP %CAP [total] 713 0.00% - - [system] 0 0.00% - - global 618 0.00% - - foo 95 0.00% 1000 9.50%

26

Resource Management

● New max-processes resource controlexample# zonecfg -z foozonecfg:foo> set max-processes=300

● prctl(1) now shows resource utilization:example# prctl -i zone foozone: 4: fooNAME PRIVILEGE VALUE FLAG ACTIONzone.max-lofi usage 0 system 18.4E max deny zone.max-swap usage 28.3MB privileged 3.00GB - deny system 16.0EB max deny

27

Zones Security

● Delegated administration (via RBAC authorizations)● Authorizations can be configured directly in zonecfg(1m):example# zonecfg -z myzonezonecfg:myzone> add adminzonecfg:myzone:admin> set user=jackzonecfg:myzone:admin> set auths=login,managezonecfg:myzone:admin> endzonecfg:myzone> commit

● Authorizations are implemented via /etc/user_attr and synced there by zonecfg.

28http://www.flickr.com/photos/clonedmilkmen/3604999084

NetworkVs.Virtualization

The Usual Scenario

29

Oracle Solaris 11 Network Virtualization

• Built-in, no cost,• Virtual NIC• Virtual switch

• Build out and controlnetwork access for applications• Dedicated: Just like the real thing,

but for each application• Integrated: Tightly joined with Solaris virtualization• Empowering: control the previously uncontrollable• Efficient: optimized for high and low loads• Observable: see and react to issues in real time

• What’s New?

30© 2010 Oracle Corporation

Network Virtualization Hardware Lane

• Linear Scalability by using Hardware Lanes with dedicated resources• Network Virtualization and QoS designed in the stack• More Efficiency due to dynamic polling and packet chaining

31

# dladm create-vnic -l bge1 vnic1# dladm create-vnic -l bge1 -m random -p maxbw=100M -p cpus=4,5,6 vnic2# dladm create-etherstub vswitch1# dladm show-etherstubLINKvswitch1# dladm create-vnic -l vswitch1 -p maxbw=1000M vnic3# dladm show-vnicLINK OVER MACTYPE MACVALUE BANDWIDTH CPUSvnic1 bge1 factory 0:1:2:3:4:5 - -vnic2 bge1 random 2:5:6:7:8:9 max=100M 4,5,6vnic3 vswitch1 random 4:3:4:7:0:1 max=1000M -# dladm create-vnic -l ixgbe0 -v 1055 -p maxbw=500M -p cpus=1,2 vnic9

Virtual NIC & Virtual Switch Usage

32© 2010 Oracle Corporation

Solaris 11 Network Flows• Networking flows based on the following attributes– Services (protocol + remote/local ports)– Transport (TCP, UDP, SCTP, iSCSI, etc)– IP addresses and IP subnets– DSCP labels

• Each flow can be assigned its own bandwidth limit• Flows maintain their own statistics– Displayed in real-time through flowstat(1M)– History through extended accounting

# flowadm create-flow -l bge0 protocol=tcp,local_port=443 -p maxbw=50M http-1# flowadm set-flowprop -l bge0 -p maxbw=100M http-1

33© 2010 Oracle Corporation

Data Link (L2/L3) Protection

• Need to protect against spoofing and harmful packets injected by VM guests and non-global zones on a shared network

• New properties allow global zone to check transmitted packets• Configured using data link property in dladm– MAC address anti-spoofing– IP address anti-spoofing– DHCP client ID anti-spoofing– Restrict protocols (IPv4, IPv6, ARP)

34© 2010 Oracle Corporation

Solaris 11 Network Observability

• Observability at the IP layer with traditional tools (for example snoop, wireshark)– It is now possible to observe

• Loopback traffic• Traffic between shared-IP zones

– For example, snoop -I lo0• DTrace provider for IP– In and out packets; dropped packets

• tcpdump and wireshark as IPS packages• Observe IPMP using ipmpstat• Observe Network virtualization using dlstat and flowstat

35

Resources

● Oracle Solaris Virtualization● http://www.oracle.com/technetwork/server-storage/solaris/overview/virtualization-163570.html

● Oracle Solaris Zones ● http://www.oracle.com/technetwork/server-storage/solaris10/containers-169727.html

● Oracle Network Virtualization● http://www.oracle.com/technetwork/server-storage/solaris11/technologies/networkvirtualization-

312278.html● zones-discuss@opensolaris.org● Solaris Zones forum at http://forums.oracle.com/

● Under Server & Storage Systems / Solaris

<Insert Picture Here>

Questions