HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Security Level:
Slide title :40-47pt
Slide subtitle :26-30pt
Color::white
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers and
partners :
Arial
www.huawei.com
Cgroups: Understanding better!
OSI Days, November, 2011, Bangalore
Ashish Pandey, Liuyang, Roopesh Keeppattu, Sanil Kumar D.
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
2
Agenda• Setting the context…
- Background and Cgroups Definition - Significance of Cgroups• Cgroups: Deeper View - Features and Design - Cgroups Subsystems - Cgroups Implementation• Cgroups Use cases - Usage Illustration - Sample Use cases• Cgroups : Challenges and Future - Key Challenges - Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
3
Background Clustering and Aggregation of Resources help
in Fairness in Large Systems Predictability in Performance depends on the
predictability in the availability of resource
Isolation and Aggregation of Resources for specific activities can improve the predictability of the performance for the specific activities. This also helps in Resource Tracking, Control and Management
Examples : Cloud, Virtualization, Containers
Cloud
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
4
Cgroups Definition
Cgroups are like process container or process groups in Linux kernel
Provide a mechanism for aggregating / partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour
Associate a set of tasks with a set of parameters for one or more subsystems
‘Cgroups’ = ‘Control groups’
Cgroups: Process Container
PP
PP
P
P
PP
P
Cgroups helps for resource isolation and resource control
Cgroups help tuning the system to make best use of available hardware and network resources.
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
5
Significance of Cgroups
Mainly 2 types of Subsystems supported:• Isolation and Special Controllers : cpuset, namespace,
freezer, device, checkpoint/restart• Resource Controllers : cpu(scheduler), memory, disk I/O,
network
As the systems are becoming more complex with dynamic resource needs, better resource management is inevitable to get optimized outout.
Resource planning is better than dynamic resource allocation
Cgroups with its tree-structured hierarchy and subsystems achieves resource planning for the native system
The significance of Cgroups is that it helps to implement the essential resource groups hierarchically in kernel to enable better resource management. Every task running in the system is in exactly one of the Cgroups in this hierarchy
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
6
Agenda• Setting the context…
- Background and Cgroups Definition - Significance of Cgroups• Cgroups: Deeper View - Features and Design - Cgroups Subsystems - Cgroups Implementation• Cgroups Use cases - Usage Illustration - Sample Use cases• Cgroups : Challenges and Future - Key Challenges - Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
7
Features and Design
• Isolate the resources for specific similar task groups
• Separate Cpuset, Namespace, Device for groups
• Controls the resource for usage limits, freezing groups or check pointing and restarting
• Freezer, Checkpoint / Restart, Cpuacct, CPU, Memory, Disk I/O and Network I/O
• Resource Tracking to measure how much resources certain systems use for e.g. billing purposes
• State of the resource is tracked
Resource Isolation Resource Control Resource Tracking
task Userspace or kernel process
cgroup One or more tasks
subsystem Module to modify the
behaviour
of the tasks in a cgroup
hierarchy Several cgroups in a tree
Hierarchical based grouping of similar tasks in terms of resources is the design principle behind Cgroups
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
8
Cgroups Subsystems
Modules which makes use of task grouping facilities provided by Cgroups to treat groups of tasks in particular way
Resource controllers allocating different levels of system resources to different Cgroups.
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
9
Cgroups Subsystems DetailsSubsystem Type Description
cpuset Isolation Use cpuset to tie processes to system subsets of CPU
ns Isolation For showing private view of system to processes in cgroup; mainly used for OS level virtualization
cpu Control Shares CPU bandwidth between groups
cpuacct Accounting Groups tasks using cgroup and accounts the CPU usage
memory Control Limits memory and memory related parameters
devices Isolation Decides the access to devices by tasks in cgroup
freezer Control Freezes/unfreezes tasks in a group. Used for HPC clusters
net_cls Control Identification of packets originating from a specific cgroup and actions on the packet.
blk_io Control controls and monitors access to I/O on block devices by tasks in cgroup
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
10
Cgroups Implementation
VFS based filesystem model implementation
Each task in the system has a reference-counted pointer to a css_set
A css_set contains a set of reference-counted pointers to cgroup_subsys_state objects, one for each cgroup subsystem registered in the system
A cgroup hierarchy filesystem can be mounted for browsing and manipulation from user space
You can list all the tasks (by pid) attached to any cgroup [cat /cgroup/tasks OR cat /cgroup/<group>/tasks]
Files: • kernel/cgroup.c• include/linux/cgroup.h• include/linux/cgroup_subsys.h• +various subsystems
implementations cgroups requires a few, simple hooks into
the rest of the kernel, these hooks are not in performance critical paths of the kernel.
• In init/main.c : To initialize C groups at boot time.
• In fork and exit calls : To attach and detach a task from cgroups
A new file system, of type "cgroup" should be mounted, to enable browsing and modifying the all the registered Cgroups subsystem
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
11
Cgroups : How to write a new subsystem
Major modifications to be done: • init/Kconfig: kernel
configuration parameters (general setup)
• include/linux/cgroup_subsys.h: cgroup subsystem definition
• kernel/cgroup_example.c: cgroup subsystem implementation [Implement Kernel APIs as per cgroup_subsys]
• kernel/Makefile: Makefile of the core kernel components
• Add the appropriate hooks into the kernel
APIs to be implemented by a new subsystem
As Cgroups provide clean kernel APIs , new subsystem addition and management are simpler
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
12
Agenda• Setting the context…
- Background and Cgroups Definition - Significance of Cgroups• Cgroups: Deeper View - Features and Design - Cgroups Subsystems - Cgroups Implementation• Cgroups Use cases - Usage Illustration - Sample Use cases• Cgroups : Challenges and Future - Key Challenges - Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
13
mount -t tmpfs none /sys/fs/cgroupmkdir /sys/fs/cgroup/memorymount -t cgroup none /sys/fs/cgroup/memory -o memory#Make the new group and move bash into itmkdir /sys/fs/cgroup/memory/0echo $$ > /sys/fs/cgroup/memory/0/tasks#Since now we're in the 0 cgroup, we can alter the memory limit:echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytescat /sys/fs/cgroup/memory/0/memory.limit_in_bytes4194304#We can check the usage:cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes1216512
Usage Illustration
mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpusetcd /sys/fs/cgroup/cpusetmkdir Charliecd Charlie/bin/echo 2-3 > cpuset.cpus/bin/echo 1 > cpuset.mems/bin/echo $$ > taskssh# The subshell 'sh' is now running in cpuset Charlie# The next line should display '/Charlie'cat /proc/self/cpuset
CPUset SubsystemMemory Subsystem
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
14
mkdir /sys/fs/cgroup/freezermount -t cgroup -ofreezer freezer /sys/fs/cgroup/freezermkdir /sys/fs/cgroup/freezer/0echo $some_pid > /sys/fs/cgroup/freezer/0/tasks#to get status of the freezer subsystem : cat /sys/fs/cgroup/freezer/0/freezer.stateTHAWED#to freeze all tasks in the container :echo FROZEN > /sys/fs/cgroup/freezer/0/freezer.state#at /sys/fs/cgroup/freezer/0/freezer.state FROZEN#to unfreeze all tasks in the container :echo THAWED > /sys/fs/cgroup/freezer/0/freezer.statecat /sys/fs/cgroup/freezer/0/freezer.stateTHAWED
Usage Illustration
#An entry is added using devices.allow, and removed using devices.deny. For #instanceecho 'c 1:3 mr' > /sys/fs/cgroup/1/devices.allow#allows cgroup 1 to read and mknod the device usually known as dev/null.#Doingecho a > /sys/fs/cgroup/1/devices.deny#will remove the default 'a *:* rwm' entry. Doingecho a > /sys/fs/cgroup/1/devices.allow#will add the 'a *:* rwm' entry to the whitelist.
Device SubsystemFreezer Subsystem
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
15
Sample Use casesWithout CPU cgroup subsystem(10 tasks in “multimedia” and 5 tasks in “browser”)
With CPU cgroup subsystem(10 tasks in “multimedia” and 5 tasks in “browser”)
Cgroups can provide better resource control and management
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
16
Sample Use cases (contd.)
Users
Professor
CPU (0-3)
Disk (50%)
Memory(50%) N/W
W W W b ro w s in g (
2 0 % ) NFS(25%)
Student
CPU((4-7)
Disk(50%)
Memory(50%) N/W
W W W b ro w s in g (
1 0 % ) NFS(25%)
A large university server with various users - students, professors, system tasks etc. The resource planning for this server could be along the following lines with cgroups
Consider a batch job management system which requires start and stop sets of tasks in order to schedule the resources of a machine .This kind of program is often used on HPC clusters to schedule access to the cluster as a whole
Batch Processing System
Batch Process1
Cpu(0-4)
Disk(50%)
Memory(50%)
No Freeze
Batch Process2
Cpu(5-8)
Disk(40%)
Memory(30%)
Can be freezed
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
17
Agenda• Setting the context…
- Background and Cgroups Definition - Significance of Cgroups• Cgroups: Deeper View - Features and Design - Cgroups Subsystems - Cgroups Implementation• Cgroups Use cases - Usage Illustration - Sample Use cases• Cgroups : Challenges and Future - Key Challenges - Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
18
Key Challenges
Throughput intensive system, Cgroups may add performance overhead
It's not currently possible to bind a new subsystem to an active cgroup hierarchy, or to unbind a subsystem from an active cgroup hierarchy
Though Cgroups acts like lighter virtualization, the same can be applied only to native system
Large multi-core systems, the linear performance scalability is needed to utilise the resource better
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
19
Future…
Throughput optimization
Virtualization Possibilities and Lower Level Resource Abstraction
Tuning and Development for Large Multi-core systems
Multicore clustering support and Specialized Subsystems
and …shaping as an inevitable component in linux kernel for effective resource control and management…
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011
Top right corner for field-mark, customer or partner logotypes.
----------------
The following nine groups of colors are an example of how our design colors can be used, please take note that you should only use one design color group per slide. For specific usage details, refer to the “Typesetting Standard”.
20
Ashish Pandey ([email protected]) Roopesh Keeppattu ([email protected])
Liuyang([email protected]) Sanil Kumar D. ([email protected])