47
Analyzing and Optimizing Linux Kernel for PostgreSQL Sangwook Kim PGConf.Asia 2017
Analyzing and Optimizing Linux Kernel for PostgreSQL
Sangwook Kim PGConf.Asia 2017
2
Sangwook Kim • Co-founder and CEO @ Apposha • Ph. D. in Computer Science
• Cloud/Virtualization • SMP scheduling [ASPLOS’13, VEE’14] • Group-based memory management [JPDC’14]
• Database/Storage • Non-volatile cache management [USENIX ATC’15, ApSys’16] • Request-centric I/O prioritization [FAST’17, HotStorage’17]
3
Fully-managed & performance-optimized for MySQL and PostgreSQL
for
What We Do
Fully-managed & performance-optimized for MongoDB
No special H/W support No code changes for DB engine
Contents
• Background tasks
• Full page writes
• Parallel query
4
PostgreSQL Architecture
5
Storage Device
Operating System
T1
Client
T2
I/O
T3 T4
Request Response
I/O I/O I/O
PostgreSQL
Database performance
* PostgreSQL’s processes - Backend (foreground)
- Checkpointer
- Autovacuum workers
- WAL writer
- Writer
- …
PostgreSQL Architecture
5
Storage Device
Operating System
T1
Client
T2
I/O
T3 T4
Request Response
I/O I/O I/O
PostgreSQL * PostgreSQL’s processes - Backend (foreground)
- Checkpointer
- Autovacuum workers
- WAL writer
- Writer
- …
PostgreSQL Checkpoint
6
http://www.interdb.jp/pg/pgsql09.html
Impact of Checkpointer
7
0
500
1000
1500
2000
2500
0 500 1000 1500 2000 2500 3000
Max
trx
late
ncy
(ms)
Elapsed time (sec)
Default
1.1 sec
• Dell Poweredge R730 • 32 cores • 132GB DRAM • 1 SAS SSD
• PostgreSQL v9.5.6 • 52GB shared_buf • 10GB max_wal
• TPC-C workload • 50GB dataset • 1 hour run • 50 clients
PostgreSQL Autovacuum
8
Free Space Map
http://bstar36.tistory.com/308
Impact of Autovacuum Workers
9
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
1 50 100 150 200 250 300
Trx
thpu
t (trx
/sec
)
# of clients
Default Aggressive AV
40%
autovacuum_max_workers 3 => 6 autovacuum_naptime 1min => 15s autovacuum_vacuum_threshold 50 => 25 autovacuum_analyze_threshold 50 => 10 autovacuum_vacuum_scale_factor 0.2 => 0.1 autovacuum_analyze_scale_factor 0.1 => 0.05 autovacuum_vacuum_cost_delay 20ms => -1 autovacuum_vacuum_cost_limit -1 => 1000
Impact of Autovacuum Workers
10
0
500
1000
1500
2000
2500
0 500 1000 1500 2000 2500 3000
Max
trx
late
ncy
(ms)
Elapsed time (sec)
Default Aggressive AV 2.5 sec
Why Background Tasks Matter
• Multiple independent layers
11
Storage Device
Caching Layer
Application
File System Layer
Block Layer Abs
tract
ion
Buffer Cache
read() write()
FG FG BG
BG FG BG BG
reorder
admission control
admission control
reorder
BG
admission control
Why Background Tasks Matter
• I/O priority inversion
12
Storage Device
Caching Layer
Application
File System Layer
Block Layer
I/O
FG wait
wait
BG user var
wake
FG wait
I/O FG lock
BG wait
FG wait
wait BG var
wake
Request-Centric I/O Prioritization
• Solution v1 (reactive) • Request-aware I/O processing in the I/O path [FAST’17] • I/O priority inheritance [USENIX ATC’15, FAST’17]
• Locks
• Condition variables
13
FG lock
BG I/O FG BG submit
complete
FG BG
FG wait
BG
register
BG
inherit
FG BG I/O submit
complete
wake
CV CV CV
[USENIX ATC’15] Request-Oriented Durable Write Caching for Application Performance [FAST’17] Enlightening the I/O Path: A Holistic Approach for Application Performance
Request-Centric I/O Prioritization
14
Caching Layer
Application
File System Layer
Block Layer
• Solution v1 (problem)
Synchronization
linux/include/linux/mutex.h linux/include/linux/pagemap.h linux/include/linux/rtmutex.h linux/include/linux/rwsem.h linux/include/uapi/linux/sem.h linux/include/linux/wait.h linux/kernel/sched/wait.c linux/kernel/locking/rwsem.c linux/kernel/futex.c linux/kernel/locking/mutex.h linux/kernel/locking/mutex.c linux/kernel/locking/rtmutex.c linux/kernel/locking/rwsem-xadd.c
linux/kernel/fork.c linux/kernel/sys.c linux/kernel/sysctl.c linux/include/linux/sched.h
linux/include/linux/blk_types.h linux/include/linux/blkdev.h linux/include/linux/buffer_head.h linux/include/linux/caq.h linux/block/blk-core.c linux/block/blk-flush.c linux/block/blk-lib.c linux/block/blk-mq.c linux/block/caq-iosched.c linux/block/cfq-iosched.c linux/block/elevator.c
linux/fs/buffer.c linux/fs/ext4/extents.c linux/fs/ext4/inode.c linux/include/linux/jbd2.h linux/fs/jbd2/commit.c linux/fs/jbd2/ journal.c linux/fs/jbd2/ transaction.c
linux/include/linux/writeback.h linux/mm/page-writeback.c linux/include/linux/mm_types.h linux/fs/buffer.c
Interface
Request-Centric I/O Prioritization
• Solution v2 (proactive)
15
Device Driver
Noop CFQ Deadline Apposha I/O Scheduler
Block Layer
Ext4 XFS F2FS
VFS
Apposha Front-End File System
Etc
Linux I/O Stack
Page Cache
- Priority-aware I/O scheduling
- Control device-level congestion
- Control writes at the upper layer
- Tag low-priority for background I/Os
• V12 Engine
Request-Centric I/O Prioritization
16
Kernel Modules
MongoDB Library
PostgreSQL Library
V12-M V12-P
User-level library - Classify task priority, optimize WAL accesses
Front-End File System I/O Scheduler
Request-Centric I/O Prioritization
17
0
500
1000
1500
2000
2500
0 500 1000 1500 2000 2500 3000
Max
trx
late
ncy
(ms)
Elapsed time (sec)
Default Aggressive AV V12-P
0.15 sec
V12-P: V12 Engine for PostgreSQL
Request-Centric I/O Prioritization
17
0
500
1000
1500
2000
2500
0 500 1000 1500 2000 2500 3000
Max
trx
late
ncy
(ms)
Elapsed time (sec)
Default Aggressive AV V12-P v9.6
1.1 sec
V12-P: V12 Engine for PostgreSQL
New feature since v9.6: checkpoint_flush_after
Contents
• Background tasks
• Full page writes
• Parallel query
18
Full Page Writes
19 Tuning PostgreSQL for High Write Workloads – Grant McAlister
0
2
4
6
8
10
12
14
0 100 200 300 400 500 600
WA
L si
ze (G
B)
Elapsed time (sec)
Default
Full Page Writes
• Impact on WAL size
20
• PostgreSQL v9.6.5 • 52GB shared_buffer • 1GB max_wal
• TPC-C workload • 50GB dataset • 10 min run • 200 clients
WAL Compression
• Impact on WAL size
21
0
2
4
6
8
10
12
14
0 100 200 300 400 500 600
WA
L si
ze (G
B)
Elapsed time (sec)
Default WAL compression
• PostgreSQL v9.6.5 • 52GB shared_buffer • 1GB max_wal
• TPC-C workload • 50GB dataset • 10 min run • 200 clients
WAL Compression
• Impact on performance
22
0
2000
4000
6000
8000
10000
12000
Trx
thpu
t (tr
x/se
c)
Default WAL compression
WAL Compression
• Impact on performance
22
0
2000
4000
6000
8000
10000
12000
Trx
thpu
t (tr
x/se
c)
Default WAL compression
What if we can safely disable full_page_writes w/o special H/W support?
Atomic Write Support
23
write()
Data Block
In memory On disk
Mapping Table
Data Block
Linux Kernel (w/ V12 Engine)
PostgreSQL (full_page_writes off)
Atomic Write Support
23
fsync()
Data Block
PostgreSQL (full_page_writes off)
In memory On disk
Mapping Table
Data Block Data Block
writeback Linux Kernel (w/ V12 Engine)
Atomic Write Support
23
fsync()
Data Block
In memory On disk
Mapping Table
Data Block Data Block
atomic update Linux Kernel
(w/ V12 Engine)
PostgreSQL (full_page_writes off)
Atomic Write Support
23
fsync()
Data Block
In memory On disk
Mapping Table
Data Block Data Block
Linux Kernel (w/ V12 Engine)
PostgreSQL (full_page_writes off)
Atomic Write Support
23
Data Block
In memory On disk
Mapping Table
Data Block Data Block
free
Linux Kernel (w/ V12 Engine)
PostgreSQL (full_page_writes off)
Atomic Write Support
23
Data Block
In memory On disk
Mapping Table
Data Block
Linux Kernel (w/ V12 Engine)
PostgreSQL (full_page_writes off)
Atomic Write Support
• Impact on WAL size
24
0
2
4
6
8
10
12
14
0 100 200 300 400 500 600
WA
L si
ze (G
B)
Elapsed time (sec)
Default WAL compression V12-P
V12-P: V12 Engine for PostgreSQL
Atomic Write Support
• Impact on performance
25
0
2000
4000
6000
8000
10000
12000
Trx
thpu
t (tr
x/se
c)
Default WAL compression V12-P 2X
V12-P: V12 Engine for PostgreSQL
Contents
• Background tasks
• Full page writes
• Parallel query
26
Parallel Query
27
User query
Backend
Worker Worker Worker Worker
Scan, aggregate, join, …
Parallel Query
27
Slow response Fast response
User query
Backend
Worker
cache
storage
cache
storage
cache
storage
cache
storage
hit hit miss!
Worker Worker Worker
miss!
Parallel Query
• Problem inside Linux kernel
28
Fair-based (communism)
Query1 Query2
18sec
20sec I/O Scheduling
10sec 10sec
Parallel Query
• Problem inside Linux kernel
28 Fair-based (communism)
Query1 Query2
24sec
26sec I/O Scheduling
10sec 10sec
Query3
10sec
30sec
Parallel Query
• Problem inside Linux kernel
28 Fair-based (communism)
Query1 Query2
?
? I/O Scheduling
10sec 10sec
Query3
10sec
?
… QueryN
10sec
…
…
…
…
…
…
…
…
… …
Parallel Query
• Problem inside Linux kernel
29
Fair-based (communism) Task-aware (utilitarianism)
Query1 Query2
18sec
20sec
10sec
20sec I/O
Scheduling
10sec 10sec
Optimizing Parallel Query
• Illustrative experiment
30
pgbench –i –s 1000 test1 pgbench –i –s 1000 test2 Q1: SELECT * FROM pgbench_accounts WHERE filler LIKE ‘%x% -d test1 Q2: SELECT * FROM pgbench_accounts WHERE filler LIKE ‘%x% -d test2 QUERY PLAN -------------------------------------------------------------------------------------- Gather (cost=1000.00..1818916.53 rows=1 width=97) Workers Planned: 7 -> Parallel Seq Scan on pgbench_accounts (cost=0.00..1817916.43 rows=1 width=97) Filter: (filler ~~ '%x%'::text)
0
10
20
30
40
50
60
70
Deadline CFQ NOOP Taskaware
Que
ry la
tenc
y (m
s)
Q1 Q2 Q3 Q4
Optimizing Parallel Query
• Illustrative experiment
31
Recall
• Multiple independent layers
32
Storage Device
Caching Layer
Application
File System Layer
Block Layer Abs
tract
ion
Buffer Cache
read() write()
FG FG BG
BG FG BG BG
reorder
Optimizing Parallel Query
• Illustrative experiment (device queue off)
33
Avg 23%
0
20
40
60
80
100
Deadline CFQ NOOP Taskaware
Que
ry la
tenc
y (m
s)
Q1 Q2 Q3 Q4 Q1
73%
Optimizing Parallel Query
• Illustrative experiment (4ms idle_slice)
34
0 10 20 30 40 50 60 70
Deadline CFQ NOOP Taskaware Taskaware (Idle 4ms)
Que
ry la
tenc
y (m
s)
Q1 Q2 Q3 Q4 Avg 17% Q1
2.8X
Optimizing Parallel Query
• Ongoing work • Handling heavy tasks • Extending to CPU scheduling
35
