Performance Evaluation Between Checkpoint Services in Multi-tier Stateful

ApplicationsDemis Gomes

Advisor: Glauco GonçalvesCo-Advisor: Patricia Endo

2

INTRODUCTION

3

Introduction

• Plataform-as-a-Service (PaaS)

DeveloperPaaS

Application

User

PaaS Provider

4

Introduction

• Multi-tier stateful applications

5

Introduction

• It is important keep an application in a PaaS running as long as possible

• A downtime causes many financial losses

6

Introduction

• The average cost of a critical application failure per hour is $500,000 to $1 million.

Source: https://devops.com/2015/02/11/real-cost-downtime/ . Last access 11 out. 2016

Checkpoint Services!

7

IntroductionDevelopers Users

CheckpointService

PaaS Providers

8

Background

• A checkpoint service is divided into three mechanisms– Checkpoint saving– Failure detection– Failover

9

Background

• Checkpoint Service

AppActiveStandb

y

Checkpoint Service

AppState

AppState

AppState

Failover

CheckpointSaving

App

FailureDetection

10

Background

• Service Availability Forum (SAF)

• Three different implementations:– Non-collocated– Collocated warm– Collocated hot

11

Background

12

Checkpoint ServicesCS Application-level CS System-level

App

Agent

State-aware application

App

Agent

HA-agnostic application

ContainerCheckpoint

Manager CheckpointManager

13

Motivation

• Works presented either app-lvl [1] or sys-lvl [2]

• Lack of consistent comparison between these services

• No implementation in accordance with the SAF standard

14

Motivation

• Carry out a performance evaluation between system and application checkpoint services, where these models follow the SAF standard and evaluate the impact of different recovery modes in time and resource consumption

15

Answer three questions

• System-level ~= App-level?• Impact of changing from non-

collocated to collocated?• Bottlenecks of the system-level

and application-level?

16

CHECKPOINT SERVICES

17

Application

• State-aware application • A multi-tier stateful chat– Frontend: provides interface and

saves user’s data– Backend: saves room messages– Database: stores information related

to rooms and users

App AgentGET /state

200 OK

18

Application

• State provided via JSON (backend)

19

CS System-level

• We used well-known tools:– LXC as container–NFS as file system– rsync to transfer files between

instances– CRIU to establish checkpoint and

restore containers

CS: Checkpoint Service! :D

20

CS System-level

• We did not implement collocated hot because CRIU does not allow restore in a running instance

21

CS System-level

• Checkpoint in non-collocatedApp

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

Container

Container

22

CS System-level

• Checkpoint in collocated warmApp

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

rsync

Container

Container

23

Container

CS System-level

• Failover in non-collocatedApp

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

Container

24

Container

CS System-level

• Failover in collocated warmApp

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

Container

rsync

25

CS App-level

• CS at application-level was developed from scratch for this work

• REST resources

Remember, CS: Checkpoint Service! :D

GET http://{manager_ip}:{manager_port}/config

RESPONSE 200 OK Content-type: application/json

26

CS App-level

• Checkpoint at Application-level

App

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

State-aware application Non-collocated

Collocatedwarm

Collocatedhot

27

CS App-level

• Failover in non-collocated

App

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

28

CS App-level

• Failover in collocated warm

App

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

29

CS App-level

• Failover in collocated hot

App

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

30

EVALUATION

31

Evaluation

• Two evaluations were conducted– Evaluation I: Failover time

comparison – Evaluation II: Checkpoint time and

resources consumption comparison

32

Evaluation

Physical Machines: 16 GB RAM, 8 cores, Gigabit Interface

33

Evaluation I

• Methodology– Backend with 1, 5,10,15,20 and 25

MB of state sizes– Experiment Manager starts the

experiment and generates a failure alert

– Failover process is executed– Failover time is collected

34

Failover time – Non collocated

Application-level has a greater failover time

The growth is linear

35

Failover time – Non collocated

We estimate the failover time with state size increasing until 100 MB

App lvl would be 66% faster

36

Failover time – Collocated

Application-level collocated warm is greatly impacted with increase of state size

The values of app lvl collocated hot and sys lvl collocated warm are very similar

37

Failover time – Collocated

Linear regression shows:

High increase of app lvl collocated warm

Slight increase on sys lvl collocated warm

Constant values to collocated hot

38

Evaluation II

• Methodology– Similarly to the previous experiment,

states are saved in same state sizes– Experiment Manager triggers a

checkpoint process– Checkpoint time is collected– Resources consumption are

evaluated

39

Evaluation II

• Methodology– Resources consumption metrics

Metrics Measured inCheckpoint Time s

CPU Load %

Memory Occupation %

Network I/O Throughput Mbps

Disk I/O Throughput b/s

40

Evaluation IICheckpoint times

41

Evaluation II – Active InstanceAt 25MB CPU Memory Network (I/O) Disk (W)

Sys-lvl collocated

warm

6,8% 9,4% 0/59,8 Mbps 1300 b/s

App-lvl collocated

warm

2,7% 9,1% 0/8,8 Mbps 9220 b/s

App-lvl collocated hot

2,53% 9,5% 0/8,64 Mbps 8340 b/s

At 25MB CPU Memory Network (I/O)

Disk (W)

Sys-lvl non-collocated

6% 9,1% 0/81 Mbps 1780b/s

App-lvl non-collocated

2% 8,92% 0/11,6 Mbps

2410 b/s

42

Evaluation II – Standby InstanceAt 25 MB CPU Memory Network (I/O) Disk (W)

Sys-lvl collocated

warm

1,8% 10,3% 5,1/0 Mbps 12500 b/s

App-lvl collocated

warm

2,5% 11,9% 8,5/8,5 Mbps 7280 b/s

App-lvl collocated

hot

4,1% 12,4% 8,35/8,35 Mbps

6900 b/s

At 25 MB CPU Memory Network (I/O)

Disk (W)

Sys-lvl non-collocated

0,16% 9,8% 0/0 Mbps 800 b/s

App-lvl non-collocated

0,2% 11,4% 0/0 Mbps 2600 b/s

43

Discussion

• Availability Analysis in a year• Mean Time To Recovery (MTTR) as

failover time• Mean Time To Failure (MTTF) as

Apache Server (788.4h/year) [3]• Assuming that the failover time is 50

times greater• High Availability (HA) = 99.999%

(five nines)

44

Discussion

MTTR in25 MB (s)

MTTR in 25 MB with

factor 50 (s)

MTTF(s) Availability with factor 50 (%)

System-levelcollocated warm

0.38636 19.318 2838240 99.9993

Application-level collocated warm

1.27823 63.9115 2838240 99.997

Application-levelcollocated hot

0.25802 12.901 2838240 99.9995

System-levelnon-collocated

3.5441 177.205 2838240 99.9937

Application-level non-collocated

1.38795 69.3975 2838240 99.997

Availability analysis (25 MB)

45

Discussion

MTTR in100 MB

(s)

MTTR in 100 MB with

factor 50 (s)

MTTF(s) Availability with factor 50 (%)

System-levelcollocated warm

0.5902 29.51 2838240 99.9989

Application-level collocated warm

3.8621 193.1 2838240 99.993

Application-levelcollocated hot

0.2677 13.385 2838240 99.9995

System-levelnon-collocated

9.7999 498.995 2838240 99.9824

Application-level non-collocated

4.321 216.05 2838240 99.9923

Availability analysis (prediction until 100 MB)

46

CONCLUSIONS AND FUTURE WORKS

47

Conclusions

Answering the questions• System-level ~= App-level?

Yes! In collocated warm

48

Conclusions

• Impact of change from non-collocated to collocated?– Failover: great decrease– Checkpoint: great increase– Resources Consumption: Similar,

except of CPU and disk (greater on collocated)

49

Conclusions

• Bottlenecks of the system-level and application-level?

– App : disk, CPU in standby (hot) and development time

– Sys: CPU, network and NFS

50

Conclusions

• CS Application-level– Private PaaS – App with large state size and high

rate of checkpoints (massive online applications)

51

Conclusions

• CS System-level– PaaS with legacy applications– App with less state size and higher

checkpoint intervals

52

Conclusions

• PaaS Business Model– Non-collocated: Free plans– Collocated: Premium plans

53

Contributions

• Short paper approved with results of Experiment I, entitled:

“Failover Time Evaluation Between Checkpoint Services in Multi-tier Stateful Applications”

IM-2017, Exp. Session (Qualis B1)

54

Future Works

As future works, we will study• Scalability of services• Resources consumption on

Experiment Instance

55

Acknowledgments

• Thanks!

#CatãoEterno

56

THANKS!

Demis Gomesdemismg72@gmail.comdemis.gomes@ufrpe.br

57

References• [1] KANSO, Ali; LEMIEUX, Yves. Achieving High Availability

at the Application Level in the Cloud. In: 2013 IEEE Sixth International Conference on Cloud Computing. IEEE, 2013. p. 778-785.

• [2] LI, Wubin; KANSO, Ali; GHERBI, Abdelouahed. Leveraging linux containers to achieve high availability for cloud services. In: Cloud Engineering (IC2E), 2015 IEEE International Conference on. IEEE, 2015. p. 76-83

• [3] MELO, R. M. D. et al. Redundant vod streaming service in a private cloud: availability modeling and sensitivity analysis. Mathematical Problems in Engineering, Hindawi Publishing Corporation, v. 2014, 2014

58

BACKUP

59

Agenda

• Introduction• Checkpoint Services• Evaluation– Experiment I– Experiment II

• Conclusion and Future Works• Acknowledgments

60

Introduction

• PaaS contains several challenges, where one is the availability of your services

• Multi-tier stateful applications

61

Introduction

• Many PaaS not have a mechanism that handles failures on application

• Some offers a backup but is not transparent

62

Introduction

Tsuru only restarts application, not saving your last state

63

VM x Container

• VMs • Containerization

64

Objectives• General– Carry out a consistent comparison between

checkpoint in system and application levels• Specifics– Develop the two modes following SAF

standard– Compare the services among following

metrics:• Failover time• Checkpoint time• Load generated in application

65

Application

• Application generates new base states if– threshold defined by developer has

reached– A time limit has reached

App 20 new messages!

App 120 seconds without updates!

66

CS System-level

67

CS System-level

• Checkpoint/Restore In Userspace (CRIU)

• Saves memory context• Freezes processes reading

memory• Restores processes in machines

with same filesystem

68

CS System-level

• Phoenix!

69

Checkpoint Services Implementation

• URLS implemented by chat

70

Checkpoint Services

• CS Application-level

App

CheckpointManager

Agent

App

Agent

Standby Instance

ActiveInstance

State-aware application Non-collocated

Collocatedwarm

Collocatedhot

71VM/Container

Checkpoint Services

• CS System-levelApp

CheckpointManager

Agent

App

AgentStandby Instance

ActiveInstance

HA-agnosticapplication

Non-collocated

Collocatedwarm

Collocatedhot

VM/Container

72

CS System-level

• LXC must be configured to allow CRIU make checkpoint and restore

73

Evaluation II

• Methodology– Checkpoint time is presented as

means with 95% Confidence Interval (CI)

– Resource consumption are means with 95% CI related to active and standby instances

74

CS System-level

• Checkpoint process is established in non-collocated– saving container via CRIU and storing

your memory context in a shared file system between Manager and Agent

• In collocated:– saving container via CRIU and send

state via rsync to all standby instances

75

CS System-level

76

CS System-level

• Failover process (non-collocated)

77

CS System-level

• Failover process (collocated warm)

78

CS App-level

79

CS App-level

• In failover process (non-collocated)

80

CS App-level

• In failover process (collocated warm)

81

CS App-level

• In failover process (collocated hot)

82

Evaluation I

• T-test between app collocated hot and sys collocated warm

83

Evaluation IINetwork received (collocated modes)

84

Evaluation IINetwork received (non-collocated)

85

Evaluation IICPU Load (collocated modes)

86

Evaluation IICPU Load (non-collocated)

87

Evaluation IIMemory occupation (collocated modes)

88

Evaluation IIMemory occupation (non-collocated)

89

Evaluation IINetwork sent (collocated modes)

90

Evaluation IINetwork sent (non-collocated)

91

Evaluation IIDisk written (collocated modes)

92

Evaluation IIDisk written (non-collocated)

93

Acknowledgments

• Family• Friends• Creators• UFRPE• Advisors (the bests)• CNPq and FACEPE