Post on 13-Apr-2017
transcript
HTTP API in ActionMax Neunhöffer, ArangoDB
Jörg Schad, Mesosphere
© 2016 Mesosphere, Inc. All Rights Reserved. 2
Jörg SchadDistributed Systems Engineer,
Mesosphere
@joerg_schad
Max NeunhöfferSenior Software Architect
ArangoDB
@neunhoef
© 2016 Mesosphere, Inc. All Rights Reserved. 3
HTTP v1 APIs ● Mesos 1.0 Milestone ● Actually multiple APIs
○ Framework■ Scheduler■ Executor
○ Operator● RPC-based HTTP API● Language independent
schedulers!
© 2016 Mesosphere, Inc. All Rights Reserved. 4
In the beginning there was libmesos
© 2016 Mesosphere, Inc. All Rights Reserved. 5
The old days…
● Not portable● Hard to
debug● Upgrade
Dependency● No
Versioning
Scheduler
Java Library
NativeLibrary
mesos.jar libmesos.so
Mesos Master
© 2016 Mesosphere, Inc. All Rights Reserved. 6
Networking ● Container?● Firewall
Scheduler
Java Library
NativeLibrary
mesos.jar libmesos.so
Mesos Master
© 2016 Mesosphere, Inc. All Rights Reserved. 7
Let us break down walls….
© 2015 Mesosphere, Inc. All Rights Reserved.
Mesos APIs
8
Scheduler Master Agent Executor
Scheduler API Executor API
Framework API
Master API Agent API
Operator
Operator API
Internal API
© 2016 Mesosphere, Inc. All Rights Reserved. 9
But then business demanded
FAST DATALet us build something new!
© 2016 Mesosphere, Inc. All Rights Reserved. 10
API V1 Goals
● Work inside firewalls / containers● Allow pure language client libraries● API versioning● Backwards compatible API evolution● Well documented
FRAMEWORK PROTOCOL• Every call is a HTTP POST request
• Content-Type: application/json or application/x-protobuf
• Successful SUBSCRIBE call results in a “200 OK” streaming response
• Record-IO formatted events
• Persistent connection
• All successful non-SUBSCRIBE calls result in “202 Accepted”
•
WORKFLOW
Scheduler Master
SUBSCRIBE
POST /api/v1/scheduler HTTP/1.1
{ "type" : "SUBSCRIBE", "subscribe" : { "framework_info" : { "user" : "foo", "name" : "Example HTTP Framework" } }}
WORKFLOW
SUBSCRIBE Scheduler Master
WORKFLOW
HTTP/1.1 200 OK
Content-Type: application/jsonTransfer-Encoding: chunkedMesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af
<event length>{ "type": "SUBSCRIBED", "subscribed": { "framework_id": { "value": "12220-3440-12532-2345" }, "heartbeat_interval_seconds": 15 }}<more events><more events>
SUBSCRIBE
Streaming Response
MasterScheduler
WORKFLOW
Scheduler Master
Offers Event<event-length>{ "type" : "OFFERS", "offers" : [ { "offer_id":{"value": "12214-23523-O235235"}, "framework_id":{"value": "12124-235325-32425"}, "agent_id":{"value": "12325-23523-S23523"}, "hostname":"agent.host", "resources":[...], "attributes":[...], "executor_ids":[] } ]}
Agent
WORKFLOW
Scheduler Master
ACCEPT
POST /api/v1/scheduler HTTP/1.1
Mesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af
{ "framework_id" : {"value" : "12220-3440-12532-2345"}, "type" : "ACCEPT", "accept" : { "offer_ids" : [ {"value" : "12220-3440-12532-O12"}, {"value" : "12220-3440-12532-O12"} ], "operations" : [ {"type" : "LAUNCH", "launch" : {...}} ], "filters" : {...} }}
Agent
Scheduler Master
WORKFLOW
ACCEPT
ResponseHTTP/1.1 202 Accepted
Agent
WORKFLOW
Scheduler Master
ACCEPT
Launch Task
Agent
WORKFLOW
Scheduler Master
Launch Task
Executor
SUBSCRIBE
Agent
ACCEPT
SCHEDULER DISCONNECTIONS & PARTITIONS
• Master tracks the persistent subscription connection
• Reconnect within framework failover timeout
• Periodic HEARTBEAT events sent by master
Operator API
• Master and Agent API• Human Operator •• Event Stream• Avoid Polling /state_summary
• RecordIO format•
21
POST /api/v1/executor HTTP/1.1
Host: agenthost:5051Content-Type: application/jsonAccept: application/json
{ "type": "SUBSCRIBE",}
Operator API
• Master and Agent API• Human Operator •• Event Stream
• RecordIO format•
22
<event-length>{ "type": "TASK_UPDATED", "task_updated": { "task_id": { "value": "42154f1b-adcd-4421-bf13-8bd11adfafaf" }, "framework_id": { "value": "49154f1b-8cf6-4421-bf13-8bd11dccd1f1" }, "agent_id": { "value": "2915adf-8aff-4421-bf13-afdafaf1f1" }, "executor_id": { "value": "adfaf-adff-2421-bf13-adf23tafa21" }, "state" : "TASK_RUNNING" }}
Versioning
• Corresponding major versions
• API compatibility
• Protobuf like
http://localhost:5050/api/v1/scheduler
http://localhost:5051/api/v2/executor
API STATUS
• Scheduler API & Executor API : Stable
• Operator API : Experimental
• Client library implementations
© 2015 Mesosphere, Inc. All Rights Reserved. 25
● multimodel Database
● extensible
© 2016 Mesosphere, Inc. All Rights Reserved. 26
Writing a framework scheduler for a stateful service is hard:
● A part of the scheduling happens in the framework (accepting offers).● Getting the scheduling logic right is difficult● A framework needs to be resilient (use zookeeper)● The ArangoDB framework consists of 5140 SLOC in C++● It handles
● deployment,● dealing with persistent volumes, ● a part of the failover, and● a part of the up and down scaling.
ArangoDB Mesos Framework
© 2016 Mesosphere, Inc. All Rights Reserved. 27
libmesos makes it challenging to develop frameworks:
● The framework executable is usually deployed as Docker image.● There are two connections needed between framework and Mesos master in
different directions (network isolation problems).● One is limited to C++, Java or Python.● The libmesos source is huge: 190K SLOC● A Docker image to build a framework with libmesos needs >3GB● Documentation is mostly hidden in the source.● Debugging is a challenge (Docker, Mesos, binary messages, network issues).
Motivation ArangoDB
© 2016 Mesosphere, Inc. All Rights Reserved. 28
Solution: Use a SDK that works via the HTTP protocol.
The HTTP protocol provides the following advantages:
● any language works that can to HTTP/JSON● JSON is clear-text and easier to debug● using an SDK removes the amount of boilerplate code● do not need libmesos● much smaller footprint
Prototype: Mesos framework SDK for node.js
Still: scheduling and failure management
HTTP based ArangoDB
Writing Stateful Frameworks - Ken Sipe & Jörg SchadSaturday, 10:25
© 2016 Mesosphere, Inc. All Rights Reserved. 29
Challenges
● State Abstraction missing (MESOS-2916) ● Mesos-framework
● Persistent Volumes● Failover● Scaling● Operations (e.g., Updates)
© 2016 Mesosphere, Inc. All Rights Reserved. 30
● Shim● Switch between old/new API● Independent code
● Mesos-framework● JS based SDK● Not production!
● DC/OS SDK
How to get started?var scheduler = new Scheduler({
"masterUrl": "172.17.10.103", // If Mesos DNS is used this would be
"leader.mesos", otherwise use the actual IP address of the leading master
"port": 5050,
"frameworkName": "My first Command framework",
"logging": {
"level": "debug" // Set log Level to debug (default is info)
},
"restartStates": ["TASK_FAILED", "TASK_KILLED", "TASK_LOST",
"TASK_ERROR", "TASK_FINISHED"], // Overwrite the restartStates (by default,
TASK_FINISHED tasks are NOT restarted!)
"tasks": {
"sleepProcesses": {
"priority": 1,
"instances": 3,
"commandInfo": new Mesos.CommandInfo(
"sleep 10;"
),
"resources": {
"cpus": 0.2,
"mem": 128,
"ports": 1,
"disk": 0
}}}
Writing Stateful Frameworks - Ken Sipe & Jörg SchadSaturday, 10:25
© 2016 Mesosphere, Inc. All Rights Reserved. 31
● Use libraries● Shim for easy switching● Persistent Connection
● Connection: keep-alive● Transfer-Encoding: chunked
● HTTP Pipelining● React to heart beats● Implement authentication schema
Best Practices
© 2016 Mesosphere, Inc. All Rights Reserved. 32
Thank You!
Learn more by visiting dcos.io and arangodb.com.
© 2016 Mesosphere, Inc. All Rights Reserved.
THANK YOU!
33