Date post: | 08-Jan-2017 |
Category: |
Technology |
Upload: | flink-forward |
View: | 7,017 times |
Download: | 5 times |
IBM Confidential 1
Apache Flink Cluster Deployment
On Docker and Docker-Compose
Simon Laws
© 2014 IBM Corporation
What We’re Going To Build
2
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
IBM Bluemix Signup
Sign up for a free
account and experiment
Various options for
academics, startups etc.
Prices here for when
you go over free
allocation
https://console.eu-
gb.bluemix.net/pricing/
6
https://console.ng.bluemix.net/registration/
© 2014 IBM Corporation
Apache Flink and Docker
7
github.com/apache/
flink/tree/master/
flink-contrib/
docker-flink
© 2014 IBM Corporation
Install Ubuntu Using VirtualBox
8
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
Setting Up The Host Operating System
If you’re on Linux you’re probably good to go
If you’re on Windows or OSX you need to run Linux on a VM As an alternative you could do this on a VM in the new VM service in Bluemix. It’s in Beta at the moment.
Get VirtualBox
Create a Linux VM (I used Ubuntu)
Install Docker
Download Flink Source
Try a Flink example
9
© 2014 IBM Corporation
Create VM from ISO in VirtualBox (and set port forwarding)
12
End result is that I can now SSH into the instance of
Ubuntu running inside VitualBox. Everything for this
point on happens inside this Ubuntu VM
To start with make sure that Ubuntu is up to date
> sudo su -
> apt-get update
sudo su -
apt-get updatesudo su -
apt-get update
© 2014 IBM Corporation
Install Docker and Flink
13
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
Install Docker and Docker-Compose
14
github.com/apache/
flink/tree/master/
flink-contrib/
docker-flink
© 2014 IBM Corporation
Download Flink Source
15
git clone https://github.com/apache/flink.git
cd flink/flink-contrib/docker-flinkThe base Docker image
source
Builds the “base” and “flink”
images
The flink Docker image
source
Docker-Compose
configuration
© 2014 IBM Corporation
Docker Images and Containers
16
SourceDockerImage
DockerContainer
docker build docker run
What Images
Do I have?
docker images
What Containers
Do I have running?
Push elsewhere (Docker Hub)
To share with others
ssh or
docker exec
docker psDockerfile
© 2014 IBM Corporation
Build Docker Images
18
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
Building The Images From the Docker Files
19
./build.sh
If you look in the script it use the “docker build” command to build a “base” image and a
“flink” image
This failed for me on the first attempt with the base build reporting
chpasswd: (user root) pam_chauthtok() failed, error:
System error
chpasswd: (line 1, user root) password not changed
It turned out I had an old version of Docker so I updated to latest and it worked
© 2014 IBM Corporation
Run Docker Containers
21
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
Running Linked Images Manually (To See What’s Going On)
22
docker run -d -p 48080:8080 -p 48081:8081 --expose 6123 --name
jobmanager flink /usr/local/flink/bin/config-flink.sh jobmanager
docker run -d --expose 6121 --expose 6122 --link jobmanager --name
taskmanager1 flink
The job manager
Add a task manager
This failed at first with a strange “Exec format exception”. I made a local fix by removing
the comment from the top of the flink/config-flink.sh file leaving
#!/bin/bash
As the first line
© 2014 IBM Corporation
Running a Test
24
I had previously uploaded flink-java-examples-0.9.1-WordCount.jar from my local Windows install of Flink
© 2014 IBM Corporation
Making Things Simpler With Docker-Compose
26
docker-compose up –d
docker-compose scale taskmanager=3
docker-compose kill
© 2014 IBM Corporation
Connecting To Bluemix
27
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
Running the Images On Bluemix
28
Log in to Bluemix:1.cf login
Run the IBM Containers cf CLI plug-in.2.cf ic login
Create a Dockerfile in the root directory of your app source files. Learn more about Dockerfiles.
Build an image from your Dockerfile. The command returns an image ID.3. docker build -t image_name
Tag the image with your private namespace in the IBM Containers registry.4. docker tag image_name registry.stage1.ng.bluemix.net/sluk/image_name:image_tag
The image name is optional. If it is not specified, the image is tagged with latest.
Push this image to the IBM Containers registry:5. docker push registry.stage1.ng.bluemix.net/sluk/image_name:image_tag
You can create a container from this image in the Bluemix Catalog, or with the following command:6. cf ic run --name container_name
registry.stage1.ng.bluemix.net/sluk/image_name:image_tag
© 2014 IBM Corporation
Install The Cloud Foundary Client Software
29
In the Ubuntu VM we have been using to run Flink using Docker get and install the Cloud
Foundary client tool
wget --output-document=cfcli.deb
https://cli.run.pivotal.io/stable?release=debian64&version=6.12.4&source=githu
b-rel
dpkg -i cfcli.deb
In the Ubuntu VM we have been using to run Flink using Docker get
and install the Cloud Foundary client
cf install-plugin https://static-ice.ng.bluemix.net/ibm-containers-linux_x64
© 2014 IBM Corporation
Push Image Up To Bluemix
33
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
Tag and Push Up Our Images
34
docker tag flink registry.ng.bluemix.net/sluk/flink
docker push registry.ng.bluemix.net/sluk/flink
© 2014 IBM Corporation
Run Containers On Bluemix
36
Windows 7 IBM Bluemix
VirtualBox 4.3.18
Ubuntu 14.04
Docker 4.3.18
Cygwin
ssh
Flink
Jobmanager
container
Flink
Taskmanager
container
Cloudfoundry
Client
Docker
Flink
image
Flink
Jobmanager
container
Flink
Taskmanager
container
Flink
image
© 2014 IBM Corporation
Creating Containers in Bluemix via CF Command Line
38
Option 1) Leverage this Cloud Foundry IBM Containers plugin without affecting the local docker environment:
Example Usage:cf ic ps
cf ic images
Option 2) Leverage the docker CLI directly. In this shell, override local docker environment to connect to IBM
Containers by setting these variables,
export DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443
export DOCKER_CERT_PATH=/root/.ice/certs
export DOCKER_TLS_VERIFY=1
Example Usage:docker ps
docker images
© 2014 IBM Corporation39
Create jobmanager Container in Bluemix via CF Command Line
cf ic run -p 22:22 -p 8080:8080 -p 8081:8081 -p 6123:6123 --expose
6123 --name jobmanager registry.ng.bluemix.net/sluk/flink
/usr/local/flink/bin/config-flink.sh jobmanager
Assign a public IP
so you can get
to the console
© 2014 IBM Corporation40
Create taskmanager Container in Bluemix via CF Command Line
cf ic run -p 22:22 -p 6121:6121 -p 6122:6122 --expose 6121 --
expose 6122 --name taskmanager1 --link jobmanager:jobmanager
registry.ng.bluemix.net/sluk/flink /usr/local/flink/bin/config-
flink.sh taskmanager