Distributed Video Rendering using Blender, VirtualBox,and BOINC. Christopher J. Reynolds.Centre for Parallel Computing,University of Westminster.
Material:
• The general concept and motivations.
• Choosing a system virtualization technology.
• Interactions with the hypervisor.
• Final comments.
Background: University of Westminster
• Approx 1600 node Windows & BOINC based DG.
• Runs applications from Enabling Desktop Grids for E-Science (EDGeS).
• 20 node GPU cluster under construction to run further EDGeS/DEGISCO applications.
• For most of 2009 we ran a public video/animation rendering service on a dedicated cluster (~256 Nodes).
Blender: http://www.blender.org
• Rastorization based rendering of frames; a Bag of Tasks (BoT).
• Fits the DG model so move from cluster to DG.
• Uses Blender, a well known animation/image rendering suite.
• Q: What are Blender’s requirements? • A: Python.
• Q: What other considerations are required for a public service?• A: Python scripts can be embedded within Blender input files (Security)
• Q: How to address these issues?• A: Virtualization
Existing Virtualization Solutions
• Existing solutions use either process virtualization or system virtualization.
• Process virtualization (using Entropia VM) proposed by Calder et al [1] is tied to Windows as the host OS.
• VM Wrapper: Generic but more complex and incomplete (Windows Host Support).
• Marosi et al described proof of concept using QEMU based on features [2] .
• Q: QEMU/KQEMU now obsolete on Windows, so what next? • A: Look at latest feature set and choose something else for a
Lightweight solution.
QEMU/KQEMU vs VirtualBox
Feature VirtualBox QEMU KQEMU
X86 guest only. Yes No Yes
Requires admin privileges to run.
No – Add user to allow execution of
VBox DCOM service.
No No
Requires admin to install.
Yes No Yes
CPU performance overhead (x86) [3]
20%(SP)10%(FP)
Huge (emulation) 100%(SP)30%(FP)
Supports disk overlay/cow images.
Yes Yes Yes
Guest to Host File I/O (Without Network)
Yes (Guest Additions)
No No
Monitor/Hypervisor interface.
COM Sockets Sockets
Windows support Yes Yes (Partial) Obsolete
Solution: A Lightweight VM Controller.
Host/DG Node
DG (BOINC) Server
BOINC Core Client
VM (Guest)
Disk image (guest OS), VM config file, inputs.
Instantiate VMand transfer
inputs
React to messages: suspend, resume,
abort
Transfer outputs
VM Controller (standard BOINC job)
Start
1
2
3
45
6
VBox Hypervisor
Transfer outputs
VirtualBox Hypervisor/Monitor ControlVM Controller (C++ Boinc application)
• Win32 CreateProcess() or fork() to VboxManage cmdln tool.
• COM preferable.
(COM)
VM Controller
• Two Threads.• Worker launches unzipper (lzma), prepares VM config files, creates I/O
shared directories, polls for Job completion.• Monitor thread deals with messages and VM control.
VM Controller (C++ Boinc application)
Worker Thread
Monitor Thread
VM Worker Script: Blender
echo 15 > /proc/$$/oom_adjecho "Setting oom_adj to 15 for PID=$$"
case "$1" in start)
mkdir /mnt/sharemount -t vboxsf Shared /mnt/share
mkdir /mnt/share/Outputs
swapon /dev/sdaswapon /dev/sdb
START_FRAME=`cat /mnt/share/input.txt | head -1 `END_FRAME=`cat /mnt/share/input.txt | head -2 |tail -1 `
sudo –u blender_user sh -c "blender -b /mnt/share/input.blend -o /mnt/share/Outputs/Frame-# -F PNG -x 0 -s $START_FRAME -e $END_FRAME –a”
echo $? > /mnt/share/Outputs/blender_return.log
echo "Finished Rendering Application";;
Further Comments
• We adhere to the “one VM per Workunit” model.
• Our BOINC user account has restricted permissions.
• This means VirtualBox.xml and MachineDefinition.xml are loaded from C:\Windows\system32\config\systemprofile\.VirtualBox
• Single xml file location restricts us to one WU per node.
• Proof of concept, though proposed roll out soon (late 2010).
References
[1] Calder, B., Chien, A., & Yang, J. W. D. (2005). The entropia virtual machine for desktop grids. In VEE ’05: Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments (pp. 186–196).
[2] Marosi, A., Kacsuk, P., Fedak, G., & Lodygensky, O. (2010). Sandboxing for Desktop Grids using Virtualization. In The 18th Euromicro International Conference on Parallel, Distributed and Network-Based Computing.
[3] P. Domingues and F. Araujo and L. Silva. (2009) Evaluating the performance and intrusiveness of virtual machines for desktop grid computing. In IPDPS '09:Proceedings of the 2009 IEEE International Symposium on Parallel & Distributed Processing.
Questions?