Date post: | 27-Jul-2015 |
Category: |
Technology |
Upload: | etcenter |
View: | 72 times |
Download: | 1 times |
Pop-Up Render Farms
Production in the Clouds and Beyond
Presented by Chris Bond
Pop-Up Render Farms
The logistics of rendering in a VFX environment can be incredibly complex; add
cloud-based cores into the mix and they become even more complex.
Different render applications, simulation tools, assets and dependencies are driven
by unique workflows tailored to the needs of the client; the artist; and the job at
hand.
The key to making cloud rendering successful, is to bring that incredibly scalable
resource to a facility's unique and local pipeline. Learn how studios can seamlessly
merge public cloud resources such as Google, Amazon, Azure, Rackspace and
others to their internal farms, adding thousands of cores as if they were inside the
studio, giving facilities the flexibility to work the way that suits them best.
Pop-Up Render Farms – Overview
Define „Rendering‟
What is and Why use a Render farm Management tool.
Clouds
Decisions you need to make
First, a little bit about me:
Began career in Commercials & Computer Graphics in early „90s
First Feature Film: Swordfish (1999) Bank Explosion (VFX Supervisor)
Production across 3 countries with 4 offices in 2001 at Frantic Films
Over 40 projects, last VFX credit – AVATAR (VFX Supervisor)
Formed Thinkbox Software November 2010
Let’s talk about Rendering
What is rendering?
Defined as (1) : to reproduce or represent by artistic or verbal
means : to depict (2)
In Media & Entertainment [M&E] industries, Rendering is the process
of having a computer do work to generate an image.
Often used to describe simulation, meshing, encoding, conversion
I prefer „Computing‟ to „Rendering‟
Render ‘Farms’
Computing can take time, often minutes, hours or days.
Generate 24, 30, 60 or more images for a second
5 seconds at 30fps, is 150 images (frames).
If each takes one hour to render – 150 hours or over 6 days
It is clear that you need to distribute that workload
Collectively these computers are known as a „farm‟ or „render farm‟
Render Farm Management
How do you distribute all that work across those computers?
Manual assignment is time consuming and prone to error.
Render Farm Management Systems distribute work.
Provide management and flexibility
Multiple artists, editors, departments and teams can share.
Advanced Rendering Systems
Sanity Checks, Dependency Systems, Load Balancing, Notification systems,
Idle checking, Scripting access, and a myriad of other features.
Render Farm Management
Example Screen:
JOBS
SLAVES GRAPHS!
JOB TASKS
Render Farm Management Continued
How complex is the workflow?
Consider the following: Image ingestion and processing – cleanup, rotoscopy, tracking, color correction
Data ingestion and processing – point clouds, camera data, shared mesh, model assets
Simulation – complex fluid simulation, animation rig/hair simulation, breaking, dynamics, skin and cloth
simulation, fire, smoke and particle generation
„Rendering‟ – generation of images using a variety of rendering tools, creating passes, iterating light,
texture, camera and color
Compositing – input of images, compositing work and color correction
Encoding – generation of video formats and codecs
Collectively, often referred to as the “PIPELINE”.
Pipelines can differ completely.
Even within the same company.
Virtual Machines vs Physical Machines
A short summary
Traditionally Physical Machines aka „bare metal‟
More are building „Virtual Machines.
Flexible platforms (OS)
Software deployments
Archival
Look for a Render Manager that supports both Physical and Virtual machines
directly.
Lets talk about ‘Private Clouds’
Studio Compute Resources:
Dedicated compute resource
Idle time
Combination of those.
Multiple physical locations
Studios, design firms were essentially rendering on a „private cloud‟
Private Cloud Benefits
The Benefits of Rendering Locally
Data exists.
Security.
Pipeline and workflow.
Multiple offices, you can share your internal, idle resources.
Cost
If used consistently
Lets talk about Public Clouds
What are public clouds?
Offsite from your facility.
owned by a 3rd party
rent per month, per hour or per minute.
Amazon, Google, Microsoft, Rackspace
No access to „bare metal‟
Control by Virtual Machines.
Public Clouds have most advantageously been used by developers to
scale their websites, host on-demand software applications and as a
frequent delivery model for business/enterprise applications.
Public Cloud Benefits
Considering Public Cloud for your Pipeline
A fairly elastic resource
Scale
On demand billing
No initial capex fees
No real build-out time
Maintenance/systems
Lowered costs for infrastructure
A Note about SAAS
Public Clouds and SAAS
SAAS [Software As A Service] model
business applications (CRM, ERP)
within a browser
mobile applications
Pros and Cons
Often simple to engage
little control of the pipeline
Lack of ownership
Premium cost
Multi-resident issues
Render Farms aka Compute 2.0
Lets get to the good stuff
Basics
Reasons to expand
Make your work better
Get results faster.
Most important resource: your talent
Artist/Designer Time
0
20
40
60
80
100
120
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100105110115120
AR
TIS
T
PR
OD
UC
TIV
ITY
JOB TURNAROUND TIME (MINUTES)
Artist/Designer Time
0
20
40
60
80
100
120
140
160
180
200
S M T W R F S S M T W R F S S
PR
OD
UC
TIV
ITY
DAY
Artist/Designer Time
0
20
40
60
80
100
120
140
160
180
200
S M T W R F S S M T W R F S S
PR
OD
UC
TIV
ITY
DAY
Pop-Up Cloud
A hybridized approach to render farms
Combine local compute resources, offsite private resources, and public
cloud resources into one logical pipeline:
REMOTE Office
VPN
LOCAL Office
Database
&
Repositor
y VPN
CLOUD Region (on demand)
Pop-Up Cloud
How does it work?
Software/ Machine configurations
Created
Stored
Budgets and Needs
Data is Synchronized
Instances are controlled by a „Load Balancer‟
Lets go over that in a bit more detail!
Pop-Up Cloud – Software/Machine Configurations
Duplicating your workflow on a Public Cloud
Review Tasks
Build a Virtual Machine Image
Some Render Management Systems provide a default installation tool.
Map your VM Image to the Cloud machines [instances]
Assign Group names
Render Management System controls the system.
Software & Machine Configurations
Pop-Up Cloud – Budget Assignment
The Public cloud requires some constraints
There is a cost.
A system should not spend money if the jobs cannot render [assets]
Provide a set of rules
Cost Estimation
Be Safe
Be very aware of the „Free Soda‟ effect
Pop-Up Cloud – Budget Assignment
Pop-Up Cloud – Data Synchronization
Determine the best approach for your task
Data
Project file
Assets (models, textures, files)
Results
A variety of approaches to project rendering in the cloud:
Synchronize that data early
Sync everything
Sync on demand
Pop-Up Cloud – Data Synchronization
Further to Synching On-Demand (or ‘just in time’)
Asset dependencies in Render Management Software ensure that a
job is not started in a Cloud Region [or a remote office for that matter]
without the assets being available.
Asset lists are generated and compared to determine an action.
Asset dependencies can trigger a „sync‟ request.
JOB: SYNC
Asset TRANSFERRING
ASSET
QUEUE
JOB:
Render SUBMITTED
LOCAL CLOUD
ASSET DEPENDENCY:
ASSET REQUIRED
LOCAL
Pop-Up Cloud- Load Balancing Turning Cloud Machines on and off as required
Enabling cloud machines by hand would be tedious
Saves on cost, disabling and enabling
Balance across regions and providers.
True cross-cloud workflows, redundancy.
Cloud Region West
VM instance
VM instance
VM instance
VM instance
VM instance
Cloud Region East
VM instance
VM instance
VM instance
VM instance
VM instance
Pop-Up Cloud – SECURITY
How important is Security to your project?
SAAS models might be less attractive.
Most secure is local rendering
Private Cloud use inside a company or across owned resources next
most secure.
Public Cloud can be secure with VPN or HTTPS control
Because you control the VM, you can better secure/harden
Ultimately you determine what you process off-site. Elements vs. final shots
Remove data once you are complete
Pop-Up Cloud Conclusions
Musings and Thoughts
Public Cloud compute can help you get work done faster.
Other Considerations:
Delay decisions as long as possible?
Emergency assist?
Redundancy to ensure up-time/responsiveness?
Do you want to build the workflow of the future?
Is your team remote?
Enable time and materials billing?
Will it provide actual costs of rendering time?