Date post: | 24-May-2015 |
Category: |
Technology |
Upload: | manageiq |
View: | 571 times |
Download: | 1 times |
Upgrading to Ruby 2.x
Joe Rafaniello
@jrafanie
2007
13
Agenda
1. History
2. Why upgrade?
3. Ruby 2.1
4. Ruby 2.0
5. "Fall cleanup" of old code
6. Slow tests
7. Building 2.0 appliances
8. Developer setup
9. Links
10. Questions?
History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
Tue Apr 23 2013:
Ruby 1.9.3 finally...
History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
Tue Apr 23 2013:
Ruby 1.9.3 finally...
540 days???
History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
Tue Apr 23 2013:
Ruby 1.9.3 finally...
540 days???
Lesson learned: Don't wait to upgrade!
Why upgrade?
Why upgrade?
We're behind!!!
Ruby 1.9.3 is ending
In maintenance until February 23, 2014
Security only mode until February 23, 2015
Ruby 2.0.0 is nearly 20 months old
Ruby 2.1.0 is nearly 10 months old
Ruby 2.2.0 is scheduled for a Christmas release
Why upgrade? ... Because ruby 2.1!
Generational mark and sweep garbage collector
http://tmm1.net/ruby21-rgengc/
String#freeze - reuse String objects
Less objects == less memory == less GC time
Object allocation tracing
http://tmm1.net/ruby21-objspace/
https://github.com/srawlins/allocation_stats
Required keyword arguments
def returns method name
Exception#cause - ActiveRecord::StatementInvalid#cause -> Real error
More...
Why upgrade? ... Because ruby 2.1!
Useless benchmark
bundle exec rspec spec/models/ems_refresh/refreshers
1.9.3-p545
70.85s user 2.23s system 96% cpu 1:15.98 total71.21s user 2.22s system 96% cpu 1:16.27 total
Why upgrade? ... Because ruby 2.1!
Useless benchmark
bundle exec rspec spec/models/ems_refresh/refreshers
1.9.3-p545
70.85s user 2.23s system 96% cpu 1:15.98 total71.21s user 2.22s system 96% cpu 1:16.27 total
2.0.0-p576
54.02s user 2.03s system 95% cpu 58.980 total49.96s user 2.14s system 94% cpu 54.923 total
Why upgrade? ... Because ruby 2.1!
Useless benchmark
bundle exec rspec spec/models/ems_refresh/refreshers
1.9.3-p545
70.85s user 2.23s system 96% cpu 1:15.98 total71.21s user 2.22s system 96% cpu 1:16.27 total
2.0.0-p576
54.02s user 2.03s system 95% cpu 58.980 total49.96s user 2.14s system 94% cpu 54.923 total
2.1.3
36.52s user 2.79s system 91% cpu 42.930 total35.68s user 2.22s system 92% cpu 40.768 total
Why upgrade? ... Because ruby 2.1!
Example allocation information:
Line number
Number of allocations by object type
Such as:
Running: ./spec/controllers/application_controller/buttons_spec.rb:91
223730 Arrays @ .../activerecord/lib/active_record/result.rb:35
203280 Strings @ .../activerecord/lib/active_record/relation.rb:27
Why upgrade? ... Because ruby 2.1!
Example allocation information:
Line number
Number of allocations by object type
Such as:
Running: ./spec/controllers/application_controller/buttons_spec.rb:91
223730 Arrays @ .../activerecord/lib/active_record/result.rb:35
203280 Strings @ .../activerecord/lib/active_record/relation.rb:27
See Issue 241 and 762
But that's 2.1, let's get to 2.0 first...
Ruby 2.0 features
Faster Rails startup
Optimizations were made to speed up 'require'
Faster Rails startup
Optimizations were made to speed up 'require'
(master) (1.9.3-p545) + time bundle exec rake environmentbundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total
(master) (2.0.0-p576) + time bundle exec rake environmentbundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total
Faster Rails startup
Optimizations were made to speed up 'require'
(master) (1.9.3-p545) + time bundle exec rake environmentbundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total
(master) (2.0.0-p576) + time bundle exec rake environmentbundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total
25% faster loading of rails environment!
Faster Rails startup
Optimizations were made to speed up 'require'
(master) (1.9.3-p545) + time bundle exec rake environmentbundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total
(master) (2.0.0-p576) + time bundle exec rake environmentbundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total
25% faster loading of rails environment!
Most obvious when:
Running tests
Loading Rails console
Keyword arguments
Simplifies conventions:
Accessing option hash values
Default hash values
Optional / can't handle all cases
Keyword arguments
Example: EmsVmware#vm_connect_all
def vm_connect_all(vm, options={}) defaults = { :onStartup => false } options = defaults.merge(options) vm_connect_disconnect_all_connectable_devices( vm, true, options[:onStartup], options[:user_event] )end
Keyword arguments
Example: EmsVmware#vm_connect_all
def vm_connect_all(vm, options={}) defaults = { :onStartup => false } options = defaults.merge(options) vm_connect_disconnect_all_connectable_devices( vm, true, options[:onStartup], options[:user_event] )end
With keyword arguments:
def vm_connect_all(vm, user_event: nil, onStartup: false) vm_connect_disconnect_all_connectable_devices(vm, true, onStartup, user_event)end
Keyword arguments
Invoking methods not changed
Valid on 1.9.3 / 2.0.0:
vm_connect_all(:vm_object1, :onStartup => true, :user_event => "event1")
vm_connect_all(:vm_object2)
vm_connect_all(:vm_object3, :user_event => "event3")
vm_connect_all(:vm_object3, user_event: "event3")
Keyword arguments
Shortcomings and gotchas:
if - valid hash key / invalid keyword argument
Required keyword arguments added in 2.1
http://magazine.rubyist.net/?Ruby200SpecialEn-kwarg
http://robots.thoughtbot.com/ruby-2-keyword-arguments
http://chriszetter.com/blog/2012/11/02/keyword-arguments-in-ruby-2-dot-0/
Module#prepend
Problem: We want to debug a slow method.
Module#prepend
Problem: We want to debug a slow method.
Wrap the method so can time it!
class Parent def run puts "Parent" endend
class Sub < Parent def run puts "Sub" super end
def run_with puts "DebugIt" run_without end
alias_method :run_without, :run alias_method :run, :run_withend
Module#prepend
Using alias_method:
class Parent def run puts "Parent" endend
class Sub < Parent def run puts "Sub" super end
def run_with puts "DebugIt" run_without end
alias_method :run_without, :run alias_method :run, :run_withend
Sub is a class with the slow run method...
Module#prepend
Using alias_method:
class Parent def run puts "Parent" endend
class Sub < Parent def run puts "Sub" super end
def run_with puts "DebugIt" run_without end
alias_method :run_without, :run alias_method :run, :run_withend
Sub is a class with the slow run method...
irb(main):01:0> Sub.new.run
DebugItSubParent
Module#prepend
Using alias_method:
class Parent def run puts "Parent" endend
class Sub < Parent def run puts "Sub" super end
def run_with puts "DebugIt" run_without end
alias_method :run_without, :run alias_method :run, :run_withend
Sub is a class with the slow run method...
irb(main):01:0> Sub.new.run
DebugItSubParent
YAY! But that's really dirty!
(alias_method_chain)
Module#prepend
Using alias_method:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent include DebugIt
def run puts "Sub" super endend
Module#prepend
Using include:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent include DebugIt
def run puts "Sub" super endend
Sub is a class with the slow run method...
Module#prepend
Using include:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent include DebugIt
def run puts "Sub" super endend
Sub is a class with the slow run method...
irb(main):002:0> Sub.ancestors
=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]
Module#prepend
Using include:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent include DebugIt
def run puts "Sub" super endend
Sub is a class with the slow run method...
irb(main):002:0> Sub.ancestors
=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
SubDebugItParent
Module#prepend
Using include:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent include DebugIt
def run puts "Sub" super endend
Sub is a class with the slow run method...
irb(main):002:0> Sub.ancestors
=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
SubDebugItParent
UGH, Sub's method comes first!
Module#prepend
Using include:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent prepend DebugIt
def run puts "Sub" super endend
Module#prepend
Using prepend:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent prepend DebugIt
def run puts "Sub" super endend
irb(main):002:0> Sub.ancestors
=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]
Module#prepend
Using prepend:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent prepend DebugIt
def run puts "Sub" super endend
irb(main):002:0> Sub.ancestors
=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
DebugItSubParent
Module#prepend
Using prepend:
module DebugIt def run puts "DebugIt" super endend
class Parent def run puts "Parent" endend
class Sub < Parent prepend DebugIt
def run puts "Sub" super endend
irb(main):002:0> Sub.ancestors
=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
DebugItSubParent
Ship it!
... but don't forget to call super!
Module#prepend
Using prepend:
Array of symbols: %i and %I
Array of symbols: %i and %I
irb(main):001:0> %i{vmware redhat microsoft}=> [:vmware, :redhat, :microsoft]
Array of symbols: %i and %I
irb(main):001:0> %i{vmware redhat microsoft}=> [:vmware, :redhat, :microsoft]
%I allows interpolation:
Array of symbols: %i and %I
irb(main):001:0> %i{vmware redhat microsoft}=> [:vmware, :redhat, :microsoft]
%I allows interpolation:
irb(main):002:0> prefix = "vm_"=> "vm_"
irb(main):003:0> %I{#{prefix}vmware #{prefix}redhat #{prefix}microsoft}=> [:vm_vmware, :vm_redhat, :vm_microsoft]
Refinements
Goal: localize monkey patches
I'm not going to explain them because:
Ruby's open classes
Many gotchas...
See Charles Nutter (@headius/jruby guy) explanation:http://blog.headius.com/2012/11/refining-ruby.html
Enumerable#lazy
Enumerable methods evaluate left to right
With lazy, chains of enumerations are evaluated right to left
Enumerable#lazy
Enumerable methods evaluate left to right
With lazy, chains of enumerations are evaluated right to left
Ruby may "cheat":
May skip creating intermediate objects
Large collection operations may be optimized
Enumerable#lazy
Example benchmark
(0...1000).select(&:odd?).take(5).to_a
(0...1000).lazy.select(&:odd?).take(5).to_a
What is this doing?
Enumerable#lazy
Example benchmark
(0...1000).select(&:odd?).take(5).to_a
(0...1000).lazy.select(&:odd?).take(5).to_a
What is this doing?
First 5 odd numbers
=> [1, 3, 5, 7, 9]
Enumerable#lazy
require 'benchmark/ips'
Benchmark.ips do |x| x.report("normal") { (0...1000).select(&:odd?).take(5).to_a } x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }end
Enumerable#lazy
require 'benchmark/ips'
Benchmark.ips do |x| x.report("normal") { (0...1000).select(&:odd?).take(5).to_a } x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }end
Calculating ------------------------------------- normal 1539 i/100ms lazy 5778 i/100ms------------------------------------------------- normal 15123.2 (±2.5%) i/s - 76950 in 5.091373s lazy 59797.4 (±4.0%) i/s - 300456 in 5.033123s
See http://patshaughnessy.net/2013/4/3/ruby-2-0-works-hard-so-you-can-be-lazy
__dir__
__dir__ path of the script without the filename
# cat test.rbputs __dir__
# ruby test.rb/Users/joerafaniello/Code/test
__dir__
__dir__ path of the script without the filename
# cat test.rbputs __dir__
# ruby test.rb/Users/joerafaniello/Code/test
We have 984 instances of File.dirname(__FILE__)!
WAT...Why?
Ruby 2.0 breaking changes and deprecations
Note: We're green on travis, so we're getting close...
Objects don't respond_to? to protected methods
Ruby 1.9.3:
respond_to?(symbol) => public and protected methods
respond_to?(symbol, true) => all methods
Objects don't respond_to? to protected methods
Ruby 1.9.3:
respond_to?(symbol) => public and protected methods
respond_to?(symbol, true) => all methods
Ruby 2.0.0
respond_to?(symbol) => public methods only
respond_to?(symbol, true) => all methods
Objects don't respond_to? to protected methods
class Worker protected
def run endend
Objects don't respond_to? to protected methods
class Worker protected
def run endend
Worker.new.respond_to?(:run)1.9.3 => true2.0.0 => false
Objects don't respond_to? to protected methods
class Worker protected
def run endend
Worker.new.respond_to?(:run)1.9.3 => true2.0.0 => false
Pass true as second argument...
Worker.new.respond_to?(:run, true)1.9.3 => true2.0.0 => true
Objects don't respond_to? to protected methods
class Worker protected
def run endend
Worker.new.respond_to?(:run)1.9.3 => true2.0.0 => false
Pass true as second argument...
Worker.new.respond_to?(:run, true)1.9.3 => true2.0.0 => true
See Pull #685 - default_value_for gem
UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"
UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"
Ruby 1.9.3
US-ASCII is the default encoding of ruby scripts
UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"
Ruby 1.9.3
US-ASCII is the default encoding of ruby scripts
Binary string literals become ASCII-8BIT:
UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"
Ruby 1.9.3
US-ASCII is the default encoding of ruby scripts
Binary string literals become ASCII-8BIT:
irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:ASCII-8BIT>
UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"
UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"
Ruby 2.0.0
UTF-8 is the default encoding
UTF-8 is the default character encoding of rubyscripts # cat test.rb FOO = "\222dL\256"
Ruby 2.0.0
UTF-8 is the default encoding
Binary string literals become UTF-8 even if invalid:
irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:UTF-8>irb(main):003:0> FOO.valid_encoding?=> false
UTF-8 is the default character encoding of rubyscripts
So, what's the problem?
Binary strings are used in many places for vm "fleecing"
Invalid UTF-8 encoded strings != raw binary:
UTF-8 is the default character encoding of rubyscripts
So, what's the problem?
Binary strings are used in many places for vm "fleecing"
Invalid UTF-8 encoded strings != raw binary:
irb(main):003:0> FOO.valid_encoding?=> falseirb(main):004:0> FOO == "\222dL\256".force_encoding("ASCII-8BIT")=> false
UTF-8 is the default character encoding of rubyscripts
Solutions:
Force binary on individual Strings:
1.9.3/2.0.0 compatible
Painful on files with many binary string literals
UTF-8 is the default character encoding of rubyscripts
Solutions:
Force binary on individual Strings:
1.9.3/2.0.0 compatible
Painful on files with many binary string literals
# cat test.rbFOO = "\222dL\256".force_encoding("ASCII-8BIT")
UTF-8 is the default character encoding of rubyscripts
Solutions:
Force binary on individual Strings:
1.9.3/2.0.0 compatible
Painful on files with many binary string literals
# cat test.rbFOO = "\222dL\256".force_encoding("ASCII-8BIT")
irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:ASCII-8BIT>irb(main):003:0> FOO.valid_encoding?=> trueirb(main):004:0> FOO == "\222dL\256".force_encoding("ASCII-8BIT")=> true
UTF-8 is the default character encoding of rubyscripts
Solutions:
Use String#b on individual strings
Not compatible with ruby 1.9.3
Copies the String in ASCII-8BIT encoding
Note: String#force_encoding("ASCII-8BIT") modifies the receiver
UTF-8 is the default character encoding of rubyscripts
Solutions:
Add #encoding magic comment at top
1.9.3/2.0.0 compatible
Good option when binary strings are expected
UTF-8 is the default character encoding of rubyscripts
Solutions:
Add #encoding magic comment at top
1.9.3/2.0.0 compatible
Good option when binary strings are expected
# cat test.rb# encoding: US-ASCIIFOO = "\222dL\256"
UTF-8 is the default character encoding of rubyscripts
Solutions:
Add #encoding magic comment at top
1.9.3/2.0.0 compatible
Good option when binary strings are expected
# cat test.rb# encoding: US-ASCIIFOO = "\222dL\256"
irb(main):001:0> require './test'=> trueirb(main):002:0> FOO.encoding=> #<Encoding:ASCII-8BIT>
Descriptors except 0, 1, 2 are closed in childprocesses
Prevents file descriptor leakage
See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041
Descriptors except 0, 1, 2 are closed in childprocesses
Prevents file descriptor leakage
See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041
Example: shared pipe to communicate data between two processes
Descriptors except 0, 1, 2 are closed in childprocesses
Prevents file descriptor leakage
See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041
Example: shared pipe to communicate data between two processes
Use IO#close_on_exec = false
reader, writer = IO.pipe writerfd = writer.fileno my_env["WRITER_FD"] = writerfd.to_s++ writer.close_on_exec = false+ pid = Kernel.spawn(my_env, "ruby #{SERVER_PATH}VixDiskLibServer.rb", [:out, :err] => [LOG_FILE, "a"], :unsetenv_others => true,
String#lines returns an array
Also String#chars, #bytes and #codepoints
Previously, returned enumerators
Above are deprecated for StringIO, IO and friends
String#lines returns an array
Also String#chars, #bytes and #codepoints
Previously, returned enumerators
Above are deprecated for StringIO, IO and friends
StringIO#lines deprecated, so use #each_line instead:
@log_stream.rewind- lines = @log_stream.lines.to_a+ lines = @log_stream.each_line.to_a lines.length.should == 1 line = lines.first.chomp
String#lines returns an array
Also String#chars, #bytes and #codepoints
Previously, returned enumerators
Above are deprecated for StringIO, IO and friends
StringIO#lines deprecated, so use #each_line instead:
@log_stream.rewind- lines = @log_stream.lines.to_a+ lines = @log_stream.each_line.to_a lines.length.should == 1 line = lines.first.chomp
See Pull #714
Ok, great, but are we there yet???
"Fall cleanup" of old code
"...Now I've only been an OpenBSD developer for 11 years, one year less thanthis header has existed, but in that brief time, I've learned a thing or twoabout deleting obsolete code. It doesn't delete itself. And worse, peoplewill continue using it until you force them onto a better path."
http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94
"Fall cleanup" of old code
"...Now I've only been an OpenBSD developer for 11 years, one year less thanthis header has existed, but in that brief time, I've learned a thing or twoabout deleting obsolete code. It doesn't delete itself. And worse, peoplewill continue using it until you force them onto a better path."
http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94
Git is our friend if we really want it back
"Fall cleanup" of old code
Path to ruby 2.0...
"Fall cleanup" of old code
Path to ruby 2.0...
83 commits
565 lines added
5,553 lines deleted
"Fall cleanup" of old code
Path to ruby 2.0...
83 commits
565 lines added
5,553 lines deleted
A good start?
"Fall cleanup" of old code
More "opportunities"
host directory
soap4r/actionwebservice (fork)
handsoap (fork) - note, useful but still forked :-(
ruport (fork)
ziya (fork) - patches rails!
prototype
old rails plugins
old gems
old monkey patches
Slow tests
Tests take 30+ minutes on CI servers
Separate tests
Minimizing setup (database inserts)
Remove invalid/not useful/duplicate tests
Allocation tracing with ruby 2.1
Cut support for 1.9.3 when 2.0 is stable
Building 2.0 appliances
Verified Ruby 2.0 on CentOS appliance:
Appliance startup
SmartState Analysis "fleecing" using vddk
vCenter inventory
Basic reporting
Building 2.0 appliances
Goal: automate building ruby 2.0 appliances
We currently use ruby 1.9.3 through SCL rpms
Not multi-platform
Restricts updating of some gems
Ruby 2.1 is not yet packaged as SCL rpms
Building 2.0 appliances
Solution:
rpms for base CentOS OS
rpms needed to build ruby and compiled gems
libxml2-devel, libxslt-devel, etc.
Building 2.0 appliances
Solution:
rpms for base CentOS OS
rpms needed to build ruby and compiled gems
libxml2-devel, libxslt-devel, etc.
Use ruby-install or ruby-build for building ruby
https://github.com/postmodern/ruby-install
https://github.com/sstephenson/ruby-build
Building 2.0 appliances
Solution:
rpms for base CentOS OS
rpms needed to build ruby and compiled gems
libxml2-devel, libxslt-devel, etc.
Use ruby-install or ruby-build for building ruby
https://github.com/postmodern/ruby-install
https://github.com/sstephenson/ruby-build
Let bundler handle what it does well...
Developer setup
2.0.0 is not much different from 1.9.3:
Install using rvm, ruby-install, or ruby-build
Manage with rvm, rbenv, or chruby
Need guinea pigs to try it and document any issues
Others tools, such as rubymine, may require some configuration
Links
2.0 open issues: https://github.com/ManageIQ/manageiq/labels/ruby%202
2.0 closed issues: https://github.com/ManageIQ/manageiq/issues?q=label%3A%22ruby+2%22+is%3Aclosed
2.0 in depth: http://globaldev.co.uk/2013/03/ruby-2-0-0-in-detail/
2.1 in depth: http://globaldev.co.uk/2014/05/ruby-2-1-in-detail/
Slides available here: https://github.com/jrafanie/manageiq_summit_ruby20
Slides written in markdown using remarkjs: http://remarkjs.com/#1
uestions?