29
Mock Your RNGs Lessons in tes)ng Stan, a probabilis)c programming language IBM AI Systems Day. Oct 3, 2018. Daniel Lee Stan Development Team / Generable
Mock Your RNGsLessons in tes)ng Stan,
a probabilis)c programming language
IBM AI Systems Day. Oct 3, 2018. Daniel LeeStan Development Team / Generable
Who's heard of Stan?(we usually don't run in AI circles)
What is Stan?
Language for Sta;s;cal Models• Goal: specify sta)s)cal models p(✓, x)
data
Language for Sta;s;cal Modelsp(✓, x)
dataparameters
• Goal: specify sta)s)cal models
Language for Sta;s;cal Modelsp(✓, x)
dataparametersmodel
• Goal: specify sta)s)cal models
Language for Sta;s;cal Models• Goal: specify sta)s)cal models
• Probabilis)c programming language (has its own grammar)
• impera)ve, sta)cally typed
• Turing complete
• User can specify any differen'able joint probability distribu)on func)onover data and parameters
p(✓, x)dataparametersmodel
Example: Hello World
Example: Logis;c Regression
Language defines the sta;s;cal modelp(✓, x)
Inference algorithms use p(✓, x)▸ Bayesian inference; Markov Chain Monte Carlo (MCMC)
▸ Approximate Bayesian inference
▸ Op)miza)on
▸ Bayesian inference; Markov Chain Monte Carlo (MCMC)
▸ approximated with
▸ Approximate Bayesian inference
▸ ex: where
▸ Op)miza)on
▸ (only holds when there's a single op)ma)
p(✓ |x) {✓(1), ✓(2), . . . , ✓(N)}<latexit sha1_base64="FpPnCNZqoMHXD02O4ilwteaVslg=">AAACHHicbVDLSgMxFM3UV62vqks3wSK0IGWmFNRdwU1XUsHaQmcsmTRtQzMPkjtCGeZH3PgrblyouHEh+Dem00G09UDg3HPu5eYeNxRcgWl+GbmV1bX1jfxmYWt7Z3evuH9wq4JIUtamgQhk1yWKCe6zNnAQrBtKRjxXsI47uZz5nXsmFQ/8G5iGzPHIyOdDTgloqV+s27ENYwbkLi5bleQU/1S1tBKDANQv9aqS2Em/WDKrZgq8TKyMlFCGVr/4YQ8CGnnMByqIUj3LDMGJiQROBUsKdqRYSOiEjFhPU594TDlxel2CT7QywMNA6ucDTtXfEzHxlJp6ru70CIzVojcT//N6EQzPnZj7YQTMp/NFw0hgCPAsKjzgklEQU00IlVz/FdMxkYSCDrSgQ7AWT14m7Vr1ompe10uNZpZGHh2hY1RGFjpDDdRELdRGFD2gJ/SCXo1H49l4M97nrTkjmzlEf2B8fgOhkKCe</latexit><latexit sha1_base64="FpPnCNZqoMHXD02O4ilwteaVslg=">AAACHHicbVDLSgMxFM3UV62vqks3wSK0IGWmFNRdwU1XUsHaQmcsmTRtQzMPkjtCGeZH3PgrblyouHEh+Dem00G09UDg3HPu5eYeNxRcgWl+GbmV1bX1jfxmYWt7Z3evuH9wq4JIUtamgQhk1yWKCe6zNnAQrBtKRjxXsI47uZz5nXsmFQ/8G5iGzPHIyOdDTgloqV+s27ENYwbkLi5bleQU/1S1tBKDANQv9aqS2Em/WDKrZgq8TKyMlFCGVr/4YQ8CGnnMByqIUj3LDMGJiQROBUsKdqRYSOiEjFhPU594TDlxel2CT7QywMNA6ucDTtXfEzHxlJp6ru70CIzVojcT//N6EQzPnZj7YQTMp/NFw0hgCPAsKjzgklEQU00IlVz/FdMxkYSCDrSgQ7AWT14m7Vr1ompe10uNZpZGHh2hY1RGFjpDDdRELdRGFD2gJ/SCXo1H49l4M97nrTkjmzlEf2B8fgOhkKCe</latexit><latexit sha1_base64="FpPnCNZqoMHXD02O4ilwteaVslg=">AAACHHicbVDLSgMxFM3UV62vqks3wSK0IGWmFNRdwU1XUsHaQmcsmTRtQzMPkjtCGeZH3PgrblyouHEh+Dem00G09UDg3HPu5eYeNxRcgWl+GbmV1bX1jfxmYWt7Z3evuH9wq4JIUtamgQhk1yWKCe6zNnAQrBtKRjxXsI47uZz5nXsmFQ/8G5iGzPHIyOdDTgloqV+s27ENYwbkLi5bleQU/1S1tBKDANQv9aqS2Em/WDKrZgq8TKyMlFCGVr/4YQ8CGnnMByqIUj3LDMGJiQROBUsKdqRYSOiEjFhPU594TDlxel2CT7QywMNA6ucDTtXfEzHxlJp6ru70CIzVojcT//N6EQzPnZj7YQTMp/NFw0hgCPAsKjzgklEQU00IlVz/FdMxkYSCDrSgQ7AWT14m7Vr1ompe10uNZpZGHh2hY1RGFjpDDdRELdRGFD2gJ/SCXo1H49l4M97nrTkjmzlEf2B8fgOhkKCe</latexit>
p̂(✓ |x) ⇡ q(�̂)
✓̂ = argmax✓
p(✓, x)
Inference algorithms use p(✓, x)
�̂ = argmin�
DKL (q(✓ |�) || p(✓, x))<latexit sha1_base64="YU71ZQxTrmorMb2/OGsQFcXUFYA=">AAACZ3icbVBtaxQxEM5uq9arL1urRfBL9BCuUI69UrAVhIJ+EFqkgmcLl+PI5mZvQ5PsNpkVj72l/7Hf/AF+8VeYvS6irQMZHp5nZjLzJIWSDuP4RxCurN65e2/tfmf9wcNHj6ONJ19dXloBQ5Gr3J4l3IGSBoYoUcFZYYHrRMFpcv6+0U+/gXUyN19wXsBY85mRqRQcPTWJLlnGsWJFJmv6jrLSTH0xtEzFNMfM6orbmZamrikD4wougH6Y/BGPjuuaKUixd9FjmAFytkMX1KdmyDZlb+li0eSilXfod89aOctwexJ14368DHobDFrQJW2cTKIrNs1FqcGgUNy50SAucOw3RCkU1B1WOvAbnvMZjDw0XIMbV0ujavraM1Oa5tY/g3TJ/t1Rce3cXCe+sjnO3dQa8n/aqMR0f1xJU5QIRlx/lJaKYk4b1+lUWhCo5h5wYaXflYqMWy7Q293xJgxunnwbDHf7B/3481738FPrxhp5QV6RHhmQN+SQfCQnZEgE+RmsB0+DZ8GvMAq3wufXpWHQ9mySfyJ8+RtxYri0</latexit><latexit sha1_base64="YU71ZQxTrmorMb2/OGsQFcXUFYA=">AAACZ3icbVBtaxQxEM5uq9arL1urRfBL9BCuUI69UrAVhIJ+EFqkgmcLl+PI5mZvQ5PsNpkVj72l/7Hf/AF+8VeYvS6irQMZHp5nZjLzJIWSDuP4RxCurN65e2/tfmf9wcNHj6ONJ19dXloBQ5Gr3J4l3IGSBoYoUcFZYYHrRMFpcv6+0U+/gXUyN19wXsBY85mRqRQcPTWJLlnGsWJFJmv6jrLSTH0xtEzFNMfM6orbmZamrikD4wougH6Y/BGPjuuaKUixd9FjmAFytkMX1KdmyDZlb+li0eSilXfod89aOctwexJ14368DHobDFrQJW2cTKIrNs1FqcGgUNy50SAucOw3RCkU1B1WOvAbnvMZjDw0XIMbV0ujavraM1Oa5tY/g3TJ/t1Rce3cXCe+sjnO3dQa8n/aqMR0f1xJU5QIRlx/lJaKYk4b1+lUWhCo5h5wYaXflYqMWy7Q293xJgxunnwbDHf7B/3481738FPrxhp5QV6RHhmQN+SQfCQnZEgE+RmsB0+DZ8GvMAq3wufXpWHQ9mySfyJ8+RtxYri0</latexit><latexit sha1_base64="YU71ZQxTrmorMb2/OGsQFcXUFYA=">AAACZ3icbVBtaxQxEM5uq9arL1urRfBL9BCuUI69UrAVhIJ+EFqkgmcLl+PI5mZvQ5PsNpkVj72l/7Hf/AF+8VeYvS6irQMZHp5nZjLzJIWSDuP4RxCurN65e2/tfmf9wcNHj6ONJ19dXloBQ5Gr3J4l3IGSBoYoUcFZYYHrRMFpcv6+0U+/gXUyN19wXsBY85mRqRQcPTWJLlnGsWJFJmv6jrLSTH0xtEzFNMfM6orbmZamrikD4wougH6Y/BGPjuuaKUixd9FjmAFytkMX1KdmyDZlb+li0eSilXfod89aOctwexJ14368DHobDFrQJW2cTKIrNs1FqcGgUNy50SAucOw3RCkU1B1WOvAbnvMZjDw0XIMbV0ujavraM1Oa5tY/g3TJ/t1Rce3cXCe+sjnO3dQa8n/aqMR0f1xJU5QIRlx/lJaKYk4b1+lUWhCo5h5wYaXflYqMWy7Q293xJgxunnwbDHf7B/3481738FPrxhp5QV6RHhmQN+SQfCQnZEgE+RmsB0+DZ8GvMAq3wufXpWHQ9mySfyJ8+RtxYri0</latexit>
▸ Bayesian inference; Markov Chain Monte Carlo (MCMC)
▸ approximated with
▸ Approximate Bayesian inference
▸ ex: where
▸ Op)miza)on
▸ (only holds when there's a single op)ma)
p(✓ |x) {✓(1), ✓(2), . . . , ✓(N)}<latexit sha1_base64="FpPnCNZqoMHXD02O4ilwteaVslg=">AAACHHicbVDLSgMxFM3UV62vqks3wSK0IGWmFNRdwU1XUsHaQmcsmTRtQzMPkjtCGeZH3PgrblyouHEh+Dem00G09UDg3HPu5eYeNxRcgWl+GbmV1bX1jfxmYWt7Z3evuH9wq4JIUtamgQhk1yWKCe6zNnAQrBtKRjxXsI47uZz5nXsmFQ/8G5iGzPHIyOdDTgloqV+s27ENYwbkLi5bleQU/1S1tBKDANQv9aqS2Em/WDKrZgq8TKyMlFCGVr/4YQ8CGnnMByqIUj3LDMGJiQROBUsKdqRYSOiEjFhPU594TDlxel2CT7QywMNA6ucDTtXfEzHxlJp6ru70CIzVojcT//N6EQzPnZj7YQTMp/NFw0hgCPAsKjzgklEQU00IlVz/FdMxkYSCDrSgQ7AWT14m7Vr1ompe10uNZpZGHh2hY1RGFjpDDdRELdRGFD2gJ/SCXo1H49l4M97nrTkjmzlEf2B8fgOhkKCe</latexit><latexit sha1_base64="FpPnCNZqoMHXD02O4ilwteaVslg=">AAACHHicbVDLSgMxFM3UV62vqks3wSK0IGWmFNRdwU1XUsHaQmcsmTRtQzMPkjtCGeZH3PgrblyouHEh+Dem00G09UDg3HPu5eYeNxRcgWl+GbmV1bX1jfxmYWt7Z3evuH9wq4JIUtamgQhk1yWKCe6zNnAQrBtKRjxXsI47uZz5nXsmFQ/8G5iGzPHIyOdDTgloqV+s27ENYwbkLi5bleQU/1S1tBKDANQv9aqS2Em/WDKrZgq8TKyMlFCGVr/4YQ8CGnnMByqIUj3LDMGJiQROBUsKdqRYSOiEjFhPU594TDlxel2CT7QywMNA6ucDTtXfEzHxlJp6ru70CIzVojcT//N6EQzPnZj7YQTMp/NFw0hgCPAsKjzgklEQU00IlVz/FdMxkYSCDrSgQ7AWT14m7Vr1ompe10uNZpZGHh2hY1RGFjpDDdRELdRGFD2gJ/SCXo1H49l4M97nrTkjmzlEf2B8fgOhkKCe</latexit><latexit sha1_base64="FpPnCNZqoMHXD02O4ilwteaVslg=">AAACHHicbVDLSgMxFM3UV62vqks3wSK0IGWmFNRdwU1XUsHaQmcsmTRtQzMPkjtCGeZH3PgrblyouHEh+Dem00G09UDg3HPu5eYeNxRcgWl+GbmV1bX1jfxmYWt7Z3evuH9wq4JIUtamgQhk1yWKCe6zNnAQrBtKRjxXsI47uZz5nXsmFQ/8G5iGzPHIyOdDTgloqV+s27ENYwbkLi5bleQU/1S1tBKDANQv9aqS2Em/WDKrZgq8TKyMlFCGVr/4YQ8CGnnMByqIUj3LDMGJiQROBUsKdqRYSOiEjFhPU594TDlxel2CT7QywMNA6ucDTtXfEzHxlJp6ru70CIzVojcT//N6EQzPnZj7YQTMp/NFw0hgCPAsKjzgklEQU00IlVz/FdMxkYSCDrSgQ7AWT14m7Vr1ompe10uNZpZGHh2hY1RGFjpDDdRELdRGFD2gJ/SCXo1H49l4M97nrTkjmzlEf2B8fgOhkKCe</latexit>
p̂(✓ |x) ⇡ q(�̂)
✓̂ = argmax✓
p(✓, x)
Inference algorithms use p(✓, x)
�̂ = argmin�
DKL (q(✓ |�) || p(✓, x))<latexit sha1_base64="YU71ZQxTrmorMb2/OGsQFcXUFYA=">AAACZ3icbVBtaxQxEM5uq9arL1urRfBL9BCuUI69UrAVhIJ+EFqkgmcLl+PI5mZvQ5PsNpkVj72l/7Hf/AF+8VeYvS6irQMZHp5nZjLzJIWSDuP4RxCurN65e2/tfmf9wcNHj6ONJ19dXloBQ5Gr3J4l3IGSBoYoUcFZYYHrRMFpcv6+0U+/gXUyN19wXsBY85mRqRQcPTWJLlnGsWJFJmv6jrLSTH0xtEzFNMfM6orbmZamrikD4wougH6Y/BGPjuuaKUixd9FjmAFytkMX1KdmyDZlb+li0eSilXfod89aOctwexJ14368DHobDFrQJW2cTKIrNs1FqcGgUNy50SAucOw3RCkU1B1WOvAbnvMZjDw0XIMbV0ujavraM1Oa5tY/g3TJ/t1Rce3cXCe+sjnO3dQa8n/aqMR0f1xJU5QIRlx/lJaKYk4b1+lUWhCo5h5wYaXflYqMWy7Q293xJgxunnwbDHf7B/3481738FPrxhp5QV6RHhmQN+SQfCQnZEgE+RmsB0+DZ8GvMAq3wufXpWHQ9mySfyJ8+RtxYri0</latexit><latexit sha1_base64="YU71ZQxTrmorMb2/OGsQFcXUFYA=">AAACZ3icbVBtaxQxEM5uq9arL1urRfBL9BCuUI69UrAVhIJ+EFqkgmcLl+PI5mZvQ5PsNpkVj72l/7Hf/AF+8VeYvS6irQMZHp5nZjLzJIWSDuP4RxCurN65e2/tfmf9wcNHj6ONJ19dXloBQ5Gr3J4l3IGSBoYoUcFZYYHrRMFpcv6+0U+/gXUyN19wXsBY85mRqRQcPTWJLlnGsWJFJmv6jrLSTH0xtEzFNMfM6orbmZamrikD4wougH6Y/BGPjuuaKUixd9FjmAFytkMX1KdmyDZlb+li0eSilXfod89aOctwexJ14368DHobDFrQJW2cTKIrNs1FqcGgUNy50SAucOw3RCkU1B1WOvAbnvMZjDw0XIMbV0ujavraM1Oa5tY/g3TJ/t1Rce3cXCe+sjnO3dQa8n/aqMR0f1xJU5QIRlx/lJaKYk4b1+lUWhCo5h5wYaXflYqMWy7Q293xJgxunnwbDHf7B/3481738FPrxhp5QV6RHhmQN+SQfCQnZEgE+RmsB0+DZ8GvMAq3wufXpWHQ9mySfyJ8+RtxYri0</latexit><latexit sha1_base64="YU71ZQxTrmorMb2/OGsQFcXUFYA=">AAACZ3icbVBtaxQxEM5uq9arL1urRfBL9BCuUI69UrAVhIJ+EFqkgmcLl+PI5mZvQ5PsNpkVj72l/7Hf/AF+8VeYvS6irQMZHp5nZjLzJIWSDuP4RxCurN65e2/tfmf9wcNHj6ONJ19dXloBQ5Gr3J4l3IGSBoYoUcFZYYHrRMFpcv6+0U+/gXUyN19wXsBY85mRqRQcPTWJLlnGsWJFJmv6jrLSTH0xtEzFNMfM6orbmZamrikD4wougH6Y/BGPjuuaKUixd9FjmAFytkMX1KdmyDZlb+li0eSilXfod89aOctwexJ14368DHobDFrQJW2cTKIrNs1FqcGgUNy50SAucOw3RCkU1B1WOvAbnvMZjDw0XIMbV0ujavraM1Oa5tY/g3TJ/t1Rce3cXCe+sjnO3dQa8n/aqMR0f1xJU5QIRlx/lJaKYk4b1+lUWhCo5h5wYaXflYqMWy7Q293xJgxunnwbDHf7B/3481738FPrxhp5QV6RHhmQN+SQfCQnZEgE+RmsB0+DZ8GvMAq3wufXpWHQ9mySfyJ8+RtxYri0</latexit>
Interfaces• CmdStan, RStan, PyStan
• C++ API
• C++ automa)c differen)a)on library
• RStanArm, brms, prophet, …
Stan: mc-stan.org• Language
• Inference algorithms
• Interfaces
• Open Source:
• BSD: lang, math, CmdStan
• GPL: RStan, PyStan
• github.com/stan-dev
If you can't remember anything else in this talk…
Mock your random number generators
Tes;ng challenges in Stan• Inference algorithms are randomized algorithms
Markov chain Monte Carlo (MCMC)
• Run the same input with a different seed, the output changes
• These algorithms have nice asympto)c proper)es,but… we don't have asympto)c )me
Tes;ng challenges in Stan• Inference algorithms are randomized algorithms
Markov chain Monte Carlo (MCMC)
• Run the same input with a different seed, the output changes
• These algorithms have nice asympto)c proper)es,but… we don't have asympto)c )me
• Stan is pre\y complex (~100k sloc, ~250k test sloc)
• Stan programs transpiled to C++
• automa)c differen)a)on library: heavily templated C++
• custom deriva)ves for speed
Tes;ng is…• Tes)ng so_ware: tough
Tes;ng is…• Tes)ng so_ware: tough
• Tes)ng a programming language: tougher
Tes;ng is…• Tes)ng so_ware: tough
• Tes)ng a programming language: tougher
• Tes)ng randomized algorithms that operate on a programming language… priceless
Things we know• Computers aren't random
(either hardware failure or unsafe memory access)
• use of pseudo random number generators
• So_ware: modularity combats complexity
• How do we extend this to tes)ngrandomized algorithms?
Mock your RNGs• Mock your random number generators
• Test the randomized algorithm with a known sequence of (not so random any more) random numbers
• Reproducible, easier to break down code
• Acts like any other unit test
• (Beware… IEEE sfloa)ng point behavior can s)ll get you)
• (Also test asympto)c proper)es for correctness)
Breakthroughs in stats compu;ng• Inference algorithms
• Autodiff
250 dimensional Normal distribu;on
Hoffman and Gelman. "The No-U-Turn Sampler." JLMR. 2014.
2d projection
1000 draws
Quick aside: HMC is not enough• Hamiltonian Monte Carlo isn't enough
Hoffman and Gelman. "The No-U-Turn Sampler." JLMR. 2014.
Autodiff comparison• For open-source C++ packages: Stan is fastest (for gradients),
most general (func)ons supported), and most easily extensible (simple OO)
