+ All Categories
Home > Documents > Microsoft AI: At the intersection of people and society · Site-specific data Observations,...

Microsoft AI: At the intersection of people and society · Site-specific data Observations,...

Date post: 13-Jun-2020
Category:

Documents
Upload: others
View: 0 times
Download: 0 times
Download Report this document
Share this document with a friend
64
Transcript
Page 1: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 2: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 3: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 4: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 5: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 6: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 7: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Stanford question & answer challenge

Page 8: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 9: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Ethical, legal, societal influences

Page 10: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Qualification problem

All preconditions?

Ramification problem

All effects of action?

Page 11: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 12: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Knowing that you do not know is the best.

Not knowing that you do not know is an illness.

- Laozi, 500-600 BCE

Page 13: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Fang, et al., 2015

Learn about abilities & failures

Performance

Successes & failures

p( fail | E, t)

Confidence

Image

H1

H2

H3

W1

W2

W3

W4

Input s

H3

Caption:

a man holding a tennis

racquet on a tennis court

H1

H2

H3

W1

W2

W3

Input t1

H3

W4

Deep learning about deep learning performance

Page 14: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Reliable predictions of performance: Known unknowns

Grappling with Open-World Complexity

Page 15: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Reliable predictions of performance: Known unknowns

Grappling with Open-World Complexity

Page 16: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 17: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 18: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Grappling with Open-World Complexity

Reliable predictions of performance: Known unknowns

Challenge of unknown unknowns

Page 19: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Expanded real-world testing

Algorithmic portfolios

Failsafe designs

People + machines

Page 20: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

M

training data

M

real-world concepts

x= (𝑓1, … , 𝑓𝑘)

wrong label

high confidence

Conceptual incompleteness

cats

dogs

Lakkaraju, Kamar, Caruana, H, 2017.

Identifying classifier blindspots

Page 21: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

M

training data

M

real-world concepts

x= (𝑓1, … , 𝑓𝑘)

wrong label

high confidence

cats

dogs

How to define & search regions of data space?

How to trade exploration and exploitation? Lakkaraju, Kamar, Caruana, H, 2017.

Identifying classifier blindspots

Conceptual incompleteness

Page 22: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

M

training data

cats

dogs

M

training data

x= (𝑓1, … , 𝑓𝑘)

wrong label

high confidence

Partition

space by

attributes

White CatsWhite Dogs Brown DogsBrown Cats Lakkaraju, Kamar, Caruana, H, 2017.

Identifying classifier blindspots

Page 23: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Transfer learning

Learn from rich simulations

Learn generative models

Page 24: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Hospital A Hospital B

Hospital C

Site-specific data

Observations, definitions

Patients, prevalencies

Covariate dependencies

Transfer learning opportunity

Hospital C

Hospital B

J. Wiens, J. Guttag, H, 2015.

A: Community hosp: 10k pts/yr

B: Acute care & teaching: 15k/yr

C: Major teaching & research: 40k/yr

Page 25: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Site-specific data

Observations, definitions

Patients, prevalencies

Covariate dependencies

Transfer learning opportunity

Hospital A

Hospital C

Hospital B

J. Wiens, J. Guttag, H, 2015.

A: Community hosp: 10k pts/yr

B: Acute care & teaching: 15k/yr

C: Major teaching & research: 40k/yr

Page 26: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

M. Gabel, R. Caruana, M. Philipose, O. Dekel

Less data with better features

ImageNet 1000, 1M photos

Cut off top layer

Page 27: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

M. Gabel, R. Caruana, M. Philipose, O. Dekel

Less data with better features

ImageNet 1000, 1M photos

Cut off top layer

Page 28: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

M. Gabel, R. Caruana, M. Philipose, O. Dekel

Less data with better features

ImageNet 1000, 1M photos

Cut off top layer

Page 29: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Raspberry Pi

Camera

Battery

Page 30: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Trillions of sessions in complex scenarios

Learn & evaluate core competencies

Learn to optimize action plans

Page 31: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Mapping

Planning

Next actions

Map

Plans

Stereo

algorithm

Depth Image

D. Dey, S. Sinha, S. Shah, A. Kapoor

CNN

Page 32: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Mapping

Planning

Next actions

Map

Plans

Stereo

algorithmCNN

Depth Image

D. Dey, S. Sinha, S. Shah, A. Kapoor

Page 33: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Learn expressive generative models

Generalize from minimal training sets

Harness physics

Page 34: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Multilevel variational autoencoder

Learn disentangled representations

Groups of observations latent models

Learning generative models

Vary IDVary style

D. Buchacourt, R. Tomioka, S. Nowozin, 2017

Smooth control over learned latent space

Page 35: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Inject physics to disentangle & generalize

Same?

Kulkarni, Whitney, Kohli & Tenenbaum, 2015

Page 36: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Inject physics to disentangle & generalize

Kulkarni, Whitney, Kohli & Tenenbaum, 2015

Illumination Nod Shake

Page 37: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Illumination Nod Shake

Inject physics to disentangle & generalize

Kulkarni, Whitney, Kohli & Tenenbaum, 2015

Page 38: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

AI attack surfaces

Adversarial machine learning

Self-modification

Page 39: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Attacks on AI Systems

Goodfellow, et al.

Papernot, et al.

“Adverserial machine learning”

Page 40: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

EnvironmentAction

Environment

AI system

e.g., see: Amodei, Olah, et al., 2016

State Perception

ReinforcementReward

Adversarial Attacks & Self-Modification

Page 41: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

EnvironmentAction

Environment

AI system

e.g., see: Amodei, Olah, et al., 2016

State Perception

ReinforcementReward

Adversary

Adversarial Attacks & Self-Modification

Page 42: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

EnvironmentAction

Environment

AI system

e.g., see: Amodei, Olah, et al., 2016

State Perception

ReinforcementReward

Adversary

Action

Adversarial Attacks & Self-Modification

Page 43: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

EnvironmentAction

Environment

AI system

Amodei, Olah, et al., 2016H. 2016

State Perception

ReinforcementReward

Run-time verificationStatic analysis

Reflective analysisEnsure isolation * identify meddling * ensure operational faithfulness

Adversarial Attacks & Self-Modification

Page 44: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Models of people & tasks

Models of complementarity

Coordination of initiative

Page 45: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Models of people & tasks

Actions, services

E1 E2 E3

H1 H2

E4

Predictions about needs, goals

Page 46: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Models of world & people

Predictions about user beliefs

E2 E3

H1 H2

E4

E1 E2 E3

H1 H2

E4

Predictions about world

Actions

H. Barry, 1995

H. , Apacible, Sarin, Liao, 2005

Page 47: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

H. Barry, 1995

Models of world & people

Page 48: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Complementarity

Page 49: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Complementarity

Page 50: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Complementarity

Page 51: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

D. Wang, A. Khosla, R. Gargeya, H. Irshad, A.H. Beck, 2016

Identifying metastatic breast cancer

(Camelyon Grand Challenge 2016)

AI + Expert: 0.5%

85% reduction in errors.

Human is superior

Error: 3.4%

Complementarity

Page 52: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Machine

perception

Human

perception

Machine learning & inference

Kamar, Hacker, H., AAMAS 2012

ComplementarityLabel galaxies in Sloan Digital Sky Survey

(Galaxy Zoo)

Page 53: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

~453 features

Machine learning & inference

Machine

perceptionHuman

perception

Kamar, Hacker, H., AAMAS 2012

ComplementarityLabel galaxies in Sloan Digital Sky Survey

(Galaxy Zoo)

Page 54: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

~453 features

Machine learning & inference

Ideal fusion, stopping

Machine

perceptionHuman

perception

Kamar, Hacker, H., AAMAS 2012

Complementarity Full accuracy: 47% of human effort

95% accuracy: 23% of human effort

Page 55: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Designs for mix of initiatives

Machine learning & inference

Human

cognition

Machine

intelligence

Page 56: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

C.E. Reiley, et al.

Initiative: Recognizing human goals, stateRecognizing intention

Page 57: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Padoy & Hager. ICRA 2011

van den Berg, et al, ICRA, 2010

Coordination of initiative

Page 58: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Trustworthiness and safety

Fairness, accuracy, transparency

Ethical and legal aspects of autonomy

Jobs and economy

Page 59: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 60: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Bernard Parker: rated high risk Dylan Fugett: rated low risk.

Page 61: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 62: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

March 2017

A. Howard, C. Zhang, H., 2017

Machine learning “contact lens” for children

Page 63: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity
Page 64: Microsoft AI: At the intersection of people and society · Site-specific data Observations, definitions Patients, prevalencies Covariate dependencies Transfer learning opportunity

Science & engineering

Human-AI collaboration

AI, people, and society

Much to do


Recommended
Covariate-Adjusted Regressionanson.ucdavis.edu/~mueller/zcar11-21.pdf · Consider the simple linear regression model Yi = ... estimated using covariate-adjusted regression by applying

Covariate-Adjusted Regressionanson.ucdavis.edu/~mueller/zcar11-21.pdf · Consider the simple linear regression model Yi = ... estimated using covariate-adjusted regression by applying

Documents

Dependencies, dependencies, dependencies

Dependencies, dependencies, dependencies

Software

A class of covariate-dependent spatiotemporal covariance functions

A class of covariate-dependent spatiotemporal covariance functions

Documents

E cient adaptive covariate modelling for extremes

E cient adaptive covariate modelling for extremes

Documents

Inference under Covariate-Adaptive Randomizationfaculty.wcas.northwestern.edu/~iac879/wp/covariateadaptive.pdfInference under Covariate-Adaptive Randomization Federico A. Bugni Department

Inference under Covariate-Adaptive Randomizationfaculty.wcas.northwestern.edu/~iac879/wp/covariateadaptive.pdfInference under Covariate-Adaptive Randomization Federico A. Bugni Department

Documents

Methods for covariate adjustment in cost-effectiveness ... · Methods for covariate adjustment in cost-effectiveness ... Methods for covariate adjustment in cost-effectiveness analyses

Methods for covariate adjustment in cost-effectiveness ... · Methods for covariate adjustment in cost-effectiveness ... Methods for covariate adjustment in cost-effectiveness analyses

Documents

Nonparametric covariate-adjusted response-adaptive design ... fileSubmitted to the Annals of Statistics arXiv: arXiv:0000.0000 NONPARAMETRIC COVARIATE-ADJUSTED RESPONSE-ADAPTIVE DESIGN

Nonparametric covariate-adjusted response-adaptive design ... fileSubmitted to the Annals of Statistics arXiv: arXiv:0000.0000 NONPARAMETRIC COVARIATE-ADJUSTED RESPONSE-ADAPTIVE DESIGN

Documents

Reflections · 2018. 1. 12. · Site-specific evidence Site-specific pts, prevalencies Site covariate dependencies Site A Site B Site C Hospital A Community hospital: 180 beds, 10,000

Reflections · 2018. 1. 12. · Site-specific evidence Site-specific pts, prevalencies Site covariate dependencies Site A Site B Site C Hospital A Community hospital: 180 beds, 10,000

Documents

Dependencies between players in Boolean games - IRITPhilippe.Muller/Seminaire_4/bonzon_261007.pdf · Introduction Boolean games Dependencies between players Conclusion Dependencies

Dependencies between players in Boolean games - IRITPhilippe.Muller/Seminaire_4/bonzon_261007.pdf · Introduction Boolean games Dependencies between players Conclusion Dependencies

Documents

Stanford typed dependencies manual - SourceForgegrammarscope.sourceforge.net/dependencies-manual.pdfStanford typed dependencies manual ... rather than the phrase structure representations

Stanford typed dependencies manual - SourceForgegrammarscope.sourceforge.net/dependencies-manual.pdfStanford typed dependencies manual ... rather than the phrase structure representations

Documents

Covariate balancing propensity score - Wiley Online Library

Covariate balancing propensity score - Wiley Online Library

Documents

Covariate-Dependent Nonparametric Mixture Models

Covariate-Dependent Nonparametric Mixture Models

Documents

C3 Enterprise Risk Management and BCM · The Intersection of Enterprise-wide Risk Management ... • Enterprise risk management policy • Risk committee charter ... Dependencies

C3 Enterprise Risk Management and BCM · The Intersection of Enterprise-wide Risk Management ... • Enterprise risk management policy • Risk committee charter ... Dependencies

Documents

The Covariate-Adjusted Frequency Plot for the Rasch ... 4 cov_freq_final_TS-057.pdf · The Covariate-Adjusted Frequency Plot for the Rasch ... To apply the covariate-adjusted frequency

The Covariate-Adjusted Frequency Plot for the Rasch ... 4 cov_freq_final_TS-057.pdf · The Covariate-Adjusted Frequency Plot for the Rasch ... To apply the covariate-adjusted frequency

Documents

Covariate Selection for Association Screening in Multi ...

Covariate Selection for Association Screening in Multi ...

Documents

Improving Covariate Balancing Propensity Score: A Doubly ...

Improving Covariate Balancing Propensity Score: A Doubly ...

Documents

Analysis of Kernel Mean Matching under Covariate Shift

Analysis of Kernel Mean Matching under Covariate Shift

Documents

pscore2: Stata module to enforce covariate balance

pscore2: Stata module to enforce covariate balance

Documents