Report by Dr. Owen Churches, Churchill Fellow

2018 Churchill Fellowship to create fairness and accountability in the use of government decision making algorithms

Awarded by The Winston Churchill Memorial Trust

1

THE WINSTON CHURCHILL MEMORIAL TRUST

Report by Dr. Owen Churches, Churchill Fellow 2018 Churchill Fellowship to create fairness and accountability in the use of government decision making algorithms I understand that the Churchill Trust may publish this Report, either in hard copy or on the internet or both, and consent to such publication. I indemnify the Churchill Trust against any loss, costs or damages it may suffer arising out of any claim or proceedings made against the Trust in respect of or arising out of the publication of any Report Submitted to the Trust and that the Trust places on a website for access over the internet. I also warrant that my Final Report is original and does not infringe the copyright of any person, or contain anything which is, or the incorporation of which into the Final Report is, actionable for defamation, a breach of any privacy law or obligation, breach of confidence, contempt of court, passing-off or contravention of any other private right or of any law. Signed

Date 25 July 2019

2

For Genevieve

3

We are both storytellers. Lying on our backs, we look up at the night sky. This is where stories began, under the aegis of that multitude of stars which at night filch certitudes and sometimes return them as faith. Those who first invented and then named the constellations were storytellers. Tracing an imaginary line between a cluster of stars gave them an image and an identity. The stars threaded on that line were like events threaded on a narrative. Imagining the constellations did not of course change the stars, nor did it change the black emptiness that surrounds them. What it changed was the way people read the night sky.

John Berger, from And Our Faces, My Heart, Brief as Photos We astronomers are nomads, Merchants, circus people, All the earth our tent. We are industrious. We breed enthusiasms, Honour our responsibility to awe. But the universe has moved a long way off. Sometimes, I confess, Starlight seems too sharp, And like the moon I bend my face to the ground, To the small patch where each foot falls, Before it falls, And I forget to ask questions, And only count things.

Rebecca Elson, We Astronomers

4

Acknowledgements When I was in primary school I wanted to be an astro-physicist. I watched a lot of Carl Sagan documentaries and spent a great deal of time thinking about space. Needless to say, I have taken a different path. But the two pieces quoted at the start of this report hold truths for me that I carry with me still because they transcend the topics of space and stars. Both Berger and Elson make the point that looking at nature is a creative act. That we are active agents in the discovery of meaningful information about the world. This brings with is an enormous freedom and excitement. But also an enormous responsibility to use this creative opportunity for good. So, I would like to thank all the teachers, colleagues and friends I have had who have reminded me to lift up my head, to appreciate the wonderful opportunity I have and to encourage me to tell the stories of my discoveries. Thank you to the Winston Churchill Memorial Trust for this opportunity. For choosing this project and for choosing me as a fellow to carry it out. It has been a great privilege and I take seriously the responsibility that this opportunity brings with it. Thank you to the South Australian Department of Education for supporting this work by continuing my salary while I was on the Fellowship. Thank you to my workplace, the Child Death and Serious Injury Review Committee. In particular, thank you to my manager, Dr. Sharyn Watts for encouraging me to apply and for supporting me at every turn. Thank you to my mentor Donna Mayhew. Your support of me and knowledge of government has been invaluable and you company supremely enjoyable. Thank you to all the people I have met along the journey so far. Your inspiration, dedication and knowledge has made this an exciting and worthwhile process. Most of all, thank you to my family. Spending these six weeks overseas together was wonderful.

5

Contact details email: ofchurches@gmail.com twitter: @owenchurches blog: ofchurches.rbind.io

Keywords Artificial intelligence, ethics, government, regulation, decision making

6

Contents Executive summary 8

Author and title 8 Background 8 Intended Audience 8 Highlights 9 Conclusions and recommendations 9

Recommendation 1: A professional network for public sector data professionals 10 Recommendation 2: A Chief Data Officer in each Department 10 Recommendation 3: No male only technology panels 10 Recommendation 4: Data skills for the executive level 11

Notes 12 A note on language 12 Practicing what I preach 12 A note on methodology 12

Background 13

Itinerary 16 Itinerary table 16 Itinerary timeline 18 Itinerary map 19

Themes 20 Fixing the technology 21

Data ownership 21 Which intelligence are we talking about? 22

Necessary but insufficient 23 Principles, principles everywhere 23 The “R” word 24 Slowing down 25

What decisions shall we automate? 26 Just the same only faster 26 Governments exist with the consent of the governed 27 Who’s costs and benefits 31

Building a culture with AI 31 Where is this work done? 31 Who owns the data? 32 For the love of data 33 Understanding 34

Learning from history 36 Background 36

7

Fisher's innovation 36 Locations visited 37

Bletchley Park 37 Gonville & Caius College 38 Broad Street Pump 38 St Thomas' Hospital 39 Bunhill Fields Burial Ground 39 Drapers Hall 40

Map of locations visited 41

Outcomes of this Fellowship so far 42 Installation at MOD 42

Recommendation engines 42 The finished product 43

public_sectR 43 Lower costs 44 Efficient oversight 44

AI Ethics Bookclub 44

Recommendations 46 A professional network for public sector data professionals 46

Context 46 Implementation plan 46

A Chief Data Officer in each Department 47 Context 47 Implementation plan 48

No male only technology panels 49 Context 49 Implementation plan 50

Data skills for the executive level 51 Context 51 Implementation Plan 51

Conclusion 53 It would have all the notes 53 It would be objective 53 It would do it quickly 53 Coda 54

8

Executive summary

Author and title Dr. Owen Churches 2018 Churchill Fellowship to create fairness and accountability in the use of government decision making algorithms

Background I am a statistician working in the public service in South Australia. I am proud of my position and believe strongly that the skillful analysis of Government data leads to better decisions, lower costs and a quicker execution of the Government’s aims in servicing the people of South Australia. I joined the public sector in 2017 and frequently found myself in meetings where the topic of data driven decisions and automated process was raised. But whether the tenor of these conversations was overly optimistic or adamantly averse, I felt that it was inadequately informed by the gains made in data policy and academic research around Australia and around the world. I was ken to see more use made of the Government’s data assets. However, I realised that a rush to implement decision making algorithms in South Australia could create brittle systems that would fail the test society would necessarily put them to. That the outcomes of the algorithms may be unfair and their makers unaccountable. And that the public and political class would both be off-put by these poorly designed processes. And that all of this would set back the useful work that could be done with data in government. I applied for and completed this Fellowship in the hope that I could create a new, lasting approach to using data for good in the Government of South Australia.

Intended Audience Everyone is impacted by the changed terms of rights and responsibilities that have accompanied the rise of automated decision making systems in government. As such, the audience for this report is necessarily broad. The final destination for the message from this report is the senior decision making level of government. In particular, there are recommendations that could be implemented by the chief executives of each Department in the South Australian Government, the Commissioner for Public Sector Employment and the Chief Scientist for South Australia. However, for the message to have an impact at this level I have written it hoping it will also find a ready audience at the middle levels of the public service, in academia and amongst the technologically and politically interested sectors of society.

9

Highlights Each one of the interviews I was granted and seminars I attended was amazing and added unique information to my knowledge about government decision making algorithms. The opportunity to learn from artificial intelligence engineers at Google, philosophers at the University of Cambridge and governance experts at the Open Data Institute was incredible. I will maintain these contacts because I realise that the highlights of this Fellowship are only just beginning. Since arriving back in Australia ten weeks ago I have been invited to speak on five different panels on ethical AI and data governance. This is a topic that is only just beginning to gain the interest of the public and I will use the platform that this Fellowship has given me to keep advocating for improvement.

Conclusions and recommendations The challenges of using data for good, of building fair and accountable systems for data driven decision making in government are numerous. Each of the experts I met with for this Fellowship had a different approach to a different part of this puzzle. The sheer number of dimensions on which experts in this field are working is dizzying. There are computationally focused solutions to the legal challenges, educationally focused solutions to the philosophical challenges, and socially focused solutions to the mathematical challenges. Hence, the four recommendations I give below are not a complete to-do list after which Government use of algorithmic decision making will be totally fair and accountable. Nor are they the only areas for improvement that I discovered throughout the course of this Fellowship. Rather, I chose these four recommendations because they were exceptional across a number of number of dimensions. Each recommendation addresses an underlying theme from the Fellowship, creates systems that will be self sustaining and is likely to lead to further improvements. Most of all, each recommendation is ambitious. I am under no illusions about the difficulty of implementing any of these recommendations. However, I am dedicated to investing my energy and recruiting others to help bring them about. Finally, I do not propose these recommendations as a means to criticize the status quo. There are wonderful, capable, bright and hard working data science professionals in the public sector of South Australia. They are supported by strong systems that help data flow and provide exceptional oversight. I humbly propose these recommendations because things can always be better. So, here are four ways to help create fairness and accountability in the use of government decision making algorithms in South Australia.

10

Recommendation 1: A professional network for public sector data professionals Like other professional groups such as teachers and doctors, statisticians and data analysts in the public sector face a continuous need to update their skills and techniques with the current best practice. I recommend that the Office for the Commissioner for Public Sector Employment implement a similar community of practice for data professionals. It would help to house this outside of any particular department to aide in the cross pollination of ideas and to provide an opportunity to avoid individual groups becoming isolated and stagnant in their approach to data analysis. It would help immeasurably to have peers in government who could help answer questions about the nature of particular data sets as well as suggest analytic strategies. I have already made a start to this recommendation by creating the public_sectR group. We are a group of data professionals based in the Departments for Education, Department of Human Services and the Department for Child Protection along with colleagues from the National Centre for Vocational Education Research who are based in Adelaide. We meet once a month to share the latest approaches to statistical analysis in the R programing language.

Recommendation 2: A Chief Data Officer in each Department Departments in the South Australian Government all have a person at the executive level in charge of finance and a person in charge of human resources. But none have a person in charge of data analysis. I recommend that the position of Chief Data Officer (CDO) be created in the senior executive group of each Department in the Government of South Australia. The role of a CDO is different to the work already done by Chief Information Officers (CIO) in the Departments. CIO’s provide a mechanism for the maintenance of routine reporting and ad hoc requests for data analysis that are set by senior management. CDO’s on the other hand will lead a team that will help senior management determine which questions should even be addressed and through their high level of skill in data analysis will communicate to senior management the true scope of what can be done with the data held by the department that will help drive service improvement. I will advocate for this recommendation by first building the skills of data professionals in the public sector and the esteem they have in their work. This will be achieved through the implementation of the above recommendation. I anticipate that this will raise the professionalism and the profile of data scientists in the South Australian public sector. And, over time this will create a culture at the senior management level that appreciates the contribution this work has in the execution of their goals and so will seek to have an expert at the executive level to raise the outcomes of this work at the top levels of decision making.

Recommendation 3: No male only technology panels The lack of diversity of the workforce producing data driven decisions is now well documented internationally. I recommend that the Government of South Australia take

11

active steps to increase the participation of people who have previously been excluded from this work. Specifically, RenewalSA should place additional requirements on technology groups who wish to hold events at the Lot Fourteen precinct. These requirements should include a guarantee that 50% plus or minus 1 of the speakers at the event should be non-male, that the event has a code of conduct and that the organisation adheres to a data privacy policy. I will first build a coalition amongst my data science colleagues who use the Lot Fourteen precinct on this issue. I am well placed in data science organisations operating in Adelaide to make these connections. Importantly, three organisations that I work with, the Adelaide R Users Group, the South Australian Branch of the Australian Computer Society and the Artificial Intelligence Collaborative Network have all taken active steps to improve the diversity of their meetings. I will then set up meetings with the staff in RenewalSA who administer access to Lot Fourteen and ask for them to implement these requirements.

Recommendation 4: Data skills for the executive level The three recommendations above will help to improve the quality of data science work in the public sector of South Australia. My final recommendation is that members of the senior executive level in all the Departments in the Government of South Australia gain a greater understanding of the power and the limitations of data. This appreciation of quantitative work is necessary so that data can better inform decisions, so that the scope of what data can reveal is properly framed and so that the procurement of data products and services is done with due oversight. I will first meet with the colleagues in the School of Mathematical Sciences at the University of Adelaide and discuss the development of a syllabus for executive data science training. I will then seek meetings with the Commissioner for Public Sector Employment, Ms. Erma Ranieri and the Chief Scientist for South Australia, Prof. Caroline McMillen and ask for their support in taking this program to the Departments.

12

Notes

A note on language I deliberately chose the term "decision making algorithms" in the title of my Fellowship as an umbrella term to cover the large number of analytic strategies including artificial intelligence (AI) and other statistical processes as well as the prerequisite processes of creating, storing and sharing the data these algorithms need. This was important because I did not want the recommendations of this Fellowship to be limited to AI, or to what decision makers may consider AI. Great hard has been done using automated data driven processes that are somewhat artificial but are by no means intelligent1. I will use all these terms throughout this report.

Practicing what I preach As I learned more and more throughout this Fellowship, a key part of the ethical use of data is transparency. When analyses are run in one program, figures produced in another and the text and formatting created in a third, it can be difficult for anyone but the author to verify the connections between the stages. In fact, it can be hard for the author to verify the connections themselves at a later point in time. A solution to this problem is the use of a markdown language. Such languages combine the analysis code, figures and text into a single script which can be rendered into a nicely formatted document. To that end, this report was written in RMarkdown2. The script with all the underlying code is available on my GitHub3.

A note on methodology This report is not a systematic review of the literature on automated decision making in government. There is no replicable process to this report’s creation. The people I spoke to were the people I thought would be interesting and informative and who happened to have the time to see me. I have however, tried my best to keep myself in the report because I was indeed a part of it. I have tried to detail the prior beliefs and biases I brought with me to this Fellowship and how they were, or were not, changed by my experience.

1 A particularly horrific example currently ongoing in Australia is the use of automated name matching by the Federal agency Centrelink. For more information follow the discussion here: https://twitter.com/hashtag/robodebt. 2 RMarkdown is an approach to literate programing in the R programing language and readily fosters notebooks that can be rendered into published documents as used here. For more information visit: https://rmarkdown.rstudio.com. 3 GitHub hosts a distributed version control and source code management platform. The repo with the code for this report is here: https://github.com/ofchurches.

13

Background I joined the public service as a statistician in mid 2017. And it seemed that I was in the right place at the right time. "Data" was seemingly the topic of every conversation. There were courses on "data" analysis that public servants could attend4 and there were high profile public talks on the use of "data" at the highest levels of government5. To test whether my perceptions had some objective basis, I collected every mention of the word "data" in Federal Hansard for statements by members between 1997 and 2017.

Figure 1. Count of times ‘data’ is mentioned in statements by members in Australian Hansard from 1997 to 2017 Indeed it seemed that there had been a dramatic increase in mentions of the word "data" in the Australian Parliament since 2014. However, around the same time, there were some high profile international instances of governments using data to make decisions and causing serious social and personal harm as a result. One of these was the case of the use of data to set the amount of bond in the criminal justice system in Fort Lauderdale, Florida. This story was covered comprehensively in 2016 by ProPublica6. This report showed that the proprietary

4 Some of these courses are listed here: http://sa.ipaa.org.au/downloads/2019%20Course%20Outlines/Analysing%20Data.pdf. 5 An example of one such talk is here: http://sa.ipaa.org.au/Events/25102017_Data_Sharing_Event.asp. 6 The story can be read here: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

14

software used to make this decision was highly unreliable: only 20 percent of the people predicted to commit violent crimes in the future actually went on to do so. It also showed that this problem of predicting future crime when it did not eventuate affected different groups of people to a different degree; it wrongly labeled people of colour as likely to commit a future crime at almost twice the rate as white defendants.

Figure 2. Books published between 2016 and 2019 on the topic of AI, data and technology ethics and idiosyncratically collected by me It seemed that for all the enthusiasm for using data in Government there were some serious issues to address. It also seemed that there was a surge of interest around this time amongst academics and activists in the ethical, social and psychological issues associated with the use of automated decisions. To gain some understanding of the issues, I started reading the published literature in this field. Much like the political interest in "data", the critical literature on the ethics of data use has been rapidly increasing over recent years.

15

This was the impetus for this Churchill Fellowship; to talk to experts about the latest advances and the most pressing risks in data and AI. I set out to gain this knowledge from a variety of perspectives and a variety of industries. To cover the breadth of different perspectives I set out to talk with statisticians, computer scientists, sociologists, philosophers and lawyers. To gain the best from a range of industries I sought meetings with academics, non-government sector workers and industry.

16

Itinerary Across the six weeks of my Fellowship I had seventeen different events. This was made up of eleven meetings and six seminars. This itinerary is displayed in the table, timeline and on the map below.

Itinerary table

Date Purpose Person Organisation

09/04/19 Meeting Dr. Melanie Smallman University College London

10/04/19 Meeting Jacob Ohrvik-Stott doteveryone

10/04/19 Meeting Peter Wells Open Data Institute

15/04/19 Meeting Prof. Paul Burton Institute of Health and Society, Newcastle University

16/04/19 Meeting Prof. Madeleine Murtagh, Mavis Machirori, Dr. Joel Minion and Dr. Neil Jenkings

School of Geography, Politics and Sociology, Newcastle University

29/04/19 Seminar Dr. Rune Nyrup Cambridge Technology and New Media Research Group

29/04/19 Seminar Dr. David Sharp Ocado Technology 10x

29/04/19 Meeting Dr. Martina Kunz Leverhulme Centre for the Future of Intelligence, University of Cambridge

30/04/19 Seminar Prof. Nigel Crook Oxford Brookes University

30/04/19 Seminar Matthew Bland Institute of Criminology, University of Cambridge

1/05/19 Meeting Prof. Sir David Spiegelhalter Statistical Laboratory, University of Cambridge

1/05/19 Meeting Dr. Kanta Dihal and Dr. Jess Whittlestone

Leverhulme Centre for the Future of Intelligence, University of Cambridge

2/05/19 Seminar Dr. Matthew Botvinick Google Deepmind

3/05/19 Meeting Vicky Clayton and Louise Reid Nesta

7/05/19 Meeting Dr. Jennifer Cobbe Trust and Technology Initiative, University of Cambridge

17

9/05/19 Meeting Dr. Adrian Weller Department of Engineering, University of Cambridge

9/05/19 Seminar Dr. David Stillwell Psychometrics Centre, University of Cambridge

Table 1. Itinerary of meetings and seminars for this Fellowship as a table

18

Itinerary timeline An interactive version of this timeline is in the online version of this report on my website7.

Figure 3. Itinerary of meetings and seminars for this Fellowship as a timeline

7 My website is here: https://ofchurches.rbind.io/.

19

Itinerary map An interactive version of this map is in the online version of this report on my website8.

Figure 4. Itinerary of meetings and seminars for this Fellowship as a map 8 My website is here: https://ofchurches.rbind.io/.

20

Themes After six weeks of travel, eleven meetings, six seminars and reading the extensive and diverse literature that was given to me along the way, I had collected a large set of notebooks filled with ideas. But the job of doing the Churchill Fellowship does not end there. All these thoughts had to be turned into...something. The guidelines given by the Trust ask for “Recommendations”, and I certainly set out with that destination in mind. But there seemed to be a step missing in the chain between “have done the Fellowship” and “give recommendations”. This step might be called reflecting, digesting, finding common threads, or just understanding what I had done. I am not the only person who has looked at a large collection of text (even one they wrote themselves) and felt the idea of finding its underlying structure to be a daunting and perhaps prohibitive challenge. Doubts flooded my mind: “What if I missed something said in a seminar?”, “What if I mischaracterize something one of my interviewees said?”, “How can I be sure the themes I end up with come from my Fellowship experience and were not just the thoughts I started with, my own subjective biases?”. In truth, I also worried about the sheer magnitude of the job: “Would I be able to write up my understandings in the time I had been given by the Fellowship Trust. Interestingly, the job of taking a large corpus of texts and pulling out the underlying topics is an area of data analysis that has seen heavy investment over recent decades. A particularly important mathematical approach to modeling the topics in a corpus of text documents is latent Dirichlet allocation (LDA9). And I genuinely considered transcribing my notes and running LDA over them. Thinking about doing this brought me a great relief. The algorithm would fix all my problems: the algorithm would not miss something that was spoken because it would have all the notes. It would not inject its prior opinions into the result because it would have none. And it would do it all quickly. Once the text was transcribed and the algorithm written I could leave the machine to do the job. But I didn’t. I read and wrote and read and rewrote, filling up more and more notebooks (these now being R Notebooks10 rather than the thoroughly analogue Moleskine notebooks I took with me on my travels). In fact, the reason I didn’t turn this section of my Fellowship over to the machines is because of everything I had learnt during my Fellowship. The truth is that using a LDA algorithm to synthesise my interviews and pull out the common themes would not have solved all my problems. And it would have created problems of its own and amounted to a great loss of human creativity and understanding. Perhaps too it would have been an abrogation of my duty to come to terms with this topic myself, to struggle, to persevere and to produce a personal piece of work, no matter how imperfect it is.

9 More information on LDA is here: http://jmlr.csail.mit.edu/papers/v3/blei03a.html . And yes, the “D” in Dirichlet should be capitalised because it is named after Peter Gustav Lejeune Dirichlet. 10 R Notebooks are a product similar in functionality to Jupyter Notebooks that facilitate literate programming. More information is here: https://bookdown.org/yihui/rmarkdown/notebook.html.

21

So, here are the four themes that I wrested from my notes with my own hands. Each references the people who informed my thinking about the theme links with one of the recommendations in the next chapter of this report.

Fixing the technology It is important to note that not all problems with technology are in the technology. Some of the problems are in the society that builds and uses the technology11. For example, in October 2018 news stories emerged that Amazon had tried to build an algorithm to facilitate its engineer hiring process but that it was scrapped because it was biased against hiring women applicants12. This bias occurred because the machine learning algorithm learnt what a good hire was from the pool of resumes provided by current engineers. This pool was disproportionately male. As such, “fixing” the bias in the algorithm does not address the underlying problem: Amazon had a history of hiring more men than women. Hence, cases such as this should be seen for what they are: social problems in need of social solutions. There are however, a set of problems posed by the use of artificial intelligence (AI) that are worth dwelling on in their purely technical terms. These are the topic of this theme.

Data ownership I took the starting position that the original owners of data are the people who created it. So, patients and health care staff have the initial ownership over health record data. However, for these data to be useful for widespread service improvement, they need to be held together. To stay within the example of health data, it has become the habit of governments to see themselves as owners of data gathered in the public hospitals that they fund. The good work that can come from analysing these data would not be achievable without this centralisation and assumed ownership. However, an insidious problem can also arise. Governments are run by people. So, when we say that the government owns the data we mean that in practice there are people, perhaps just one person, who controls access to it. This can give rise to a kind of economy of obligation and favouritism. The person who controls access to these data can then effectively control their use. There is an upside to limited access to data, it helps to protect the privacy of the original creators of the data (the patients and health care workers in the health example). But what if there was a way to have access to the analysis of the data without accessing the data itself? This would keep the data confidential and remove the bottleneck of access

11 There has been some great recent writing on this topic including https://medium.com/s/story/the-seductive-diversion-of-solving-bias-in-artificial-intelligence-890df5e5ef53 and https://www.vox.com/future-perfect/2019/4/19/18412674/ai-bias-facial-recognition-black-gay-transgender. 12 More information is here: https://www.aclu.org/blog/womens-rights/womens-rights-workplace/why-amazons-automated-hiring-tool-discriminated-against.

22

through an individual. Well there is. It is called DataShield13 and it was developed by Prof. Paul Burton at Newcastle University. I met Burton in the cafeteria at the Newcastle University Medical School on a particularly cold afternoon in the northern city. DataSHIELD is an infrastructure and series of R packages that enables the remote and non-disclosive analysis of sensitive research data. Users can conduct analyses that are pre-specified by DataSHIELD and access results that can also be limited in the level of information they disclose. A different approach to thinking about data ownership was the subject of a great conversation I had at the Open Data Institute (ODI) with Peter Wells. Wells is the head of policy at the ODI and has spent a long time working in the same data sharing space as Burton focuses on the legal and policy frameworks that can be established to make data safely accessible.

Which intelligence are we talking about? The “Artificial” in “Artificial Intelligence” seems readily meaningful. However, there are some serious challenges to finding a single meaning to the “intelligence” part. This is perhaps in part because the field of human intelligence is itself the subject of development and debate. Two of the computer scientists that I heard from during this Fellowship were particularly aware of this limitation to the current conceptions of artificial intelligence. Dr. Matthew Botvinick is now the Director of Neuroscience Research at Google’s Deepmind division in London. Interestingly, before working in academia and industry on artificial intelligence, Botvinick studied medicine and then a PhD in psychology. He says that straddling the worlds of human and artificial intelligence has been a big part of his ability to develop AI that is more creative, more generalisable, more human. I heard Botvinick speak at the MRC Cognition and Brain Sciences Unit in Cambridge. One of the themes of his talk was about the role of dopamine in human learning and the way his team have used reward prediction error to simulate it in AI. This raised a particularly interesting round of questions following the talk. Differences in dopamine levels are associated with a range of clinical conditions in humans including depression and schizophrenia. What then does it mean to be varying the artificial analogue of dopamine in a machine? Do we need to start considering the limitations brought by different forms of thinking when we take their advice? This thread was taken up further in a talk I attended at Darwin College in Cambridge by Dr. Adrian Weller. Weller is a fellow in engineering at Cambridge and has built a strong research and public output on the interpretability and governance of AI. Weller took the time to chat with me over tea in the dining room of Darwin College after his talk and discussed his role in several governmental and intergovernmental organisations focused on the interpretability, privacy and safety of AI. One of the risks in the widespread adoption of AI that we covered was the possibility of too narrow a range of intelligences

13 For more information see: http://www.datashield.ac.uk/.

23

being used in decision making. There is strength in diversity. The fabled wisdom of crowds is reliably found to outperform selected experts in a large range of situations14. What then does it mean to have a particular AI making our decisions for us? The last person to contribute to my thinking about the diversity of thinking in AI that I heard on my Fellowship came to this topic through a distinctly different route. Dr. Nigel Crook studied maths and philosophy before moving into studies of design and technology, especially AI. I heard Crook speak at the Faraday Institute for Science and Religion, in the Garden Room of St Edmund's College Cambridge. He raised the question “In whose image is AI made?”. This is, at least in part, a question about the diversity of the people who are building AI. If the builders of AI seek to make machines that are better than they are on the dimensions that they value then the particular intelligence that has ruled Silicon Valley over recent decades will become an even more widespread mode of thinking throughout the world. This is particularly concerning considering the growing literature on the problems created within Silicon Valley itself by a mode of thinking which rewards constant growth, anti-collectivist ideals and techno-exceptionalism15.

Necessary but insufficient As noted in the Background to this report, recent years have seen a surge in critical literature on the ethical AI. And, to some extent this is beginning to make an impact. But only a beginning. This theme is about acknowledging the steps that have been taken by governments around the world so far while ensuring that we keep our eyes on the final goals. It is about ensuring we never mistake means for ends and are never satisfied with change that is always “in progress” but never actually progresses.

Principles, principles everywhere In the last few years there has been somewhat of a proliferation of AI ethics principles drawn up by government and intergovernmental bodies around the world. In the last twelve months there have been three different documents drawn up by different governments departments in Australia alone: the Australian Human Rights Commission16, the Australian Department of Industry, Innovation and Science17 and Standards Australia18.

14 A good survey of the literature is here: https://www.penguinrandomhouse.com/books/175380/the-wisdom-of-crowds-by-james-surowiecki/9780385721707. 15 A good start to reading about the social problems created by these modes of thinking is here: https://www.simonandschuster.com/books/Valley-of-the-Gods/Alexandra-Wolfe/9781476778952. 16 The document can be accessed here: https://www.humanrights.gov.au/our-work/rights-and-freedoms/publications/artificial-intelligence-governance-and-leadership. 17 The document can be accessed here: https://consult.industry.gov.au/strategic-policy/artificial-intelligence-ethics-framework/. 18 The document can be accessed here: https://www.standards.org.au/getmedia/aeaa5d9e-8911-4536-8c36-76733a3950d1/Artificial-Intelligence-Discussion-Paper-(004).pdf.

24

This is a trend observed by two academics at the Leverhulme Centre for the Future of Intelligence in Cambridge. Dr. Kanta Dihal is a post-doctoral researcher working on a project that seeks to understand humanity’s relationship with AI through the stories we have told, for millennia, about sentient machines. Dr. Jess Whittlestone is also a post-doctoral researcher and is working with policy makers to develop a more appropriate level of trust and a less exaggerated perception of the capabilities of AI. We met together in the meeting room of the Leverhulme Centre for the Future of Intelligence on Mill Road in Cambridge. The AI Narratives Project that Dihal is involved with charts the long history of human thought about AI. It is a history full in equal parts with wonder and fear, hope and disgust. I was particularly struck by the link Dihal saw between societies stories about AI and societies stories about slaves. This commonality is true not just for the uses to which owners have put them both - menial, unpleasant and dangerous work. It is also true for the end of ownership - the slave/AI revolt. These narratives belie an uneasy relationship the owners of AI have with the status of their machines. The Leverhulme seeks to make its work eminently applicable to policy development. And Whittlestone was well up to date with the latest government hearings and outputs on AI ethics in the UK, in Europe and around the world. Our discussion centered on the role of principles and discussion papers as a first step in developing policy and ultimately legislation with clear definitions of the behaviour that is within and outside the law. There is a risk Whittlestone pointed out, at this stage in the development of AI ethics, that governments will become complacent, thinking that having documented the principles of AI ethics is having done enough.

The “R” word From a legal perspective, principles are not enough. Principles are good intentions. Principles are a hope and a dream. And it’s important to have vision, to set out the goals for where we want to go in broad brush strokes. Principles do well to describe in prose not just the minimum standards of behaviour we expect in society but also our aspirations for the best version of ourselves that we can imagine. But unless principles are turned into actionable legislation there will be no justice for those who infringe against the standards we expect. There will be no justice for those who are harmed when someone violates those standards. And, there will always be a fear amongst the financially competitive that others are not sticking to the standards, that doing the right thing is costing them money. This was the topic of an excellent discussion I had with Dr. Jennifer Cobbe in the famous Cambridge landmark of Fitzbillies cafe on Trumpington St. Cobbe has a PhD in law and is now a lecturer in the Department of Computer Science and Technology at the University of Cambridge. We talked about the need to regulate AI and about ways to engage with some of the arguments against it.. “Watch out for the argument that regulation will slow down innovation” Cobbe warned me. Indeed, I have heard that

25

countless times. However, until my conversation with Cobbe, I had failed to really critically parse the argument. I am a public servant and believe strongly in the power of government for good. In the case of AI, I took it as a starting position that it would need to be regulated. But I also accepted the argument that doing so would impede progress toward better and better AI. However, over the course of our conversation, Cobbe explained the false antagonism between regulation and innovation. Regulation doesn’t slow down innovation, it turns it in the direction that we most value as a society. For example, speed limits on cars have not stopped all innovation in automobile manufacture. Rather, this regulation has encouraged car makers to innovate for the qualities that society has decided are more important than more speed: better safety, greater comfort and less pollution. The question then is not whether AI use by government should be regulated but rather what regulations should be applied to turn AI in the direction we most value as a society.

Slowing down Deciding what then to regulate was the topic of my conversation with Jacob Ohrvik-Stott from the think tank doteveryone. The doteveryone offices are in the beautiful neoclassical Somerset House on the banks of the Thames next to Waterloo Bridge and we met in the cafe on the ground floor of the building. Doteveryone researches and promotes responsible technology that will encourage a fair, inclusive and thriving democratic society. One of the cautionary thoughts that Ohrvik-Stott got me thinking about was whether a new regulator was really necessary. One the one hand it seems that the use of widespread data collection and automated decision making represents a distinct break with previous ways of working in government. And, to effectively regulate anything first requires understanding it. So, perhaps a new regulator, armed with the skills and resources to investigate the workings of algorithms should be established. To some extent, the UK has already done this with the creation of the Centre for Data Ethics and Innovation19 and the Office for Artificial Intelligence20. But this may well create more problems. First, data and algorithmic decisions are already being used in a wide variety of contexts. So, the new regulator would need to understand the nuances of AI being used for diagnostics in the Department of Health and data being collected for automated essay marking in the Department of Education. Such diverse topic knowledge may be beyond the capabilities of any single agency. Secondly, government regulators have a history of fighting between each other for a chance to take on the actual object of investigation. So, if a new AI was used in the Department of Child Protection, would it be assessed by the new regulator or the existing Guardian for Children and Young People? In the end, the question Ohrvik-Stott posed went

19 For more information see: https://www.gov.uk/government/organisations/centre-for-data-ethics-and-innovation. 20 For more information see: https://www.gov.uk/government/organisations/office-for-artificial-intelligence.

26

unanswered for me. There are clearly benefits and costs either to creating a new regular or to strengthening existing regulators with new skills, powers and resources.

What decisions shall we automate? It seems plausible that in the near future, there will be decisions currently made by people in government that will instead be made automatically by AI. There has been plenty of discussion in the literature on AI and society suggesting that we need to maintain the presence of a human-in-the-loop at some stage during these decisions21. Sometimes this discussion forgets that there is already a human in the loop in at least one place: the start. AI do not make the decision about which decisions to make on their own. A human has decided which decisions can or should be automated. We could of course build an AI to decide which decisions to make but this would simply push the problem one step back: a human would still have to decide on the nature of possible questions and the material used to meta-decision about decisions. So, this theme is about that human not just in the loop, but at the start of it.

Just the same only faster The very first meeting I had for this Fellowship was with Dr. Melanie Smallman. It was an exciting time. Just days after arriving in London I made my way from my rented apartment in Bloomsbury, past the British Museum and along Euston Rd toward King’s Cross station and into the British Museum. Within the British Museum is the now well established Alan Turing Institute and it was there that I met Smallman between whiteboards filled with equations, an excellent coffee maker and an actual Enigma Machine. Smallman is a sociologist at the Turing and has written extensively about the unasked questions that the use of AI often presupposes. Our conversation turned to the question of what decisions tend to be automated by governments. I realised that there is a common theme to the use cases for AI in government that most readily came to mind: child protection notification screening, welfare payment processing, police deployment and criminal sentencing. These were all decisions that tend disproportionately to affect the poorest and most disadvantaged people in society. Neither of us had heard of a suggestion to use AI to automate the processing of property development applications. So why the focus of AI on decisions that impact the lives of the most vulnerable in society? To some extent, there is some more gain to be made by automating these decisions because there are more of them. Governments make more welfare payments than they review high profile property development plans. But Smallman suggested, and I find myself agreeing, that the extent to which this bias exists is a reflection of the degree to which a government wants to do this work at all.

21 For more information see: https://hackernoon.com/what-is-human-in-the-loop-for-machine-learning-2c2152b6dfbb.

27

Figure 5. An enigma machine outside the Alan Turing Institute in the British Library where I met Dr. Melanie Smallman

Governments exist with the consent of the governed The functions of child protection and policing are both devolved to individual counties in England. I was fortunate to hear about the use of AI in both policing and child protection during my Fellowship. First, I attended a talk by Matt Bland from the Institute of Criminology at the University of Cambridge. Matt spoke to a small assembly of the Cambridge Statistics Discussion Group one evening in the cafeteria of Amgen, a biotechnology company with an office in a science park outside of Cambridge. Bland talked about the use of AI in policing in England. The uses it is already put to include suggestions for where and when officers should be preemptively be deployed and in predicting the risk of re-offending by perpetrators of domestic violence. Bland was quite upfront about one of the drivers for using AI in policing: cost cutting. Budgets are constantly decreasing and police force administrators are trying to do more with less. AI is sold to these police forces as a way to save money and still stop crime. Or even increase the force’s performance on the metrics by which they are judged. This too comes in the context of the same management meme that prompted me to undertake this Fellowship: it seems important at this time to talk about being “data driven” and “evidence based” even if it is only talk.

28

I was concerned nonetheless by some of the enthusiasm Bland showed for some of the applications of AI in policing that seemed to be statistically naive. Bland spoke about how several police forces are using AI to decide where police officers should be sent to patrol. These systems look at the extent of crime in different areas and send more police to areas with the greatest crime. Bland explained that these locations are then observed not to be such outliers the next time crime is measured. The problem, is that attributing these reductions in crime to the deployment is fraught because this design can not rule out the role of a statistical phenomenon called regression to the mean. This is the process by which the measurement of a random variable which sits at the extreme of a distribution on its first measurement tends to fall closer to the mean on a subsequent measurement. In this case, if crime rates are a random variable then a location with an extremely high crime rate at one point may be expected to have relatively less crime at the next measurement whether extra police were deployed there or not. Now, it is likely that crime rates are not completely random and that the deployment of police has some role. But these results do not on their own support the notion that this shifting deployment of police officers is effective in reducing crime. Bland finished with a point I had not considered but that I now realise is fundamental to the issue of automated decisions being used by any government agency. Bland pointed out that police forces should consider the effect on public relations of it becoming known that an AI is determining the locations of police officer patrols. Bland argued persuasively that this is not an idle concern with “public fallout” or “negative press”. Rather it goes to the heart of how governments gain their legitimacy: they operate with the consent of the people. So, before AI is broadly deployed it would be necessary to ensure that the public is well educated about the merits of using automated decision making. AI should be used with the public rather than on it. The overtly political nature of the decision by governments to use automated decisions was echoed in an excellent conversation I had by phone with Vicky Clayton and Louise Reid from What Works for Children’s Social Care. This is an independent organisation funded by the Department for Education in the UK that aims to bring evidence based practice to the work of the children’s social care sector. I got in touch with Clayton and Reid because while I was in England, What Works commissioned a review of machine learning in children’s social care22. The scope of the review specifically mentioned the “acceptability” of the use of automated decision making in this context. That is, the review was concerned not only with the algorithms themselves, whether they were fair, accountable and transparent, but also with the conditions outside of the algorithms, with the public’s understanding, appreciation and acceptance of these algorithms in these contexts. Hence, Clayton and Reid stressed that this review would not be limited to reviewing the technology. It would review the whole of the occupational and social context in which it

22 The announcement of the review is here: https://whatworks-csc.org.uk/blog/what-works-centre-for-childrens-social-care-announces-the-rees-centre-department-of-education-university-of-oxford-and-the-alan-turing-institute-as-research-partners-in-ethics-review-of-ma/.

29

is used. This struck me as incredibly important information to review and a rare approach to this sort of work. It then struck me as strange that I felt this was rare. But, as Clayton and Reid explained, social workers, families and children themselves are rarely involved in the decision to automate decisions about them. First of all, this is rude. These are people with expertise in this field and to disregard this knowledge is highly arrogant. By failing to review the learnings of people already involved in this space it is not clear that the new automated approaches will be better than the status quo. Second of all, even if on some relevant metric the new automated decisions outperform the current approach, without a consultation process that brings the stakeholders along on the journey and creates a sense of ownership amongst them, the new approach is set up from the start to fail. There is a certain hubris to the mindset that accompanies much of the technology sector to think that just because a system is better it will be accepted. There is a great deal of research on implementing change in social systems which shows the value of engaging the end users from the beginning23. Without taking this step, the organisation implementing the change runs the risk of the new system being left unused, underused or subverted by a workforce and public who are at best unaware of the system but may also actively resent it. The last event I attended during my Fellowship which informed my thinking about this theme was a seminar run by the Leverhulme Centre for the Future of Intelligence on the methodology and ethics of targeting. The key speaker was Dr. David Stillwell who is the University Lecturer in Big Data Analytics & Quantitative Social Science and Deputy Director of the Psychometrics Centre at the University of Cambridge. Stillwell was an author on a 2013 paper published in the Proceedings of the National Academy of Sciences titled “Private traits and attributes are predictable from digital records of human behavior”24. The paper announced the discovery that there was a robust association between a person’s personality and their pattern of “Likes” on Facebook. Like so many scientific discoveries in the past, the discovery was quickly put to use by others for purposes the original researchers could never have imagined and certainly did not endorse. The discovery paved the way for the now prevalent process of “micro-targeting”, showing slightly different versions of an advertisement to different people depending on their personality but all of which have the indent to change their behaviour in the same way. Perhaps the most notorious use of micro-targeting was by the firm Cambridge Analytica in the service of President Donald Trump’s 2016 presidential election bid. And a former colleague of Stillwell’s who was not on this original paper went on to work with Cambridge Analytica under allegations of improperly using University resources to harvest Facebook profiles25. The seminar was held in the Winstanley Lecture Theatre at Trinity College, a modern lecture theatre accessed by winding through narrow alleys, up wooden stairs and across flagstoned courtyards. Stillwell spoke about the specific work he has been involved with 23 See for example: https://www.apsc.gov.au/changing-behaviour-public-policy-perspective. 24 The paper can be read here: https://www.pnas.org/content/pnas/110/15/5802.full.pdf?3=. 25 The circumstances and the people involved at all levels of the University of Cambridge are detailed in this article: https://www.theguardian.com/education/2018/mar/24/cambridge-analytica-academics-work-upset-university-colleagues.

30

and more generally about the ethics of doing it for academic or industrial applications. What surprised, and impressed me about Stillwell’s frame for talking about the ethics of using data about people scraped from platforms such as Facebook was the emphasis Stillwell placed on testing for the public acceptability of the project. The same test that Clayton and Reid talked about in relation to child protection data and Bland spoke about in relation to policing. For example, Stillwell talked about an advertising campaign run by Netflix in 2017 that used the viewing behaviour of users and in some cases made fun of their viewing habits.

Figure 6. Tweet sent by Netflix as an advertising campaign in December 2017 that used customer data but ridiculed those customers instead of helping them This produced a strong, negative public response26. Stillwell suggested that the ways in which Netflix went wrong in this campaign were numerous but that there are steps that organisations should take to avoid this sort of action in the future:

1) Give users control It helps to be clear with users how they can allow their data to be used and for what purposes and to let them change this at any time.

2) Use the data for purposes that benefit the users The advertisement run by Netflix benefited the company but not the individual users whose data made the campaign possible.

These steps, I realised, were as applicable to the actions of governments as they are to the private sector.

26 See for example: https://mashable.com/2017/12/11/netflix-a-christmas-prince-tweet-privacy/.

31

Who’s costs and benefits With these thoughts in mind, I went back to the Leverhulme Centre for the Future of Intelligence in Cambridge to meet with Martina Kunz. Kunz is a lawyer who is now doing a PhD in international AI governance. In a powerfully meta-analytic process, Kunz is using her PhD to map the global uptake of AI regulations using the tools of AI, including natural language processing and machine learning. We talked about the social costs and benefits of the use of AI and how they are dispersed across society and across the world. As with technological changes in the past, there will be costs and benefits to the use of AI by governments. Kunz lead me to realise that the process of governments procuring AI from private companies risks spreading those costs and benefits quite unevenly across society and across the world. The costs of using automated decisions include being surveilled to an ever closer degree, having decisions made by biased and inaccurate algorithms and having little capacity to appeal these processes because it is in their nature to be opaque. Because of the bias in the use of AI for decisions involving the poorer and most vulnerable discussed above, the costs will be borne disproportionately by these groups. The benefits of AI on the other hand, include increased efficiency, lower wage costs and the return of a surplus to the owners of the algorithms. These benefits will be held most by those with the current spare capital to invest in owning the algorithms from the outset. That is, when governments procure AI from private companies to make decisions about the poorest members of society they are likely exacerbating the inequality that already exists.

Building a culture with AI And so to the last theme I found in the meetings and seminars of my Fellowship. I started this Fellowship thinking a lot about how governments could better use or not use automated decisions to make society at large a better place. But I realised at the end that there is one sector of society that is particularly in need of being made better: the sector of people making the algorithms. There is a role for government in reforming the culture of this sector, both within government and in industry more broadly. There is good reason to hope that by implementing reforms to the people working in this space, the algorithms they produce will improve in their social responsiveness.

Where is this work done? While in Cambridge for other interviews and seminars I saw a poster in the Centre for Mathematical Sciences advertising a recruitment talk by Dr. David Sharp from Ocado. Ocado is a grocery delivery company operating in the United Kingdom. They have a relationship with the retailer Waitrose in which they deliver items that are available in the Waitrose stores27. Sharp drew in around fifty maths and computer science students to his talk in a seminar room at the Centre for Mathematical Sciences. He offered an insight

27 Although this relationship will soon end. For more information see: https://www.reuters.com/article/us-m-s-outlook/ms-targets-doubling-of-food-business-after-ocado-deal-idUSKCN1U41DW.

32

into working for a successful technology company, plus free pizza. It would be hard for a student to resist! I was there passing the time between other meetings and I thought it might be an interesting interlude. But as Sharp talked through the reasons why the soon to be graduates should come and work for Ocado I I realised that there was a lot that government could and should learn from the success of this company. First, Sharp was there, on a university campus recruiting. Ocado had sent a representative to unashamedly bring in new, predominantly young workers. Sharp leaned heavily on the theme that Ocado is constantly putting the newest data science approaches into production and that new graduates coming into the company can have an immediate impact because their skills and knowledge are highly respected and valued. This is in stark contrast to the public sector where, by 2016, only 10.8% of the Australian federal public sector was under 30 years old28. The next step in Sharp’s sales pitch was to point out that all the AI Ocado uses is built in house by the teams of mathematicians and engineers that these students could soon be joining. These AI make decisions that include which items to show to customers on the purchasing app, how to order the stacking of produce in their warehouses and the routes that drivers should take in making their deliveries. Sharp stated that the company decided early on that the decisions made by these AI were a key part of the company’s work and so should not be tendered out. Sharpe went on to say that the company attributes its market leading position to this decision. Furthermore, the company now sees itself as a technology company, not a grocery retailer. This reframing is no doubt helped by a recent and probably unexpected benefit of building these AI in-house: Ocado has now sold its entire software ecosystem to grocery delivery companies working in markets outside the United Kingdom, including Coles in Australia29.

Who owns the data? On a bright but freezing April morning I met another group of Tyneside based researchers in their offices in the School of Geography, Politics and Sociology at Newcastle University. Prof. Madeleine Murtagh, Dr. Joel Minion, Mavis Machirori and Dr. Neil Jenkings are researchers in sociology and bioethics and have built a considerable research base in the critical overview of the systems for collecting, storing and making decisions with health data. We drank tea in their warm office and shared an exciting conversation about the social steps that governments can take to better build, deploy and regulate their use of automated, data driven decisions. The first step they suggested was in finding the data gatekeepers. The second step was in removing them. The third step was in replacing them with transparent governance structures. The gatekeepers they referred to are the people that arise in large

28 For a full breakdown see: https://www.apsc.gov.au/table-13-ongoing-employees-age-group-sex-30-june-2002-30-june-2016. 29 For more information see: https://www.reuters.com/article/ocado-coles-deal/ocado-lands-partnership-deal-with-australias-coles-idUSL8N21D00P.

33

organisations such as governments to occupy a position in which they are largely responsible for regulating access to a particular resource. On a small scale this could be a person with the only key to the stationary cabinet for an office. But on a larger scale, this could be a person who controls access to a particular dataset. Organisations hope that the existence of these gatekeepers fulfills a useful function, one that management is notoriously averse to: the widespread stealing of assets by staff. This could be the surreptitious stealing of pens from the stationary cabinet or the removal of confidential data from a server. The risk of this happening is real and, in the case of data, the costs of it happening could be considerable. However, the work of these gatekeepers may be neither sensitive to all instances of stealing nor specific enough to allow legitimate use to occur efficiently. On the sensitivity side, theft of data could occur by the gatekeeper being deceived or by being complicit themselves in the theft. On the specificity side, these gatekeepers may form cliques of other employees around them that are always given priority access to the data even though other employees may have uses for the data that would be of greater benefit to the organisation and the people it serves. The suggestion of this group of researchers was to replace these gatekeepers with appropriate governance frameworks that review access to data, provide transparency in their decision making and appeals when necessary.

For the love of data I am a statistician and I realise that when I think about the ethical use of data and of ethical AI, I tend to think about those ethics along the dimensions which relate to my training and practice. But there is another discipline that has thought systematically about ethics for a long time: philosophy. From the perspective of philosophy, the ethics of AI is just the latest application of thinking about ethics to a problem that society is facing. So, I was keen during my time in Cambridge to hear what professional philosophers are making of the discussion around the ethical use of data and of ethical AI. To that end, I attended a seminar in the lecture theatre on Mill Lane in Cambridge hosted by Dr. Rune Nyrup who is a philosopher of science, working at the Leverhulme Centre for the Future of Intelligence. I was reminded immediately of the differences between the disciplines and of the trappings of authority and the signifiers of status that spring up in one context and eventually become taken for granted and thought to be the normal or only way to operate. In short, when I entered the seminar room, I had no idea who was in charge, or which of the people sitting around a table and talking was the advertised speaker, Dr. Nyrup. In seminars I am used to, attention is focused on one person, at the front, probably with a slide deck. But that is not the way of philosophy. And I am glad I saw another approach in action and saw that it worked so well. Once the seminar started, Nyrup (who was indeed one of the dozen or so philosophers sitting at the table) spoke about the ethical challenges posed by AI. With a careful and measured argument, Nyrup explained the ways in which there are aspects of these challenges that are novel but that there are also aspects of these challenges that are echoes of ethical challenges faced previously by philosophy. One of these was a

34

phenomenon that Nyrup described as data-philia. This is, literally, the love of data, but is really something more like the unquestioned acceptance of data. Nyrup reminded the room of a paper by David McCabe and Alan Castel from over a decade ago, at what now seems to have been the height of enthusiasm for neuroscience30. In this paper McCabe and Castel show that people more readily believe the results of neuroscience research when the text is accompanied by a brain image than if the same text is accompanied by bar graphs, topographical maps of brain activation, or no image. They conclude that the brain images are “appealing to people’s affinity for reductionistic explanations of cognitive phenomena”. So to, argued Nyrup, are the trumpery of data-science now held in such esteem that a company report or departmental memo hearing complex plots, convoluted terms and a barrage of numbers is likely to be more readily believed than one without these trappings. I suspect he may well be right considering that in January 2017, “data-science” past “neuroscience” as the more searched for term on Google.

Figure 6. Google search trends for ‘neuroscience’ and ‘data science’ worldwide between 2004 and 2019

Understanding The last person I met in this final theme, and so in this catalogue of meetings and events was a true hero of mine. Prof. Sir. David Spiegelhalter is the Winton Professor of the Public Understanding of Risk in the Statistical Laboratory at the University of Cambridge. Across his many and varied output of publications are books on Bayesian statistics, sex

30 The paper can be read here: https://www.sciencedirect.com/science/article/pii/S0010027707002053?via%3Dihub.

35

and most recently, interpreting statistics. Spiegelhalter invited me to meet him in the cafeteria of the Centre for Mathematical Sciences in Cambridge. Spiegelhalter greeted me warmly and then took me to lunch while we talked. We discussed his recent book and his plea for statisticians to aim for their work to be trustworthy rather than trusted (the difference is that being trustworthy is under one’s own control, being trusted is at least partly controlled by the one doing, or not doing, the trusting). As lunch time finished and the crowd in the cafeteria thinned around us, our conversation carried on. Spiegelhalter was keen and curious with his questions as much as he was ready and willing with his answers to mine. As we refueled with a post lunch coffee, the conversation turned to a problem to the use of AI that I had not really considered previously: it avoids understanding. An algorithm that can make decisions but which can not readily be understood misses a good part of the point, and the fun, of statistics argued Spiegelhalter. And I am inclined to agree, I want to know why things are the way they are. Causes are worth finding both because they let us get to the origins of a problem and also because we have the curiosity to find them. The topic of our conversation that I was most excited by however, was Spiegelhalter’s suggestion regarding professional development for public servants. We talked about the fact that few senior members of any public service workforce are trained in a quantitative discipline. This was a problem he was already aware of in the British civil service and I confirmed that the case was similar in South Australia. “But”, he suggested, leaning forwards with a keen energy “South Australia is much smaller than all of Britain. And with less inertia, it would be easier to change.”. The argument had a certain logic and more than anything I was swept up in Spiegelhalter’s vision. “What if”, he continued “there was professional development in quantitative skills that was necessary to gain promotion above a certain level in the public service?”.

Figure 7. My signed copy of Prof. Sir David Spiegelhalter’s book The Art of Statistics with a message I hold dear

36

Learning from history

Background In addition to the people I met with and heard speak in person, I took the opportunity of this Fellowship to commune with and seek some consolation from a group of experts in statistics and society who can no longer speak for themselves. There is so much that is new in AI. New mathematics, new hopes, new perils. But AI did not come out of nowhere. It was not conjured fresh from the soils of Silicon Valley at the dawn of this millennium. Neither the mathematics nor the social problems they create are without precedent. We have built new ways of thinking about probability before and we have dealt the consequent challenges to established economics and personal privacy with new laws, new social institutions and a never ending quest for further advancing the frontier of mathematical understanding. So, in a bid to remember the past and seek the lessons of history, I took the opportunity of being based in the United Kingdom to visit some sites of great statistical significance. Not all the locations of statistical significance in the United Kingdom are important because they contributed lasting benefits to society. In fact, there are buildings and lecture theatres at University College London (UCL) that are still named after eugenicists. Several prominent statisticians in the early 20th Century adhered to this racist world view. UCL has developed a podcast which looks into this history and its legacy31.

Fisher's innovation This statistical history tour was also spurred by my home town of Adelaide's role in global statistical history. The University of Adelaide was the final academic affiliation of Sir Ronald Aylmer Fisher who was a particularly influential 20th Century statistician. One aspect of R. A. Fisher's work provides an elegant microcosm of the situation we again find ourselves in when we talk about the impact of AI on society; Fisher developed a new mathematical approach to probability but this allowed for the rise of a serious social problem. The "p" value and its notorious threshold of .05 were developed by Fisher to help experimental scientists interpret their results but has since become a benchmark for publishability32. Before leaving on the Fellowship, I visited the memorial to Fisher at the Adelaide Cathedral with my friends Prof. Josh Ross, Dr. Phil Cassey, and Dr. Lewis Mitchell.

31 The podcast can be heard here: https://www.ucl.ac.uk/culture/projects/bricks-mortals. 32 For an explanation of this effect see: https://journals.sagepub.com/doi/10.1177/1745691612465253.

37

Figure 8. Dr. Lewis Mitchell and me at the Fisher memorial in the Adelaide Cathedral Then, while I was on the Fellowship I traveled to six sights of historical statistical importance.

Locations visited

Bletchley Park Edward Simpson and Bill Tutte (statue pictured) worked here as code breakers.

Figure 9. Statue of Bill Tutte at Bletchley Park

38

Gonville & Caius College This was R. A. Fisher's College.

Figure 10. The side entrance into Gonville & Caius College from the Senate House in Cambridge

Broad Street Pump John Snow Snow pioneered the use of dot density maps to trace the source of cholera in Soho in 1854 to this water pump

Figure 11. The John Snow Snow memorial pump (without a handle) in Soho

39

St Thomas' Hospital Florence Nightingale built an evidenced based nursing school here. There is now a museum about her work where you can buy prints of her pioneering data visualisations.

Figure 12. One of the coxcomb charts invented by Florence Nightingale

Bunhill Fields Burial Ground Thomas Bayes is buried here.

Figure 13. The grave of Thomas Bayes at Bunhill Fields

40

Drapers Hall John Graunt was one of the inventors of life tables. This was his Guild.

Figure 14. Drapers Hall in the City of London where John Graunt was a member

41

Map of locations visited An interactive version of this map of these locations is available in the online version of this report on my website33.

Figure 12. Map of locations visited during my statistical history tour

33 My website can be accessed here: http://www.ofchurches.rbind.io.

42

Outcomes of this Fellowship so far A lot has happened in the ten weeks since I returned to Adelaide from this Fellowship. I have been fortunate to be involved with several new projects that have helped move forwards on the recommendations I have developed for creating accountability and transparency in government automated decision making in South Australia.

Installation at MOD

Recommendation engines When we look for a film to watch on Netflix, food to eat on UberEats or news to read on Facebook, the interface we see presents us not with an open list of all the possibilities stored by the service but rather with a curated, personal set of recommendations. These recommendations are produced through a class of machine learning algorithms called recommendation engines. In short, by constraining our choices, recommendation engines give us a shortcut to our own personal pleasures. But for all the increased purchases and add views these algorithms bring, there are costs to individuals and society that are increasingly being detected and described. In the case of YouTube, both the nature of the alarming content that is routinely recommended and an explanation of the way the recommendation engine works have been detailed by Guillaume Chaslot34. Early in 2019, the excellent Museum of Discovery35 in Adelaide sent out a call for scientists and artists to create an installation for their next exhibition which would be on the topic of "pleasure". It occurred to me that at the retail end of recommendation engines, whether we are using them to navigate the news or find friends or digest dinner, there is a sense in which we are always asking for the algorithm to deliver us the same thing: our greatest pleasures. Using data from an experiment on word associations I ran with Dr. Simon De Deyne at the University of Melbourne and Assoc. Prof. Hannah Keage at the University of South Australia, I built a recommendation engine for pleasure and proposed it to MOD as an installation for their exhibition.

34 A good start to Chaslot’s writing on this topic is here: https://medium.com/@guillaumechaslot/how-youtubes-a-i-boosts-alternative-facts-3cc276f47cf7. 35 The MOD website is here: https://mod.org.au/.

43

The finished product MOD accepted my installation for their exhibition. The installation is on display along with the rest of the HEDONISM exhibition until October 2019. You can see it in person at MOD in Adelaide36. You can also see the finished product online37.

Figure 15. The opening of the HEDONISM exhibition at MOD. The developers of the 'It's My Pleasure' installation, Assoc. Prof. Hannah Keage, Dr. Simon De Deyne and me, with our work in the background

public_sectR I have started a professional group for members of the public service who are based in Adelaide who use the R language for statistical computing in their work or would like to start using it. We meet once a month in the boardroom of my agency, the Child Death and Serious Injury Review Committee. The meetings follow a community of practice model in which everyone teaches what they already know and learns in turn when there is something they do not yet know. There are currently five members of the group. I am encouraged by the interest of this small group in sharing the latest advances in the R statistical programming language and in telling stories about how they are using it in their professional work and foresee that the group will continue indefinitely.

36 Information about MOD can be found here: https://mod.org.au/visit/getting-here/. 37 My pleasure recommendation app can be accessed here: https://mod.org.au/mypleasure/.

44

The skills we share in this community of practice are directly applicable to the goals of our departments. Writing code and hosting it in distributed version control systems is an improvement over individual analysts using spreadsheets in the following ways:

Lower costs Much of the analytical reporting done in Government is routine. Annual, quarterly and monthly reports are the staple of any data analysts work. This is frequently achieved by using a labour intensive process of writing and clicking in spreadsheets and repeating this process every time the report is required. Writing the instructions for these reports as code instead decreases the labour cost of running it each time the report is required.

Efficient oversight If the steps in a data analysis project are done on different computers using spreadsheets with opaque calculations that are all stored locally, it can be hard for management to establish how a result was arrived at. But if the steps are written in code and shared by the analytics team through a decentralised version control system then the steps taken to achieve the final result can be efficiently traced back.

AI Ethics Bookclub I was an early member of a professional group of AI practitioners called the Artificial Intelligence Collaborative Network (AICN)38. Since starting my Churchill Fellowship I have worked with the AICN to develop a monthly book club in which we read and discuss books on the topic of AI ethics. So far we have 26 members of the club. Some are based here in Adelaide but others are around Australia and around the world. While some members of the book club are AI practitioners, the book club has also attracted a diverse range of people with a personal or professional interest in AI ethics. This includes lawyers, teachers and public servants. Our first meeting was in June at ThincLab39 and online and we discussed Made by Humans: The AI Condition by Ellen Broad.

38 For more information about the AICN see: https://www.collaborativenetwork.ai/home. 39 Information about ThincLab is here: https://www.adelaide.edu.au/thinclab/.

45

Figure 12. The first meeting of the AI Ethics Bookclub hosted by AICN The discussion and interest further enhanced my knowledge of AI ethics. Most of all it was encouraging to see how many people were motivated to talk about AI ethics and to learn more from the published literature on the topic. Considering the rate of publication of books on AI Ethics and the interest shown by members of the book club so far, I foresee that this bookclub will only gain in members and momentum.

46

Recommendations

A professional network for public sector data professionals

Context It is tempting to write quite prescriptive recommendations to this theme. In this light, the challenge from narrowly defined AI raised by Dr. Adrian Weller and Dr. Matt Botvinick is solved by using only ensemble learning that includes a diverse range of constituent parts. And the solution to data monopolies in government that was raised by Prof. Paul Burton is solved by platforms such as DataShield. But the truth is, recommendations at this level will always have weaknesses. Ensemble learning is still made up of parts that were chosen by humans and DataShield is still customised by human operators. So, my actual recommendation is to raise awareness of these challenges amongst the people using data in government. This includes the statisticians, data analysts and data warehouse managers who control the flow of data throughout government and decide on the analytic products put into production. I recommend that a professional network be formed for these public sector workers to join that would help maintain their professional skills. This network would then facilitate regular mandatory professional development. This could be in the form of external training or internal professional networking. There is a sense in which the collegial processes of journal clubs and colloquia used in universities suit this task. Likewise, grand rounds attended by doctors in hospitals and in-service training attended by teachers fulfil this role for their professions. It would need to be seen as a part of one’s job to stay up to date with the latest critical theory and advances in data management and analysis by attending these groups on a regular, preferably weekly, basis.

Implementation plan This is already the most advanced recommendation in its implementation. I have instigated a group for public sector employees who use the R programming language for statistical computing. We meet once a month and share our latest successes and stumbling points in using R to analyse government data. This group will now form part of the premise for the formalisation and funding of a larger group with broader aims. I will seek meetings with the Commissioner for Public Sector Employment, Ms. Erma Ranieri and with the Chief Scientist for South Australia Prof. Caroline McMillen. I will seek their support to go to each department asking if their data and analytic staff would be allowed to attend regular meetings to further develop their professional skills. I will also seek funding from the Office of the Commissioner for Public Sector Employment and the Department of Innovation and Skills to pay for event space, catering and the regular attendance of outside speakers.

47

A Chief Data Officer in each Department

Context The recommendations to this theme are about the direction we go now. What comes after we’ve written all the principles we can muster? When we’ve pushed back against the tide of techno-exceptionalism promulgated by those who stand to make the most money from a consequence free corporate environment? When government is equipped to monitor its own use of data and AI effectively? There are clearly several approaches to thinking about this. And I alone am not equipped to decide which path to take. So rather than lay a course for everyone to follow. I recommend that more people with critical expertise in the field of data science be employed at senior levels in the Government of South Australia. My recommendation is that the Departments of the South Australian Government each acquire senior staff with the necessary technical skills to work on all these issues. To work with the legislation drafters to take the next steps from the principles we have, to see through the jargon and cliché pushed out by corporate interests and to build the capacity to regulate data and AI use across government. In South Australia each ministerial department has people at the chief executive level with responsibility for finance, human resources and information technology. But there is no one with the specific responsibility to maintain and use the organisations’ data and to critically address questions with evidence. In the private sector the role of Chief Data Officer (CDO) is already well established40 but this has not been implemented at the departmental level in the South Australian Government. The reason for not implementing this position and staffing it that I have heard the most is cost. It will cost a lot. But, I would argue that not creating these positions is costing the Government more. For example, it is clear in a lot of my conversations with leaders in Government that the Government has a great deal of resources that are not being deployed in the best way to service the needs of South Australia. In industry, CDO positions have revolutionised staff deployment practices by finding times and places of greatest demand and working out ways to move surplus staff from other areas to meet this unmet demand41. One example of why a Chief Data Officer would save money and help a department achieve the aims of its Minister comes from a recent episode in the South Australian Department of Child Protection. The Minister for Child Protection the Department to state

40 For further information see: https://www.forbes.com/sites/insights-intelai/2019/05/22/rethinking-the-role-of-chief-data-officer/#51970d1a1bf9. 41 It is important to note that some of these approaches have proven detrimental to the health and welfare of the workforce (see for example: https://www.nytimes.com/interactive/2014/08/13/us/starbucks-workers-scheduling-hours.html) . However, this is not a reason to avoid using data driven approaches to deploying staff. Rather it is a reason to develop more responsible data driven approaches that are optimised for the health and welfare of staff and end users.

48

how many drug tests of parents had returned a positive result42. However, the Department stated that it could not do this without manually reading all the files “due to its IT system”43. A skilled data scientist with a voice at the highest levels of the Department could have made clear to the senior management that even though there is no specific flag in the Department’s database for a positive drug test, the information, stored in unstructured text could still be analysed to answer the question without manually reading the files. Analysing unstructured text is a standard part of a data scientist’s skill set. For example, I have published work analysing text from Twitter for this purpose44. With these skills in place, time and money would have been saved by the Department.

Implementation plan There is an historical analogy that may help describe the path this plan could take. It was only in the 1920’s that large organisations and government departments began creating the positions that were the first ancestors to today’s Directors of People and Culture45. Over time, it came to be understood that an organisation’s objectives could be more readily met if the human resources of the organisation were effectively managed. Furthermore, it was understood that this management was different to the skills of the payroll department typically controlled by a Chief Financial Officer. Likewise, I hope that through showing the added value that skilled data analysis can bring to the work of the Departments, it will be observed that the digital resources of our organisations need to be effectively managed. And furthermore, that this management requires a different set of skills than those of the information technology department typically controlled by the Chief Information Officer. So, as to concrete recommendations, I suspect that there are necessary attitudinal changes that need to occur at the level of senior management in the Departments of the Government of South Australia for this idea to take hold. These include:

1) Data skills being taken seriously as a professional endeavor. 2) An acknowledgement of the contribution that data skills bring to achieving the

goals of the Departments.

Hence, my plan is to continuously demonstrate the professionalism of my work and the value that the skilled analysis of data brings to the work of the Departments of the

42 For more information see: https://www.adelaidenow.com.au/news/south-australia/child-protection-department-orders-1500-drug-and-alcohol-tests-of-parents-putting-children-at-risk/news-story/84eb0414ca7f6ba1da8dd0962be2482d. 43 https://www.abc.net.au/news/2018-11-07/streamlined-random-drug-tests-of-parents-begin-in-sa/10468500. 44 The paper is here: http://maths.adelaide.edu.au/lewis.mitchell/files/Tiggemann_BodyImage_2018.pdf. And the permanent link is here: https://www.sciencedirect.com/science/article/abs/pii/S1740144517305375. 45 For a good history of human resources see: https://www.linkedin.com/pulse/historical-background-human-resource-management-vinaykumar-s.

49

Government of South Australia. To this end, the creation of the professional network for public sector data professionals in the recommendation above will help create an environment suitable to changing these attitudes over time. Hence, achieving the previous recommendation is part of the plan for building the coalition necessary for achieving this second recommendation.

No male only technology panels

Context So what are the right questions to answer with automated decision making processes? Where should we aim our ever increasing AI powers? On what features of people and place should we routinely collect information? What should my recommendations be? These seminars and interviews all focused on asking the question for which an answer is already implied when we ask critical questions of AI systems. This lead me to think about why we use AI at all? Why has the fairly recent computational innovation caught the attention of policy makers? There is a sense in which the current fervor for AI in government sits in stark contrast to the historic antipathy toward statistics that politicians and senior public servants have sometimes shown. I wonder if one of the reasons for this apparent change of heart is that AI, especially as it is delivered by machine learning algorithms, asks a slightly different question to standard statistics. Machine learning looks to make decisions on individual cases. It uses the matrix of data from past individuals, their characteristics and the outcomes of decisions about them to train an algorithm that can make these decisions about future cases. Standard statistics on the other hand, seeks to understand the way these characteristics have interrelated in the lives of these previous cases to produce these past outcomes. That is, the focus of the analyses in AI is the individual people, the characteristics that they share describe the differences between them. The focus of the analyses in standard statistics on the other hand, is the characteristics that the people share, and individual people are examples of these general patterns. This focus on the individual as the site of analysis, decision making and action sits more easily with a philosophy that implicitly or expressly diminishes the role of shared social factors in shaping the lives of individuals, a philosophy that now envelopes the mainstream of political discourse in countries such as Australia. This approach raises several questions but the one that I find relevant to this theme is the question of what we will do if we build perfectly functioning AI for making governmental decisions about people? What, for instance, will we do if we had an AI with perfect accuracy in detecting instances of child abuse or neglect. Child protection authorities would still have to remove these children from their homes, these children would still have to be housed and cared for by the state, and worst of all, these children would still have been abused or neglected. With its focus on the individual, the AI did not

50

seek to address the underlying problems that lead to the abuse and neglect in the first place. This was the question presupposed by the question the AI asked. Since this is a section headed “Recommendations” let my recommendation be this: whenever governments look to employ automated decisions that affect the lives of individual people, they need to involve more people in the decision making process than the technology company that will act as vendors for an algorithm and the procurement department within government. Domain knowledge experts, the public that is affected and those skilled with social science statistical training should all be involved in every step, at every stage. From a really practical perspective there is a bare minimum that the Government of South Australia could do to improve the diversity of voices in the room when data driven decisions are being debated: stop hosting all male panels at technology focused events. In June 2019, the South Australian Department for Trade, Tourism and Investment held a panel on “disruptive technologies” with nine members, all of whom were male46. The Government should commit to never doing this again.

Implementation plan The Government of South Australia frequently gives technology groups in South Australia access to the Lot Fourteen47 lecture theatre and meeting spaces for events free of charge. RenewalSA who administer the site already administer terms for the use of the space such as acknowledging the Government as a sponsor of the event. I will seek a meeting with the staff in RenewalSA who administer Lot Fourteen and ask them to also include terms that will help increase the diversity of people attending the event. These should include:

1) The organisers guarantee that at least half the invited speakers will be non-male. 2) The organisers submit a code of conduct for the event. 3) The organisers submit a privacy statement for the event.

The coalition I will bring to these meetings will include my peers in the Artificial Intelligence Collaborative Network (AICN). AICN has already taken these steps on its own48 and so is a demonstration of how this can be done. I will also seek support from

46 Details of the event are here: https://mailchi.mp/f8da7d4e8a70/invitation-disruptive-technologies-seminar?e=859adf296f and details of calls for the Government of South Australia to end male only panels are detailed in this thread: https://twitter.com/TTISouthAust/status/1130284527685914635. 47 More details are here: https://renewalsa.sa.gov.au/projects/lot-fourteen/. 48 For more information about AICN’s implementation of these steps see: https://www.collaborativenetwork.ai. More information about steps organisations can take to end all male panels is here: https://www.opensocietyfoundations.org/publications/end-manels-closing-gender-gap-europe-s-top-policy-events. Information about establishing and enforcing codes of conduct are here: https://frameshiftconsulting.com/code-of-conduct-book/.

51

the South Australian Branch of Code Like a Girl49 and the Chief Scientist for South Australia, Prof. Caroline McMillen. I will also offer to write the templates for RenewalSA to administer.

Data skills for the executive level

Context And so, this is my final recommendation: that professional development in quantitative skills be part of the pathway to senior management in the public service. There are numerous ways in which this could be implemented and I am open to seeing which would work most effectively. These plans differ in the people that would do the professional development and in the nature of the professional development itself. The quantitative training could be implemented for more junior public servants and we could then wait for this generation to rise through the ranks. Or we could ask current public servants who control budgets above a certain threshold or who have a certain level of management responsibility to take the training now. Importantly, the goal of this recommendation is not to train all public servants to be statisticians or data scientists. Rather it is aimed at creating a culture in which an informed appreciation of what data and AI can and cannot do defines the scope of discussion about the use of data in the public service. It will help avoid the potential for an “Emperor’s new clothes” scenario, in which a vendor sells a department a data product which is unsuited for its purpose but no one in the department is willing to admit that it seems lacking for fear that they will look ignorant.

Implementation Plan As for the training itself, there is already a robust quantitative skills training program for employees in the Australian Public Service50. A similar program could be implemented locally. This could also be run in conjunction with local Universities. The University of Adelaide already offers a free, online course in “Big Data Fundamentals”51. There may even be scope to create a syllabus for “Executive data science” as a joint project between the Office for the Commissioner for Public Sector Employment, the Chief Scientist for South Australia and the School of Mathematical Sciences at the University of Adelaide. This syllabus could draw on examples of data and AI use that are pertinent to the public sector to make the training engaging and relevant. Hence, I will first seek meetings with my colleagues in the School of Mathematical Sciences at the University of Adelaide: Dr. Lewis Mitchell and Prof. Josh Ross. I will take

49 For more information see: https://codelikeagirl.org/states/sa/. 50 Details of the program are here: https://www.apsc.gov.au/building-digital-capability. 51 Details of the program are here: https://www.adelaide.edu.au/learning/adelaidex/free-online-courses/big-data-fundamentals.

52

this coalition to meetings with the Office for the Commissioner for Public Sector Employment, the Chief Scientist with the final goal of meeting with department heads themselves.

53

Conclusion So, I did it. It has been hard. Really hard. But I am truly glad that I did not turn my notebooks over to an AI for the distillation of themes and the creation of recommendations. The ways it would have failed are illustrative of some of the most important aspects of this Fellowship. I will carry these forward in my work and I feel they are important to communicate to others.

It would have all the notes There is no such thing as all the data. All the data that is collected is a finite set. But the data that could be collected is infinite. Worse still, the data that goes into an algorithm is unlikely to be randomly drawn subset of all the data. Rather, the data that goes into an algorithm is limited by the bounds of the practicality achievable. In the case of this Fellowship, there were interviews that were shorter than others because of time constraints on one side or both and I could not travel forever. In the case of government data collection it is important to consider that governments in Australia will typically collect more data on poorer people because wealthier people can avoid the use of Government services. Their medical treatment can be in private hospitals, their education in private schools and their homes owned independently.

It would be objective Writing an algorithm involves choices. There is no objective, final, perfect algorithm. Algorithms in all fields, including the modeling of topics in corpora of text are active areas of research and development. What else keeps all these highly paid scientists at Google employed and computer science departments at Stanford and MIT inundated with applications? In the case of this Fellowship, there are a variety of different computational approaches to implementing LDA and there are other algorithms altogether for modeling the most important themes in text. Each of these algorithms may give slightly different results. But more importantly, each of these algorithms has been built by a human. A brilliant and skilled human but a human nonetheless. The definitions of a “topic” and all the other constituent parts of the algorithm were chosen by these programmers and their algorithms work within this subjective scope. But these are not necessarily a universal truths. They are questionable. They may not apply aptly in all circumstances. So, far from saving us from our own subjectivity, using an algorithm would enslave us to the insidious subjectivity of the machines creators.

It would do it quickly Efficiency. Frictionless ease. Moving fast. This is the modern dream. It is the aesthetic that drives so much of our relationship with technology. But there are costs to living so smoothly. To go from journal notes to coherent themes without any intermediary step

54

would be to lose one of the great opportunities afforded me by this Fellowship: to think, to wonder, to be curious. Even if there was an LDA algorithm that would have suggested the exact same four themes that I found, I’m glad it did it myself. It has been a rich and rewarding experience to think through this topic. I have learnt to appreciate the nuances and vagaries of this field by being part of it, by rolling up my cuffs and jumping in. Indeed it is because it took time and toil that it was most worth doing by hand. Furthermore, reducing my, human, role to that of data entry and result reading would be a loss to the reader of this report. The LDA algorithm could well take over a part of my job of choosing the themes from my Fellowship but there is much that I have gained, and I hope communicated, that is additional to that. In that same way, a government service that replaces it’s human interaction with an algorithm is missing at least some of the work done by the human. Ticket selling machines on trains may sell the tickets as well as a conductor but they don’t give directions to tourists and maintain civility amongst passengers.

Coda Perhaps the most telling part of my brief flirtation with letting an algorithm choose the themes from my Fellowship was the fear I felt in doing it myself and the relief I felt at the thought of relinquishing control to a machine. It makes me think that at least some of our motivation for using automated decision making systems is born from fear. Fear of how hard the job is, fear of getting it wrong. In this light, it is worth considering which parts of government services may be the focus of the most interest in the application of automated decisions in the near future. The application of AI to a topic can then be read to some extent as an indication of the degree to which a government is disinclined to roll up its sleeves and get its hands dirty with the topic at hand. Reflecting back on this process, I wonder if some of the relief born from the thought of using a machine to make my decisions comes from the desire to avoid responsibility. By using an algorithm to decide on the themes from this Fellowship I would be blameless for the outcome, absolved of responsibility. As I think more about this I realise that I now regularly blame machines for actions where I have devolved responsibility. I missed the meeting because I didn’t get the computer reminder, I wrote the wrong thing in a message because auto-complete suggested the wrong thing. I have become a selfish and uncharitable master to these machines. Much has been written in recent years about the legal challenges posed by the need for someone to take responsibility for the actions of machines52. But I feel now that the need to take responsibility is also existential. I feel better knowing that this report is mine and that I made the decisions that brought it into the world. So, this report is my report. Its strengths are drawn from the many people I met along the way but its limitations are mine alone. It is not perfectly objective. However, I hope I

52 A good survey of the literature is in Jacob Turner’s recent book https://www.palgrave.com/gp/book/9783319962344.

55

have detailed my processes enough that the strengths and weaknesses of my work can be read along with my findings. This will not be the last word written on government use of data and automated decision making systems. I hope this report is a useful step along the way.

56

Figure 13. Me at the Churchill memorial in London