AAAI/CCCSymposiumonAIforSocialGood

TalkSessions3:AIforUrbanPlanningSession Chair: Dr. Virginia Dignum

FeiFang: Todaythefirstdiscussionwillbeonurbanplanning,andweareveryfortunatetohaveVirginiaDignumtochairthesessionforus,andVirginiaisassociateprofessorattheFacultyofTechnology,PolicyandManagementattheDelftUniversityofTechnology.ShereceivedherPhDin2004fromtheUtrechtUniversity,andherresearchareaisagentbasedmodelsoforganizations,andshe'soneoftheleadingresearchersinthisarea,andshewastheco-organizerofAAMAS2005,andalsotheco-chairofEuropeanAIConferencein2016.

Andwiththatlet'swelcomeVirginiafortheopeningtalk.Andbytheway,IalsowanttothankStevenSmith,fromCMU,forhissupportforthissession,althoughhecannotmakeittothesymposium.

VirginiaDignum: Thankyou.FirstIhavetoapologizeastomakeanoteontheDelftblueoftheslides,IthinkDelftwillbeveryunhappyifthisisbeingfilmed,becauseweareveryproudofourDelftblue,anditbecomegreeninthisprojector,sodon'ttakethegreenliterally,andwehaveverystrongconnectionwithDelftblue.Anyway.Sothissessionisonurbanplanning,andurbanplanningisanincreasinglyimportantareaforAI,andforthetechnologydevelopment.Alreadytoday,morethanhalfofthepopulationoftheworldislivinginurbanareas,andexpectationisthattwothirdsofthepopulationwillbelivinginurbanareasby2045.

Oh,andthatoneisred,somethingisverywrong.Anyway.So,andthiswholemigrationofpeoplefromruralareastourbanareasismeaningincreasinglyhigherstressbeingputonthecontemporaryurbansystems,andalotneedstobedoneinordertokeepcitiesbeingsustainable,andlivable,inthecomingyears.Sothisismanagingtheurbanareas,isoneoftheveryimportantchallengeforthe21stcentury,andoneinwhichAIcandoalot.

AIandurbancomputingfocusonapplicationofintelligentcomputingtechnologytotheproblemsfacedinurbancommunitiesalsotriesdevelopsolutionsandtechnologythatcansupportsolving,oratleastminimizingthistypeofproblemsindifferentkeyareas,andtheseareasaresomeoftheareasinwhichthepapersofthesessionoftodayaregoingtobefocusingon,butIthinkbeforewereallygointothepapers,whatIwouldliketoconveyinthisshortintroductionistomyideaweneedto,notonlylookathowtosolvetheproblems,butalsotakeakindofastepbackandthinkaboutwhy,andhow,andwhatarewegoingtosolve,andwhyshouldwesolvingthat.

So,inasense,whatdoesitmeantohavegoodAIorAIforthesocialgood,appliedtourbanplanning.Oneoftheimportantissuesistolookatwhoisplanning.Inmanycase...that'sverywrongwiththisslide,somethingdidn't...

Talk Session 3 - AI for Urban Planning Page 2 of 30

Anywaythesepinkguysaresupposedtobeabunchwhiteguyssittinginsomeofficeandplanning,andthatIthinkiswhatisnotion,thatonegirlIthinkinbetween,butthey'reallwhite.Andthisisalothowurbanplanningisdonenowadays.Peoplewithexpertiseinplanning,theygoanddecideandmakethedecisionsonhowcities,andhowurbanenvironmentslooklike.

Thereis,ofcourse,alotofgoingintoparticipatoryplanning,whichistheotheroneoneachsideshouldbesayingsomethinglike,"Whatwouldyouliketobebuilthereinthisareaofcity."Itisverybadslide.Andinthisparticipatory,it'sastepinagooddirection,butalotofit,whatwenoticeisthatpeoplewhoareparticipatingarerepresentingveryspecificareasofthepopulation,sotheyaretypicallythepeoplewhojointhesekindofdiscussions,arepeoplewiththehigheducation,andmoreverbal,andmorecommittedtothesocietywheretheylive,andnotallthelowerandpoorerareas.Poorpeople,orsomeoftheneighborhoodsonwhichtheyarealotdeprivedpeopleandpopulationslivingon.

Onetheissueswithurbanplanningisthat,literally,thedecisionswearetakingonthedesignofcities,aregoingtobesetinstone,andbybeingsetinstone,orininfrastructure,theywillimpactthelifeofthesocietiesforalong,longtime.Andtheyarereallymakingabigdifference,thewayweimplement,orwedecideindecisionsforurbansystems,it'sreallytostaythereforalongtime,sothat'ssomethingwhichisveryimportanttoconsider,beforewereallyaregoingtodeterminewhattodo.Andofcourse,weallhaveverygoodintentions,andIdon'tdoubtthatthepinkguysthereareallverywellintentioned,andtheyreallyknowwhattheyaredoing,butnotalwaysthosegoodintentionsleadthisnecessarilytoagooddesign.

Andthereismuchmoretothinkaboutwhenwearetakingthosedecisions.Soexamplesofpolitics,whichare...ordecisions,whichreallymakeanimpactonthevaluesofpeople,isthoselowhangingbranches,whichareoften...orthereareallkindsofstudieswhichshowtheyareusedkindoftopreventpopulations,frompoorparts,tobestistogototheotherpartsofthecity.Youhavethoseparkbenchesinthosepicturesthere,whichyouhardlycansee,whicharedesignedexplicitlytoavoidthatpeoplesleeponthem,soit'ssomekindofavoidinghomelesspeopletotakeaspaceinthoseparks.

Youhaveallthese...youcanhardlyseethem.Allthesegates,whichprevent,oravoidpeoplewithoutthetickets,togointothemetroandthetrainstations.Soit'sallkindsofdesigns,whichareexplicitlyrepresentingsomevalues,goodorbadvalues,ordesirableorundesirablevalues,buttheyareimplementingthosevaluesinaverystrongway,ortheyarenudgingyouintakingsomekindofbehavior.Andinthesamewayasthisveryphysicalartifactshavepolicies,wehavethesamekindofpolicies,orthesamekindofvaluesbuiltintoAIsystems.Thinksaboutmoraldilemmas,andmoraloverloads,sohowwebuildsystemstodealwiththefactthatyoucannotreallysolveandachieve,atthesametime,alldifferenttypesofvalues.

Talk Session 3 - AI for Urban Planning Page 3 of 30

Andissuesaboutbias,aboutinclusionandaccessibility,andsoon.I'msureyouallhaveseeninthebest,themoralmachinerydevelopedatMIT,whichistryingtoidentifyhowself-drivingcarsshouldmakedecisions,orwillmakedecisionsbasedonwhatwe,aspeople,thinkisdesirableornot.Ofcourse,whenwedevelopalgorithmstoclassifydogs...andnowyoucannotreallyseenothinganymore.Theywillclassifyandidentifydogsinallkindsofscenes,eveninaplateofspaghetti.Youcannotreallysee,thereareallkindsofadded,showingthatitappearssomedogsthere.

So,excuseme.Sowearedoingalotofdesign,andalotdecisionsintensionally,orunintentionally,whichreallymakeadifference.Sobeforewereallylookatthedesigninurbanplanning,isoneofveryconcretearea,weshouldreallylookatwhatarethevaluesandethicalrequirements,whichweareputtingthere.Andarethosevaluesthatwewanttoimplement,ortoenforcethings,whichitmustbeimplement,shouldbeimplementorcouldbeimplemented.Also,ifwelookatreallyatethics,andgobacktophilosophy,therearemanydifferentethicalsystemswhichallleadtoverymanydifferentdecisions,soaUtilitarianAIsystemwilltakeverymanydifferentdecisionsfromaKantianAIsystem,orAristotelianAIsystem,becausetheylookatthewayofmakingdecisions,thewayofdecidingwhatisgood,whatismorallyacceptable,inaverydifferentway.

Sothosethingsareimplemented,desiredornot,inoursystems.Sooneoftheissuesthereaswell,issocialacceptanceofwhatweagreethatit'sokay.Wevoteforgovernmentsbymechanismsofsocialacceptance,andsocialchoice,andthatisnotnecessarilythesameaswhatwouldbemorallyacceptable.AndI'veseenbothintheNetherlandsandUS.Wehaveseenrecentlyverymanycasesinwhichthatisnoteffective...aligned.

SocanAIsystemsdogood?Dogoodinurbanplanningorotherways?Itdependsalotonhowwedefine,andhowgoodisunderstandingthosesituations.Therearecultural,individual,andsituationaldifference,whichshouldbetakenintoaccount.Andthesocietyforwhichwearedesigningisshapedbywhatwedo,butit'salsoshapingourdesigns,andIthinkthat'soneofthe...I'mfinishing,yes.Oneofthethingsiswecantrytoimplementethics,ormoralreasoning,inoursystems,butstilljustknowingabouttheethicsdoesn'tmean,necessarily,thatthosemachineswillbebehavingethicallythemselves,orrepresenting,orimplementingtheethicalvaluesthatwethink,orwewouldwishthemtoimplement.

AndlikeIalreadysaid,yesterday,responsibleAIisnotonlyaboutthesystemsthatwedevelop,butalsoaboutthewaywedevelopthosesystems.Wearetheoneswhoareresponsibleattheend.Andyesterday,inseveraltalks,Iheardthingslikethealgorithmtoldme,orthemachinedecide,ortheoutcomehasdeterminedthat,whichverymuchledtosingleoutthissketch,whichweprobablyallknow.Ifyoudon'tknow,pleasegoonYouTubeandlookatit,inwhichshekeepssaying,computersays,"No."Andwhateveryouaskher,she

Talk Session 3 - AI for Urban Planning Page 4 of 30

willsay,"Computersays,No."Itisveryeasy,andvery...tryingtoexcuseyourselffromthedecision.

ButIthinkwereallyneedtolookatAIfromtheperspectiveofaccountability,responsibility,andtransparency.Andwe,wearedevelopingthosesystems,aretheoneswhoareresponsibleforwhatthesystemsaregoingtodo.Thankyou.SobeforeIcontinue,maybenowisnumbertwospeakerforthepapernumbertwo,Syedisherenow?Yesokay.AndAminishere?Notyet,sowecontinueuntilnumber...

Speaker3: Numberthree.

VirginiaDignum: Numberthree,andthenweseeifwechangednumberfour,andnumberfive.OkaysoSujoy,yes.

Speaker3: [inaudible00:12:25]

VirginiaDignum: Okay.

Speaker3: Let'sthankVirginiaDignum.

VirginiaDignum: Withthe[inaudible00:12:36]standard,isasuggestion.Maybeitwillbecomeastandard,but,sothisthebasisforEuropeannetworkofexcellence,whichisjoining27countriesinEurope,andoutsideEurope,there'sfew,andaround100researchersareinvolvedinthatnetwork.Sothisiskindofthecardinalmessageofthenetwork,sowehopefully,aftertheprojectstarts,itwillbecomemorevisible.

AudienceMember: ThepresidentoftheUnitedStates,veryinterestedin.

VirginiaDignum: WedohavesomeUnitedStatesparticipantsaswell,yep.

AudienceMember: Soononeofyourslidesyousayweareresponsible,soIguessthequestionis,whoiswe?

VirginiaDignum: Yeah.

AudienceMember: Isitus,thedesigners?IsittheAIsystemitself?Isittheusers?Alloftheabove?

VirginiaDignum: Yep.Alloftheabove.Weasresearchersareresponsible,systemsshouldbeabletoatleastbeabletoindicatewhattheyareresponsiblefor,andwhattheynotresponsiblefor,soitshouldbeclearwhatarethekindofscopeofresponsibilityofasystem,butasdesigners,asresearchers,asdevelopers,asimplementers,asusers,asowners,asmanufacturers,Ithinkweallhavesomeresponsibilityinthedecisions,andthewaysthesesystemsaregoingtoaffectourlives.AndthereisnowallthishugediscussioninthemediaabouthowAIisgoingtotakeovertheworld,andreplaceallusbyrobots,orwhateverotherkindsofvery

Talk Session 3 - AI for Urban Planning Page 5 of 30

dystopicviewsofthefuture.AndIthinkthatthebestanswerwecanhavetothatkindofdiscussion,isreallytoassume,andtoindicatetheresponsibilities,andalsothelimitationsoftheresponsibilities,atalllevelsofthedesign,yes.

Okay,sothenwewelcomethefirstspeakerSujoyChatwterjee,onsmartcityplanningwithconstrainedcrowdjudgmentanalysis.

SujoyC.: Thankyou.Sogoodmorningeveryone,partofmytalkisonsmartcityplanningwithconstrainedcrowdjudgmentanalysis,andthisworkwasdoneunderthesupervisionofDr.AnirbanMukhopadhyay,AssociateprofessorofDepartmentofComputerScienceandEngineeringattheUniversityofKalyani,andDr.MalayBhattarcharyya,assistantprofessorofDepartmentofITIIEST,Shibpur,India.

Sothesearethebriefoverviewofmytalk.Solet'ssupposethatsomeonehaspostedaquestion,insocialmedia,andhe'saskingthat...hewantstoknow,andfindtheanswerofthatisthequestionis,soshowingmysontheDisneyAladdinwouldnotmakehimwanttojoinISIS,willit?SoI'llgiveyoutheanswerofthisquestion,soweknowthatcrowdsourcingcanhelpustounderstand,tosolvethedifferentreallifetasks,wehavetheproblem.Soitcanhelpusinharnessingthepowerofcrowdandutilizingthe[inaudible00:15:47]andresourceswecansolvedifferenttypeofreallifeproblems.

Sohecan...thepersoncanoutsourcethisproblem,thisquestioncrowdworker,andhecanoptimizeallthefeedbackfromthis.Sonowsupposethattherearethreepossibleoptions,thereisyes,no,andskip,andnowthequestionistherehowto,ifwehavemultipleopinions,thenhowtogetthefinalanswer,howtofigurethefinalanswerfromthismultipleopinions.Soyouseethataccordingtothemajority,votingnowillbethefinalanswer.But,sothejudgmentanalysisisbasicallyidealtosolvethisproblemtofindoutthe[inaudible00:16:26]judgmentfrommultipleopinions.SOit'sageneralwayoflearningof[premonition00:16:29]frommultipleopinions.

Sointhiscrowdsourcingenvironment,thereisaresponsematrixthatcontains...ontheleftside,thelefthandside,youcanseethattherearesoannotators,sothattheyarelyingrows,andtherearesomequestions,thatarecolumns.Andtheselfalreadyknowstheopinionsofdiscussionforparticularannotators.Sotherearefiveoptions.Thereisyes,no,skip,unsure,andIcan'ttell.Andontherighthandsideyoucanseethereisajudgmentmatrix,that'swheretherowstherearesomequestions,andbasicallytherearefiveoptionsforthisresponsematrix,sotherearefiveoptionshere.

Theselfalreadyknowstheweightofthisparticularoption,foraparticularquestion.Soyoucanseethatforquestionq1,whatwillbethefinalanswer,becausethewhatwillbethemaximum.Wehatismaximumforthisquestion?Sowhatwillbethefinalanswer,forthequestionq1?Now,inthisproblem,youcanseethattheresponsematrixhavetheoptions,yes,no,unsure,Ican'ttell,andskip.Nowindifferentreal-lifeproblem,youcanseethattherearesome

Talk Session 3 - AI for Urban Planning Page 6 of 30

questionsthathavesomesubpartsthatcomprisessomeofthecomponent...subcomponent.

Forexample,supposethatagovernmentorganizationistryingtoinstallthreeATMcountersinyourcity,orinyourlocality.Sohegatheredtheinformationabouttheappropriatelocation,orexactpublicdemand.Toknowthatexactpublicdemandisnoteasy.Sotoseekthepublicopinion,exactpublicinterest,overtheATMcounters,youcanoutsourcethisproblemtocrowd.Sothattheycangivetheirpossiblyoptions,possiblelocations.Andtherearesomeconstraint.Againthat,becausetwoadjacentATMcountercannotbesidebyside,sothereshouldbesomespecificdistancebetweenanytwoATMcounters,sothesearesomeconstraint.

Soalthoughthe[straight00:18:31]oftheirapproachdealswiththisproblem,butinthoseproblem,thequestionarebasicallyofsingleorbinaryopinion,singleormultipleopinions,butthequestiondon'thaveanycomponent.Butinthisconstrainedcase,thisisthe[inaudible00:18:48]Thiswasthenormalcase,andthisistheconstrainedcase.Inconstrainedcase,forexample,aswearediscussingaboutthreeATMcounters,orfiveATMcounters,sotherearethreedouble...thisiscalledtripletofdoubles.Sobecausetherearetwopossible...threelocations,withXandYcoordinate.Thatisforfirstlocation.Thisisforsecondlocations,andthisisforthirdlocation,andthereshouldbesomedistance,andthereshouldbesomerelationsfordoingthis.

Sowhileattemptingthequestion,the[calculator00:19:19]shouldalsosatisfytheconstrainedalso.Sothisbecomesachallengingproblem.Nowtheproblemisthathowtoindicatethissolution,fromthismultiplesolution,howtoindicatethefinalsolution.Soinmeasuringresponseinthenormalcase,theresponsematrix,wecanseethatifwecanapplymajorityvotingforthepastquestion,yeswillbethefinalanswer,becausetwopercentaregivingyes.Butinthiscase,majorityvotingfail.Althoughthe...allofthestraightupdataapproachesfailshere,becauseifthenumberofrangebecomeshigh,thenitwillbetoomuchtoughtofindoutthemajorityvoting,andit'saverylesschancetorepeatthisopinionforallofthesecomponent.

Sothereisnochancetoapplymajorityvoting,sowehavetorelyonanothermethod.Weareproposinganothermethod.Sothesearesomeofthesechallenges,andthesearetheproblemsfromrelation,andotherthingisthat,inthiscase,forthisresponsematrix,fornormalcase,therearesomeoptionstherealso.Butinthiscasewearejusttryingtosettherangeofthisoption[inaudible00:20:27]Thereisastartingrangeandendingrange,butwe...astartingpointandendingpoint,butwedon'tknowthatwhatwillbethefinaloptions.Sofirstwehavetohelpwiththeoptions,becausewithouttheseoptionsthere,wecan'thearthejudgmentmatrix.

Sointhisproblemformulation,thereisasetofquestions,asetofannotators,andwearethesetofopinionvector,wehavecreatedthisopinionvectorby

Talk Session 3 - AI for Urban Planning Page 7 of 30

baseendbinning,tofindtheoptimalnumberofoptions.Sofinallywehavetocomputethefinal[inaudible00:21:00]judgmentforeachofthesequestion.Sowehavetocomputetheopinionvector,wehavecomputedtheGaussianbinning,toscalethislargescaleofopinionsintoasmallvectors.Butduetothe...

ThisisthePGMmodel,wehaveconsideredannotatedaccuracy,andquestiondifficulty,andthesearethequestioncomponentofdifferentquestions.Andthisisthejudgmentmatrixforthis[tobetest00:21:30]thatisforX/Ycoordinate.Xcoordinatethisisrow,columnwisethisisYcoordinate,butduetothetimeconstraint,Iamnotgoingtothedeeperendofthismodel.Therearedifferentfactors,thatisaccuracy,andquestiondifficulty.Wehaveusedthelogisticformula,andwehaveusedtheexpectationestimationalgorithmhere.Andtheaccuracyplaysavitalrole,andforthiscase,wehaveusedthecoverageofthis...actuallythecoverageofthispoint,thatiscoverageofthisATMcounter,forexample,thatifthepointsarewelldistributed,thatmeansthatitwillbe...everyonewillbeservedbytheseATMcounters,sothat'swhatyoucallareaoftheentire,enclosedbythisthreepointwillbetakenintoaccount.

Soforthiscaseofcreationofthisdataset,wehavepostedagridmap,andthisisthequestionthatatopUSuniversitywishestostartthethreeextensioncenter,andpleasegiveyouropinionaboutthepossiblelocation,thereshouldbesomeconstraint.Andafterthatthecreationofthisjudgmentmatrix,wecangetsomescoreofthismatrixbasedonthe...andwecantosortthescore,basedonthisposteriordistributional[inaudible00:22:45]andforeachofthesecomponentwecangaugesomesolutions,andthebestsolutioniscalledaRank-1solution.AndthenwearecomputingthisRank-1solutionforbothofthesecomponent.Andwecancheckthatwhetherthissatisfytheconstraint.

Soforexample,itwillbeRank-1,andforthiscomponenttwo,itwillbeRank-2,andforcomponentthree,itwillbeRank-3.Andifyousatisfythatconstraintthenitwillbetreatedasthefinalanswer.Sothesearetheopinionsfor[thefaster00:23:15]thisisthe[roaming00:23:16]ofoptions.Thatis,wearegettingtheseoptionsfrommultipleopinions,multiplecrowdworker,thisisafterbaseendbinning,andforinthiscasewegettingtheRank-1solution,andthisisforRank-1andRank-2solutions.Soforthiscasewehaveconsideredtheaccuracy,andthequestiondifficultyofthisques-...ofaccuracyandofthe[inaudible00:23:43]andquestiondifficulty,butwecantrytofindoutthevastnessofthe[inaudible00:23:47]andotherdifferentparameterthatcaneffecttheoveralljudgment,andthisisthereferencesthatIhaveused.Andthankyou.

AudienceMember: Becauseyou,intheend,getthreelocationschosen,butaretheythebestlocations?Aretheyusablelocations?[crosstalk00:24:13]Doyouknowthatpeoplethatactuallyvoteforthethingshaveanyinfluence,maybenoneofthemhasanythingtodowithuniversity,butthinkit'sfunnyto...

Talk Session 3 - AI for Urban Planning Page 8 of 30

SujoyC.: Forthefirstcasethereis...whenwearefindingtheaccuracyofthisannotator,firsttothemeanvalueiscomputed,andthedeviationfrommeanvalue,toannotatethatisour...thequalityofthisannotatorareactuallycomputed,sothiscanremovetheannotatorwhowas...orthecrowdworkerwhoarenotgivingtheiropinion,orwhoarenotconfidentontheiropinion.Sointhiswaywecanjustifythequalityofthiswork.

VirginiaDignum: Howdoesyourworkcompareto[counterset00:25:00]mechanisms?

SujoyC.: Thereisnothingabout[counterset00:25:06]mechanism.

VirginiaDignum: Becauseyouareranking,andthentryingtobethecreator.

SujoyC.: Well...no,no,no.Yeah,thisisthe-

VirginiaDignum: Theaverageoftherankings.

SujoyC.: No,wehavenotused[counterset00:25:13]ranking.Wehavejustsortedtheposteriordistribution[inaudible00:25:17]foraparticularoption,thatisforjudgmentmatrix.

VirginiaDignum: Yeah.

SujoyC.: Forthiscase,ifthisisthemaximumposteriorvalue.0.06.Thatmeansof40and40.40comma40,willbethefasteroptionsforthecomponent.For[analytic00:25:35]component,itwillbethat40comma40.30comma50willbethebestpossibleposteriordistribution.Sowearesortingthisvalueindescendingorder,andthetopwillbetheRank-1solutionforthisparticularcomponent.Andinthiswaywearecombingthis...checkingthesatisfying,whethertherelationissatisfiedornot.

VirginiaDignum: Okay.Anyothercomments?

AudienceMember: Justaquestionaboutthepossiblesolutionsthatyoucangetfromthesystem.Isittruethatthesolutionthatyouget,willmatchoneparticularannotatorsolution?

SujoyC.: No.

AudienceMember: Orwilltheeventbeacombinationof-

SujoyC.: Yeah,it'sagoodquestion.BecauseIwasexpectingthisquestion,becauseiftheopinionsarediverse,thenitisverylesschancethatopinionscanbelongtoanyoftheseparticular[datically00:26:38]matchtotheanyoftheseparticularannotator,andiftheopinioncenterbecomesveryless,thanthereisnochancethatitmightbefromthisopinionvector,wearegettingfromthisinput.

Talk Session 3 - AI for Urban Planning Page 9 of 30

AudienceMember: But,thereisnosenseofcentroidcombina-...thatit'sgoingtobeachoicethatsomeonehasactuallysaid,notanaverageofmultiplechoices.

SujoyC.: No,forthefirstcase...fortheinitial...'causeasyousawithere,iterationwise,wearecomputingtheaccuracy.Forthefirstcasewearecomputingtheirdeviationfromthismean,afterthat,accuracyofthisannotator,orthemeasurebythecoverage.Soaftersomeiterationitcanbeseenthat...youcanseethatfromthisgraph,fromthisplot,thatitmaybethatthisisnottheexactly,fortheredcross,thisisnotinthemean[case00:27:30]but,thisisleftaligned,thatmeansthatalthoughthemeansolutionexists,butisnotagoodannotator.That'swhysomeothersolutionsaretreatedasthefinalsolution,basedonthiscoverageandaccuracy.

AudienceMember: Okay,thanks.

VirginiaDignum: Anyotherquestions?Okay.

AudienceMember: Inknowyou'rerunningoutoftime,butIwantedtoknowwhat'sontheXandY?What'sontheX,andYaxis,andhowtoreadthesegraphs?

SujoyC.: Yeah,thisistheXaxisforlocation,Xcoordinate,thisisforYcoordinate.

VirginiaDignum: Sothatmapplot.That-

AudienceMember: Whatarethedifferentcolors?

SujoyC.: Thedifferentcolorsmeansthereisaopinion...Therearethreepossiblelocations,thatisforfirstratingencounter,thered.Thegreenaresecondratingencounters.Andthebluearethesethirdratingencounters.

AudienceMember: Thankyou.

SujoyC.: Thankyou.

SyedAliR.: So,sofarwehavetalkedalittlebitabouttheGDPRregulationsthatEUhasjustreleased,andso,inthispaper,whatwewantedtotalkalittlebitaboutiswhatarethemorerelevantsectionsthatwefoundinthatregulation,andwhatarethethingsthatweareconcernedabout,andwhatwethinktheseregulationsmightinadvertentlycausemorediscriminationinthemachinelearningapplicationsthatthey'retryingtomitigate.Solet'sstartofffirstoffbylookingat...

Okay,sowhilethatisbeingsortedout.Sowhatbasicallywewant[inaudible00:29:24]Ah,yeah.Sowhatwefirstwantedtodefinewaslikewhatisdiscrimination?Andtherearethreemaincomponentsofwhatdiscriminationis.Thereisthatthealgorithm,whateveryou'reusing,hasasignificantblindspot.Thesecondthingisthat,thatblindspotadverselyaffectsaparticularsectorof

Talk Session 3 - AI for Urban Planning Page 10 of 30

yourpopulation.Sotherearethreethings,itadverselyaffectspeople,acertainsectionofpeople,anditisarisingoutofablindspotinthealgorithm,andwearedefiningthatasdiscriminationinamachinelearningalgorithm.

AndsotheGDPRregulationsareactuallytryingpreventthissortofdiscrimination,andtheyareinthatrim,theyhavecomeupwithaverylong,detailedupdatetoregulationstheyintroducedbackin2007,andthesewere...Oh,brilliant,thanks.

Sowehavelookedatthis.NowintheGDPRregulationsthattheyhaveintroduced,thereareacoupleofdefinitions,andthesetwoarethemostimportantonesthatIthinkweneedtoknow,thatconcernprocessing,andprofiling.Soboththesedefinitionshaveverybroadramifications,theseareverygeneraldefinitions,andI'lljustgiveyouamomenttojustreadthemastheyare.

Sothefirstone,whichisaboutprocessing,concernsthatanyset,whichiswhetherornot,byautomatedmeans,issubjecttothis.Anditincludesthingslikecollecting,recording,organizing,structuring,storage,adaptation,alteration,retrieval,consultation,use,disclosure.Sothere'salotthatit'scovering,inprocessing.

Andtheninprofiling,whatitsayingisthat,anydatasubject,whoissubjectthisprocessing,isprofiledbyanalgorithm,whenthey'resubjecttoanydecisionthatismadebythatprocessing,sothat'sprofiling.Andinparticular,whenwetrytoanalyze,orpredictaspects,abouttheperson'sperformanceatwork,economicsituation,sothingslikeinsurancechecks,health,soanyhealthapplications.Personalpreferences,soadstargeting,interests,reliability,behavior,performance,soallthesethings,includinglocations,ormovements,soincludingpeoplewhomightbemigrants,andyouwanttotrackthings.They'reallsubjecttothisregulation.

Andsowecanseeit'saverybroadregulation,andthat'swhyit'simportanttoknowsomeofthemoreindepththingsthatitcovers.Sothefirstthingthatis,it'sarightofaccess,bythedatasubject.Sowhatthiscreatesisthatthedatasubjectcandecidehowtheywanttheirpersonaldatatobeaccessed,orwhethertheywantthatdatatopartofsomeprocessing.Andsothat'sthefirstandforemostrightthatthisregulationintroduces,controloveryourdata.

Thesecondthingitintroducesistherighttoobject.Youcanobject,andopt-outofyourdatabeingprocessedbyanyofthemeansthatweredescribedinthepreviousslide.Andthethird,andmostimportantthingsthatitintroduces,isautomatedindividualdecisionmaking,andarighttoexplanation,essentially.Sothisisthethingthatcreatesarighttoexplanation.Sothedatasubjecthasarightthatiftheysubjecttoadecision,thattheycanaskwhatexactlyistheexplanationofwhattheyarebeingsubjectedto.

Talk Session 3 - AI for Urban Planning Page 11 of 30

Andthisisatoughone.Sonowlet'sbringthisastepback,andjustlookforamomentatwhatthismeansforthemachinelearninglifecycle.Sothisisthebroadgeneralmachinelearninglifecyclewheretheoutsideblueiswhatthealgorithmdoesinthiscycle,andtheinsideiswhatthedatadoesinthislifecycle,andthey'rebothentwinedlifecycles.Theybothgothroughaprocessofintegration,wherewegetthedata,andthealgorithm,wemakeupdates.Weimplementit,weprocessthedata,andallofthesesectionsarebeingeffected,buttwoofthemarebeingeffectedthemost.

Inthedatalifecycle,becauseoftheirrighttoopt-out,andthedatalifecyclegetseffectedattheseparatepartofdatacollectionthemost.Andinthealgorithmlifecyclethelast,third,kindofregulationthatIshowed,thealgorithmlifecyclegetseffectedatthisstageofthealgorithmselection.Andsowhatdothesetwothingsendupcreating?

Sothefirstthingendsupcreatingadiscriminationrisk,becausetheoriginsofthedataaresortofnownotrepresentativeofthepopulationsyou'retryingtoeffect.Nowthinkofitthroughanexample,ifyouareahealthcarekindofstartup,andyou'retryingtoimplementanalgorithm,yourdatamightnotcontaincertaintypeofpeopleanymore,becausetheyhavedecidedtoopt-outforwhateverreasonsthattheymightpersonallyhave,thatmaybethisdata,ifIreleaseit,hassomeinsurancecosts,butbecausenowthatdataisoutofyouraccess,youralgorithmcannevercapturethosethings,becauseitjustdoesn'tseethem,orenoughofthem.

Sothat'sonethingthat,whenyouhavebiascomingin,youwillhavebiascomingout.Andsoinadvertentlywesee,andwethink,thatthisregulationcouldcreatethisinadvertentconsequence.Theotherthingthatit...So,andhowdoyoumanagethatbias?Well,youhavetobeverymindfulofwhereyourdataisbeinggeneratedfrom,andhowareyouusingotherdatasetstofillthosegaps?Becausethey'reextremelygood...there'sextremelygoodresearchhappeningrightnow,whichistryingcoverhowtocovergapswhereyourdataininsufficient,butthenweneedtobehyper-awareoftheoriginsofthedata.

Thesecondthingthatitcreates,theblack-boxrisk,itexacerbatesthat,becausecurrentlyweareusingalgorithms,thatwearemakingthemmore,andmoreadvanced.Morecomplex.Wemaynothaveinterpretabilitypreciselyforthem.Somaybe,forsomeapplications,choosingasimpleralgorithm,whichiseasiertoexplain,but,whichmightlackthenuancefortheedgecasesforwhichwecreatedthecomplexalgorithmsinthefirstplace.Sothereisaselectionbiasproblem,andalotofthisworkissortoftryingtostartaconversationonwhatdoesthisactuallymeanforus,asdatascientists?

AndsoIthankyouforyourattention.

VirginiaDignum: Anyquestions?Yes?

Talk Session 3 - AI for Urban Planning Page 12 of 30

AudienceMember: Haveyoudoneanyapplication?

SyedAliR.: So,theapplicationsthatwe...mypersonalPhDresearchissortofapartfromthis,Iworkwithfinancialdata,butoverthelastsummer,inChicago,IworkedwithDataScienceforSocialGood,andthere,alotofapplicationsthatwewereworkingon,wouldhavedirectconsequencesifthisregulationwas,let'ssay,appliedtotheUS.Becausethereweareworkingwithschoolkidsdata,policingdata,andifyoujustforamomentimaginethatyoutakeoutwholechunksofthepopulation,inthat.Becauseifyouhavethatright,youprobablywouldwanttoexercisetheserights,anditwouldcreatebiasesinthedata,sothat'swherethe,kindof,ideastarted,thattheseapplicationswouldmostimmediatelygeteffected,bytheseregulations.

AudienceMember: Sothere'sarecent...sorecentworkbyJonKleinbergshowsthatsomepossibilitytheoremsthat,ifyoulookatthecriteriaforwhat...for,exampleseven[inaudible00:37:43]guidelinesfittingnondiscriminatory,youcannotsatisfyallthecriteriasimultaneously.SoIwasjustwondering,ifgettingintothinkingthatwhethertheseregulationsmightactuallybeimpossible.

SyedAliR.: Sothat'sanexcellentpoint,andIagreewithyou,toacertainextent.That,maybe,thereareasetofconstraints,whichmightbeimpossibletosufficientlyexercise,andoneofthebiggestsections,oftheregulations,thatmightbeveryhardsimplify,istheinterpretabilityproblem,because...andsomeoneusedthisexampleyesterday,thatgoingtoadoctor,wekindofgetawatereddownexplanation,butalotofusleavefeelingveryunsatisfied,butthatonlyhappensbecausewedon'thaveanotherchoice.Thereisnoalternativedoctors,thatexist,thatwecangoto.

Butinthecaseofautomatedprocessing,thereisanalternative,youjustoptoutofbeingprocessedthatway.Butthatbringsusbacktosquareone,wherethenwearebackintotherealmofmanualprocessing,andthenthekindofprogressthatwearemaking,indatascience,becomes,sortof,nullified.

VirginiaDignum: Justoneissueonthebias,humangenerateddata,orhumanrelateddata,isnecessarily,andbydefinition,bias.'causepeoplearebias,andthebiasappearinthedata,thatyoucollect,orthatyoustore,orwhatever.Asyougoaboutremovingthebiasfromthedata,youarekindofintroducingthebiasoftheoneswhoaredeterminewhichbiasshouldgooutofthedata.YouunderstandwhatImean?Howdoyouplan,orwhataretheways,togoaboutthat?

SyedAliR.: SotherearethreethingsIwouldsaythatrelatetoyourquestion.Thefirstthingisthatmachinelearning,ordatascienceasweknow,isnomagic.Itwon'tautomaticallymarginalizeouthumanbias,itwillpropagateit,anditwillformalizeit,ifthereisthat,andthatisyourapplication,thentalkingtoexpertsis,ofcourse,thewaytogo,anditcanbeverychallengingattimes,becauseyouhaveapersonalmoralcompass,andwhenwesawthepredictivepolicingapplication,thesensethatthereissomethingickyaboutit,thatarises,andyou

Talk Session 3 - AI for Urban Planning Page 13 of 30

mighthaveamismatchwithyourclient'sexpectation,versuswhatyouwanttheworldtobe.Andthat'sonebigproblem,whichIdon'thaveananswerfor.

Butthesecondthingisthattheinherentdatabias,ofcoursethereneedstobequalitymechanismsthatwe,asdatascientists,needtocomeupwith,thatthisdatanowpassesaqualitythreshold,anditwon'thaveunnecessarybiases,justthatareartifactsofthedatacollectionprocess.That'sonething.Andthesecondthingthatisveryimportant,istoactuallysee,whatdoesourerrormarginmeannow?Becauseit'sveryeasytosay,"Ohwell,I'mgoingfroma95%to95.1%error,maybeit'snotthatmuch."Butlikewesawyesterday,goingfroma93%toa47%percentper-...orsorry,a83%precision,issomethingthathappens,butwhatisthat2%,thatismaybe20peopleslives,soamIwillingtomakethatcall?

Sonowourprecision,indatascience,takesawholenewmeaning,becauseyouarenolongerjustconcernedwith3,000moreimagesbeingclassified,you'reactuallytalkingabout20peoplemaybebeingreleased,ornotreleased,basedonyouralgorithmchoice.

AudienceMember: SoIhaveathoughtthatemergesfromthis,thatIhaven'theardexpressedyet,andfromtheperspectiveofsocialwork,andothersocialjusticeorientedsocialscientists,wearenotonlyconcernedaboutbiascomingoutoftheseprocesses,butwe'realsointerestedinuncoveringbiasthatmaynothavebeenknown.Soforexample,inhealthcareaccess,orincriminaljustice,wehavepoliciesthatarelaws,thatareintendedtopreventbias.Youknow,you'renotsupposedtohavepolicingpractices,whichareinfactracist,orsentencingpractices,whichareIfactracist,andsoIwonderifoneoftheunintendedgoodconsequences,ofthissortofdata,sortoflookthatyouallaredoing,isthatyoumaybeuncoveringbiasesinsystemsthatthinkthattheydonothavebias.

Soratherthan,sortof,despairingthat,oh,ouralgorithm,itturnsout,inpredictivepolicing,istellingusbasicallythatifyou'reblack,you'renevergoingtogeta...you'renevergoingtobereleasedonyourownrecognizance.Infact,whatthatreveals,isthatthesystemthatwasbuiltoffofthedataofthesepractices,meansthatifyou'reblack,younevergetreleasedonyourownrecognizance,anditactuallyquantifies,andmeasures,thissortofbias,thatpeopledon'tthinkisthere.Anditstrikesmethatthisisactuallyawholenewareaforpursuit,thatyou'dac-...thepursuitofbias,insystems,andthenobviouslyyouwanttotrytothenmakedecisionsrulesthatcomeoutofthat,thatdon'thavethosebiases.

ButIthinkyouallareinapositiontoidentifythings,whichsystems,whichthinkthattheyarenotdoing,youknowthatthey'renotbiased,areactuallybiased.AndI'mwonderingifthisis,inanywayshapeorform,partofthedialogue.

SyedAliR.: SoIcompletelyagreewithyou,anditis,especiallyaparticularpaperthatreallyinspiredus,andwereferenceit,it'scalled:Istechracist?Thefightagainst

Talk Session 3 - AI for Urban Planning Page 14 of 30

digitaldiscrimination.Andinit,ortheworkbehindit,triestouncoverwhetherfundamentallytheprocesswearetryingtodescribeisinherentlydiscriminatory,andsoarewegoingtojustossifythediscrimination?ButI'mawareoftime,sothankyousoverymuchforlistening.

DanielaRosu: Let'ssee.Actuallyyeah,Icanuseyours.Okay.Allright,let'ssee.Idon'tlikehow...itlikeerases.

Helloeveryone,goodmorning.MynameisDanielaRosu,Iamthefirstauthoronthispaper.I'mwith...I'ma[professor00:44:44]fellowwiththeCenterforSocialServicesEngineeringattheUniversityofToronto.We'reagroupofcomputerscientists,mostly,butalsoindustrialengineerslookingatapplyingcomputerscience,andengineeringprinciplestoefficient,andhopefullyeffective,designanddelivery,ofsocialservices.It'saveryambitiousgoal.Wehavesomepeoplefromthefacultyofsocialwork,aswell,workingwithus,andwhatmotivatedthisparticularwork.

Sojusttoletyouknow,thisisabigprojectthatI'mgoingtotalkabouttoday.Theindividualtechnicalpartsareactuallybeingsubmittedastechnicalarticlesatvariousconferences.Thisisjustanoverviewoftheproblemwe'retryingtotackle,andthesolutionsthatwe'reproposing.Whatmotivatedourworkwasthefactthatanincreasingnumberofpeople,wecallthosevulnerablepopulations,areinneedofproducts,andservices,theycannolongeraffordtobuy.Andbyvulnerablepopulationswereallymean,youknow,thepeoplewhoarenottrulyprivileged,theyliveinpoverty,thefrailelderly,peoplewithverycomplexhealthneeds,thehomeless.

Andforus,inCanada,it'sarealissue...well,forEuropeansaswell.Thefactthatlastyear,accordingtotheestimatesoftheUnitedNations,65millionpeople,morethanatanyothertimesincetheSecondWorldWar,so65millionpeople,wererefugees.AndCanadagotabout40,000ofthem,inavery,veryshortspanoftime,mostofthemactuallysettledinToronto,andmanyofthemwereprivatelysponsored,whichmeansGroupofFive,Canadiancitizens,wouldsponsorafamilytocomeintothecountry,andtheyhadtoprovideeverythingforthem,car,apartment,furniture,clothing.Getthemenrolledinschool.Get'emtutorsfortheEnglishlanguage.Everythingyoucanimaginehadtobesuppliedbythisfiveprivateindividuals.

Butontheotherhand,weliveinaveryrichsociety,atleasthereinNorthAmerica,andalsoinEurope,we'reveryprivileged.However,we'renotverygoodatdistributingthiswealthofresourcestothepeoplewhoactuallyneedthem.So,youknow,wefigurethatsincewebearit...likeourcenterisacenterforsocialservicesengineering,we'lltryandlookintothisissue,andwefiguredthatthethingsthatwe'rereally,reallypassionateaboutsolvingarethefollowing.

Talk Session 3 - AI for Urban Planning Page 15 of 30

Howdo...notjustdeterminingtheactivedemand,whatpeopleknowtheyneed,whatpeopledon'treallyknowtheyneed,yet.Andanexamplethatmy[professorial00:47:32]supervisorisveryfondofgivingis,youhavethisSyrianpregnantladycomingwithherfamilyintoCanada,shedoesn'tknowsheneedsacarseat.Shedoesn'tneed...shewillneedavaccinationforhernewborn.Shedoesn'tneedanumberofthings...shedoesn'tknowshewillneedanumberofthings.Sothisislatentdemandthatwe'retryingtoactuallyanticipate,andfulfill.

Sowewanttobeabletoidentifytheproblemsthatpeopleneedtohavesolved,notjusthowwe'regoingtodelivertothem,youknow,amattress,oratutorfortheirkid,orsomethinglikethat.Also,howdowediscoverthewholesupplyside.Again,wedoknowwhattheactivesupplysideis,buthowcanwelearnmodelsofwhatpeoplemighthave.Whoarethesepeoplewhomighthavegoods,orbeabletoprovideservices?Andtheydon'tevenknowit,right,becausenobodyhasaskedthem,theyneveractuallythoughtaboutit.

Andofcourse,afterwefigurethisout,howdowebestmatch,andallocatethem?Thisisveryimportant.Howdoweallocatesupplytodemand?Sowehaveaplethoraofdemandandsupply,butchoosingthebestmatch,basedonneeds,basedonsubsumption,orequivalenceofproduct,servicesfunctionality,etcetera.Thisisvery,veryintriguing,andI'mverypassionateaboutthis,youcantell.Andofcourse,lastbutnotleast,howdowetransfertheproduct,andservices,fromwherethesupplyistowherethedemandis?Soalotonourplate.

Whilewewanttoactuallyproposeasolution,isthenextgenerationmarketplace.We'renottryingtobeKijiji,we'renottryingtobeaCraigslist.ThereareproblemsthatCraigslist,andKijijisolveverywell,thisisnotwhatwe'reallabout.Sowewanttobeabletomatchthedemandsside,andthesupplyside,andwheneverneeded,wewanttobeabletoschedulevolunteersforeitherthedeliveryofgoods,oraprovisionofservices.Ofcoursethisisallveryambitious,we'reyearsfarfrombeingabletosolvethat,butwestartedlookingatsomeconcreteexamplesthatwecanactuallytackle.

Oneofthefirstoneswas...no,itstartedfromthefoundation,foundationwork.Howdowerepresentdemandandsupply?Andherewehaveanumberofissues,becauseinordertoactuallyattempthighqualitymatchofsupplyanddemand,wereallyneedverydetaileddescriptionsofwhatthegoodsandservicesarethatwe'retryingtomatch.Andhere,ifyou'veeverlookedonCraigslist,orKijiji,orevenonthewebsitesofserviceproviders,likegovernmentorganizations,orNGO's,theterminologythatpeopleuseisveryambiguous.Even,wehumans,scratchourheadssometimesandarelike,whatisthis?Notsure.

Manyofthetermsarereallyadhuc,likeyou...it'sinverycontextual,likeyouwouldn'tevenknowfromonecontexttotheotherwhattheyactuallymean,

Talk Session 3 - AI for Urban Planning Page 16 of 30

andhowtheyrelatetoeachother.Intermsofharmonization,howdoweintegrateallthosedescriptions,andmakethemworkwitheachother?Andofcoursebecausemanyofusareinknowledgerepresentationandreasoningfield,classicalAIthatis,non-machinelearning,wellwe'redoingmachinelearningaswell.Weuseformalrepresentation,oranthologies,forgoodsandservices,andnotjustthat,butthisissomethingthatwe'reactuallyfocusingonrightnow.

Asfaras...sothisissortofaprogressreport,thisiswherewe'reat.Itoldyoutheloftygoalsthatwehaveinmind,I'mgonnatellyounowalittlebitaboutwhatwe'reactuallydoing.Intermsofmatchingandallocation,thebiggestissuethatwehave,isthatthereisnoperfectmatchbetweensupplyanddemand,asfaraswecouldtell,whichmeanswehavetoreallyfigureoutawaytodefine,firstofall,whatthebestmatchis,andthatis,again,contextual.Andalsobecausewewantedtomakethisasunbiasedaprocessaspossible,wefiguredwewouldtryanduseformalmethodsforthat,sowearetryingtousetheequivalenceandsubsumptionofproductandservices,asgivenbyformalrepresentation,so[versatile00:52:02]logic,inthiscase,forus.

Wealso...maybeI...yeah,no.Thiswecanskip.Soasfarasrepresentingandsolvingourlogisticsproblems,sohowwehavethisfleetofvolunteers,theyhavecarswhenyouneedproperties,theyalsotheirnewconstraints.Whatisitthatwe'retryingtosolve?Soevenrepresentingthisproblem,actuallyhadit'schallenges,andI'lltalkaboutitalittlebitlater.So,theverybasicwouldbe,nowhowdoweschedulethepickupanddeliveryofgoods?Andyou'llseethatit'sactuallynotthateasy.Ourcolleagues,ProfessorBeck,andhisstudent,useconstrainprogrammingfordoingso.Actuallyit'sahybridmodel,I'mgonnatalkaboutitalittlebitlater.

Comingbacktoactuallymyareaofexpertise,wetriedto...westarteddiggingdeeperintoissuesrelatedtorepresentingproductsandservices,sowewantedtoknow.Wedividedthepieintosmallerchunksthatwecouldingest,andthisislikethesmallestonethatIcouldactuallylookatfirst.Someoftheissuesareactuallyquiteintriguing.Thedemandsidesneedtoknowtheconditionoftheusedgoodsoffered.AgainifyoulookatCraigslistandKijiji,peopledescribethedifferentgoodsthattheyhavetooffer,inavarietyofways,verycreativeways,becausemanytimestheywanttoactually...theywanttobetruthfulandsay,yeahthere'salittlebitofdamagehereandthere,butthelanguagethattheyuseisnot...theymakeitsoundlikethedamageisactuallylighterthanitis,andespeciallyif...

Soonethingthatwediscoveredisfurniturebanks,inToronto,donotacceptbabyfurnitureofanykind,soyoucannotfindthereacribs,youcannotfindstrollers,youcannotfindcarseats.Wehavetosourcethose,fortherefugees,fromthegeneralpopulation.Butweneedtoknow,right?Wasthisasubject...wasthiscarseatsubjectofarecall?Isitingoodcondition?Isitsomethingthatpeoplecanuse?Samewiththecrib,isitsafe?Andit'sjustbasedonthedescriptionspeoplegive,that'snotimmediatelyobvioushowtodo.

Talk Session 3 - AI for Urban Planning Page 17 of 30

Also,checkingeligibilityfortransportation.Sorecallwehaveafleetofvolunteers,andsomeofthemhavepickuptrucks,someofthemhaveminivans,someofthemhavejust,youknow,sedans.Forus,oneoftheissueswas,okay,alotofpeople,atleastinToronto,useIkeafurniture.Canwedismantlethosepiecesoffurniture,sothattheyactuallyfitinthecar?Sothatwassomethingimportant.ThisisnotinformationthatpeopleusuallygiveyouonCraigslist,onKijiji.Weneededtobeabletosourcethat.Andalsowhenweschedulepeopletobetransported,let'ssaytomedicalappointments,iftheyhavemobilityissues,weneedtobeabletodothat.

HowmanyminutesdoIhave?

VirginiaDignum: No.

DanielaRosu: No,okay.Allrightso,justonemore.Sowe'vedoneasmuchaswecould,sowehaveaprototype,weimplementedthatusingaBlackboardarchitecture,andit'sonline.We'reactuallypilotingthiswithanNGOinToronto,andwewanttodosomefutureworkaswell.Thankyou.Wow,thatwasfast.

VirginiaDignum: Okay,questions?

DanielaRosu: Actually,maybethisone,yeah.

AudienceMember: Hi,firstoffIwouldliketocommendourneighborstothenorthintheirapproach,andproactiveresponsetotheSyrianrefugeecrisis.

DanielaRosu: Thankyou.

AudienceMember: It'sbeenmuchmoreeffective,Iwouldthink,thanourcountryhasbeenabletodo,so...Second,therewasaverygoodNewYorkTimesarticlebyCantorandEinhorn,thispastSaturday,aboutwhat'sbeendubbedinCanada,Month13,whichistheseyearlongadoptedrefugees.

DanielaRosu: Yeah.

AudienceMember: Are...haveacontractofoneyear,andthetransitionfrommonth12to13hasbeenasocial,kindof,crisis,atleastinCanada,aboutwhattodoaboutthis.Andnowthesepeoplearehere,they'vebeenhere,they'vebeenassimilated,tosomedegree,theculture,andtheirsociety,howdoyoudealwiththat?Isthereanyroleinthismarketplace,sinceyou'redealingwitheconomicsofrefugeemanagement,isthereanyrolethatthatcanplaytohelpmitigatethatproblem?

DanielaRosu: Absolutely.SothisworkthatI'vejusttalkedtoyouabout,weactuallyhaveprogrammerswithus.Soatthebeginningitwasaresearchprototype,thatIandundergradstudents,helpeddevelop.Nowwehaveafulltimeprogrammersworkingwithus,sothisisbeingpilotedbyafewsettlementagenciesinToronto,andthat'sexactlytotryandalleviatethisshock,'causeafter12...so

Talk Session 3 - AI for Urban Planning Page 18 of 30

whatIdidn'tsay,isthat,thisfiveindividuals,whoareprivatesponsors,havetheobligationtotakecareofthisfamilyfor12months.Everythingthisfamilyneeds,thesponsorshavetoprovide,buttheirlegalobligationstopsafter12months.

So...but,thesepeoplestillneedavarietyofservicesandgoodsafterthat.Yeah,absolutely.Andthisiswhatwe'retryingtodo.It'sanongoingprocess,it'snotlike,yeah,wegottheserefugeesin,andnow,youknow,it'snotourproblemanymore.

VirginiaDignum: Anyotherquestions?Doyouhaveanyresultsalready,fromtheworkyouaredoingwiththeNGO?

DanielaRosu: No,notyet,'causeit'slikeliterally,soIwas...Ijustgotatext,asIwassittinghere,andsomebodywastellingme,yeah,wehavethegoahead,we'lldeploythis.So,notyet.Andwe'rereallylookingforwardtohavingthisuse,becauseallthiswork,whichrequiresmachinelearning,cannotbedonewithoutthedatathatwe...

VirginiaDignum: Yeah.

DanielaRosu: 'causerightnow,wejusthavearun,andtest,justwithsyntheticdata.

VirginiaDignum: Sure.

DanielaRosu: Sowe'redoingverygood,butsyntheticdata.Yes?Ithinkitwasa...oh,sorry.

AudienceMember: Soit'swonderfulthatyouhavethiscenter.Iguess,asyouknow,therearethesecentersthatarecomingup,wehaveaoneatUSC.Istheresomethingthatcanbedone'crosscenters?Istheresome,likeyouknow-

DanielaRosu: Iwouldloveto.

AudienceMember: Somesortofacollaboration,butI'mnotexactlysurewhatstructureitwouldtake,anditcertainly,youknowthedeansigningamemorandumofunderstanding,orsomething,thatdoesn'treallykindofadvancescientificcollaboration,orotherkindofcollaborations,exactly,soitsortofgottoberesearchertoresearcher,butIdon'tknow...soImeanthismightbesomethingthatcouldbetakenuptomorrow,but-

DanielaRosu: Oh,absolutely.Iwouldlovetotalktoyou,andalsototheirdeanaboutit.

VirginiaDignum: Okay,let'sgo.Yeah,okay.Soshestart,andthenit'shimandthen...

AudienceMember: Ijustwantedtoknowhowyouweregettingtheadditionalmetadatathatyouneedtomatchsupplyanddemand,inthiscontext?

Talk Session 3 - AI for Urban Planning Page 19 of 30

DanielaRosu: Sowe'vehadalotofinterviews,weactuallytalkedtosettlementagencies,andinparticularwithsettlementworkers,'causetheyaretheoneswhoknowwhattheneedsoftherefugeesreallyare.Ihaven'tpersonallytalkedtorefugeesdirectly,butIalsotalkedtosomeoftheprivatesponsors.Actually,oneofthewebdevelopersthatwehavewaspartofonetheseGroupofFive,people,soheknowsintimatelywhatkindofstrugglestheyhad,inordertoprovideforthosepeople.Whattheyneeded.

Andalso,intermsof...Actually,whatIdidn'tgettotalkaboutwasrepresentationofservices,'causewewanttoalsobringAIplanningintothefold,'causesomeofthesepeoplehavereallycomplexneeds,intermsofservices,andifyouthinkabouttheultimategoal,istointegratethemintoCanadiansociety,thenyoustart[decomposing01:00:07]thatintoactuallypracticaltasksthatyoucanachieve,andthenyoucanseehowthiscouldactually,youknow,itkindoflendsitselfnicelytotheplanning.

Soforcomingbacktoservices,weagainlookedatwhattheprovidersthemselves,NGO's,orthegovernment,describeashavingservices.And,youknow,eligibilityand[inaudible01:00:29]careandallthat,getsenough.'causewewerelookingatservicecomposition.

VirginiaDignum: Questions,sticktoquestions.

DanielaRosu: Sorry.

VirginiaDignum: Canyou?

DanielaRosu: Ididn'texpect[crosstalk01:00:36]soI'mhappytosee.

AudienceMember: Howdoyoudealwiththeallkindofpracticalities?Sothere'salotsupplyforvets,andnotenoughdemandforit,atthisparticularmoment,doyouhavesomestorageroom,istherea-

DanielaRosu: Yes,so-

AudienceMember: DoyouhavethiscontactwithNGO's?

DanielaRosu: Right,right.WhatIdidn'tgettotalkaboutisthefacttwo,oneofthethingswe'retryingtoachievewiththis,thisisreallythegiftthatkeepsongiving,'causewe'reextendingwaybeyondourcapacity.We'reactuallythinkingofhelpingNGO's,thatworkonsimilarproblems,becomeavirtualorganization,soseamlessintegration.Andforthisparticularissue,thatyouraise,weapproachedfurniturebanksinTorontowiththeideaofbuildingavirtualinfrastructureforthem,soexactly.Theextracapacitythatwecannotprocessrightnow,wecouldactuallydivertthewarehouses.

VirginiaDignum: Let'smovetothenextquestion.

Talk Session 3 - AI for Urban Planning Page 20 of 30

AudienceMember: Hi,Iwasjustwondering,youmentionedthatyouwerefocusedonrefugeesthatweresettledbyprivatedonors.

DanielaRosu: Yeah.

AudienceMember: AsfarasI'maware,thesettlementorganizationsalso...theygenerallysettlerefugeesbyusinggovernmentallysponsored.'causeIvolunteeredatoneinOttawa.

DanielaRosu: Yeah.

AudienceMember: AndsoIwaswonderingwhetherthiswillbealsoavailabletothem,forthemtobeableto-

DanielaRosu: Absolutely.

AudienceMember: Great,okay.

DanielaRosu: Absolutely.Actually...welleveryNGOinCanadaisreally,mostly,fundedbythegovernment.

AudienceMember: Yeah,exactlyso-

VirginiaDignum: Okay,so-

DanielaRosu: WhatIdidn'ttellyouisthatwehave60,000NGO'sinOntarioalone,whichis-

VirginiaDignum: So,thenwecontinueatthecoffeebreakwithts.[crosstalk01:02:16]

DanielaRosu: I'llbehappytotalkwithanybodywhohasquestions.Thankyou.

VirginiaDignum: Thankyou.

Speaker8: Youcaneitherholdit,orleaveitonthere.

AminGhafouri: Sure.

Speaker3: DidyoupressJ-U-N?

AminGhafouri: Yes.

Speaker10: [inaudible01:03:07]

AminGhafouri: Wegotit.Uh,thecolorsare.

Speaker10: Ithinkthat'swiththe[inaudible01:03:24]projector.

Talk Session 3 - AI for Urban Planning Page 21 of 30

Speaker8: Yeah.

VirginiaDignum: It'swrongcolor.

AminGhafouri: Oh,okay.Ihopethisdoesn'tcauseanytroubles.

Helloeveryone,thisisaworkbyme,andmycolleagues,atVanderbiltUniversity.Inasmartcities,realtimetrafficsensorsareusedfordifferentapplications,suchasrealtimecontroloftrafficsignals,orrouteplanning,however,trafficsensorsarepronetofailures,andfailurescanresultindegradedperformance.Thechallengethatwefaceistoquickly,andaccurately,detectfaults,andanomalies,insensors.

However,detectorsareimperfect,thatistherearedetectionerrors,whicharefalsepositive,andfalsenegatives.Asyoumightknow,afalsepositive...excuseme.

Speaker8: Sorry,couldyoujustusethehandheld?

AminGhafouri: Sure,sure.[inaudible01:04:30]justgonnasetmytimer,okay.

Afalsepositivemeansraisinganalarmwhenthebehaviorisnormal.Ontheotherhand,afalsenegativemeansraisingnoalarmwhenthebehaviorisanomalous.

Speaker8: Sorrycouldyoumove-

AminGhafouri: Oh,sure,sureIforgot,thatwasagoodpoint.I'mnotusedtothis.

Speaker8: Iknow.

AminGhafouri: [Showofhands01:04:52]Whileitisdesirabletoreducebothfalsepositive,andfalsenegative...Igotit.

Whileitisdesirabletoreducebothofthem,itisshownbythesecurityliterature,thatthereisatradeoffbetweenthetwo,thatis,decreasingone,increasestheother.Sotheproblemthatwearegoingtosolveisminimizinglossesduetofalsepositive,andfalsenegativeerrors,consideringsmartcityapplication,inthisworkweconsiderrouteplanning.

Sogivenasetofqueriesinarouteplanningapplication,thegoalistofindtheshortestpaths.Let'ssayGoogleMaps,orAppleMaps.Andafaultis...oh,thefiguresarereallychanged.Andafaultismodeled,asyoucansee,sothemeasuredvalueifit'sM,isdifferentfromtheactualvalueA,becauseofthefaultepsilon.Andthisresultsintheroutesthatarecomputedbeingsuboptimal,youknow,whichcausesincreaseddelays.Tobeabletodetectanomalies,we

Talk Session 3 - AI for Urban Planning Page 22 of 30

introduceadetector.Ourdetectorhastwoparts,apredictorandastatisticaltest.

Forthepredictor,wemaketheassumptionthatthenumberofsensorsthatareanomalousislow,whichistrueinpractice.Sothenwecanusethevalueofnearbysensorstodotheprediction,becausethenearbysensorsgiveusarelativelycorrectmeasurement.WeuseGaussianProcessregressionwithARDsquaredexponentialfortheprediction.So...okay...tosummarize,weusethemeasurementsofothersensorstodotheprediction,andthenwecomputethedifferencesbetweenpredictionsandmeasurementtocomputeresiduals,andthenwefeedtheseresiduals,thesedifferences,toastatisticaltest.WeuseCUSUM,whichiswellknown,andisdefinedas...itintroducesavariableSofK,thatusesthepreviousvalueofSplusthisresidualtocomputethatthecurrentvalueofS,andifit'sgreaterthanathreshold,weraiseanalarm.Weraiseadetectionalarm.

Sothiskindofshowsthewholeidea,thattheoverview,foradetectorwiththresholderrorofK,sothethresholdcanchangeintime.AnalarmisraisedifS,ifthere'sastatisticS,isgreaterthanthreshold.Soifthethresholdissmall,ourdetectorsishighlysensitive.Ifit'slarge,it'snotsensitive,whichmeansitresultsintoomanyfalsenegative.Soourgoalistofindthattherightvalueofthreshold,thesweetspotonthiscurve,thistradeoff,thatminimizestheloss,duetofalsepositivesandfalsenegative,consideringtheapplication,inthiscaserouteplanning.Sowechangethethresholddynamically,ateverytimestep,tominimalizethelosses.

Sotodoso,weformulatewhatthislossmean,sointhecaseofrouteplanning,itmeansextradelayintraveltime,soweformulatecostoffalsepositive,andcostoffalsenegative,meaningwhenthereisafalsepositive,wediscardthemeasurement,weusetheprediction.Ifwearedoingtherightthing,whatistheextradelay?That,that'swhattheseequationsmean,[inaudible01:08:23].Andoncewecharacterizethecostoffalsepositiveandfalsenegative,wedefinetheloss.Theloss,L,thatgivenathreshold,insteadofrouteplanningqueries,itjustsayshowmuchpotentialextradelayiscaused.

Sowecomputethisloss,thefirsttermisforlossduetofalsepositive,thesecondislossduetofalsenegative,andwedefineoptimalthresholdasthethresholdthatminimizestheloss,minimizesitsexpecteddelay.Sosincetradeoffcurvesarenonconvex,thisproblemischallenging,butweproposeanalgorithmthatpreformsprettywell.Ittakesasinput,thequeries,thepredictions,andthemeasurements,anditcomputestheerrorofthethresholds,anditfeedsittothealgorithm...totheapplication,anditdoesitateachtimestep.

Sowearealsoableto...usingthisframework,wearealsoabletoidentifycriticalsensors,whicharesensorsthatcausehighlossesinoursystem,and

Talk Session 3 - AI for Urban Planning Page 23 of 30

identifyingthemletsusknowthatthesearetheonesthatneedtobesecured,andbemaderobust,first.

Sowetalkedaboutthetheoreticalframework,nowlet'stalkaboutthecasestudy,whatitmeansinrealworld,inreallife.WeuseareallifedatasetsofdowntownLA,homeoftheconferenceorganizers.Weuse115,000datapoints,istrafficdatafortwoweeks,firstweekisfortraining,andsecondisusedfortesting.Wealsoconsidertwofaultmodels,oneasthreetosevenpercenterror,theothersubtractssevento13percenterror.

Thisisourdetectorsperformance.ThisissimilartoROCcurve,butinsteadofjusttruepositive,wefalsenegative.Andwecanseethatourdetectorsperformverywell.Nowournextgoalisto...soeachpointonthiscurvecorrespondstoathreshold,right?Ourgoalistofindtherightthreshold,theright...thesweetspot,thatminimizesourlosses.Soweconsiderascenario.First,weusethereallife,anopenssourcerouteplanner,thatisactualuse,youcanuseOpenTripPlanner,andweconsider1,000routeplanningqueriesinaday,andfixingthisscenario,nowwerunouralgorithmateachtimestep.

Andatonetimestepweseetheresult,andweseethatouralgorithmisabletofindtheoptimalthresholdvalues,thatminimizetheloss.Soitisexpected,youknow,20seconds,andouralgorithm,inthiscase,performedtheglobaloptimum,andwealsoidentifycriticalsensors,sothissensoriscritical,andweseethatourdynamicapproach,meaningchangingthresholdateverytimestep,performsmuchbetterthanfixingthethreshold,whichiswhatiscommonlydone.Youfixthethreshold,adecisionthreshold,andyou'redone.

Weseethatourapproachreducesthelossbyupto40%.Sothisistheconclusionthatwemade,thatourapproachminimizesthelossby40%.HowmuchtimedoIhave?

Speaker8: You'realmostdone.

AminGhafouri: Good.SoI'mgonnaspendlike10to20secondstalkingaboutongoingwork,theextensiontothiswork.Sotheapplicationisreally,ithasnothingtodowiththeframeworkitself,itcouldbeanyapplication.Asanexample,wecanapplythistotrafficsignalcontrol.Ireallylikethisfigurebytheway.Sowehaveatrafficsignal,wehaveatrafficsignalcontroller,andwehavetrafficflowmeasurements,andapredictor.Andthenthere'sthisdetector,thatitswitches.Itsayswhetherthemeasurementsinaccurate,orthepredictor.

Sothisisjustanexample,thatchoosingthisthreshold,effectswhat'sgoingonhere.Ifweextendthisevenfurther,itcouldbeanysystem,anyphysicalsystem,thatyouhaveadetector,acontroller,andtheapplication.Andthisframework,theideaistominimizethelossesduetofalsepositive,andfalsenegative,byselectingthisthresholdthataffectseverything.Thisistheongoingwork,thelossfunctionwouldlooklikethis.SohereIrelaxtheassumptiononnearby

Talk Session 3 - AI for Urban Planning Page 24 of 30

sensorsnotbeingfaulty.Ihaven'tsolvedit,IhavefourtofiveminutestodefendmyPhD,andgoodlucktomeonsolvingthisproblem.

Soinconclusion,wedesignedaneffectivedetectorusingGaussianProcess,thatperformswell.Wefindthatwepresentedanapproachforcomputingoptimalthresholdsthatminimizelosses,duetofalsepositiveandfalsenegative.Wealsocharacterizecriticalsensorswhosefailurehashighimpactontheapplication,andweimplementedanevaluator,evaluatedourapproachusingareallife,realworlddataset,andarealrouteplanningapplication.

Thankyouverymuch,andpleaseletmeknowifyouhaveanyquestions.

AudienceMember: Sohaveyoudoneanyevaluation,incomparisontoothertechniques?

AminGhafouri: Goodquestion.

AudienceMember: Howitisgoing,inrespecttowhatotherpeoplearedoing?

AminGhafouri: Right.Sotypically,intheliterature,theyconsiderstaticthreshold,meaningthattheyjustfixthethreshold,andtheyjustkeepitfixedfortheentirelifetime.Sointhiscase,thisstatic,meansthebestscenario,inafixedthresholdcase.Sothecomparisonismaderighthere.Onlefthandside,thisiswhatweobtained,andontherighthandside,youseewhatiscommonlydoneintheliterature.Andweseethatournumbersareupto40%betterthantheexistingwork.

VirginiaDignum: Okay,anyotherquestions?

Isyourworkonlyapplicableforfixedsensors,orcouldyoualsouseitwhenyouhavemovingsensors,likeinthecars,andusethecarsasthesensorsforallthecity?

AminGhafouri: Oh,okay.Soyoumeanthesensors-

VirginiaDignum: Thesensorsarenotanymorestuckinaplace,inacrossroute,buttheyaremoving.

AminGhafouri: Ofcourse,ofcourse.Aslongaswecanhaveapredictorforthatsensor,wecanimplementapredictorthatpredictsthefuture.

VirginiaDignum: Theneighborsensorsarechangingallthetime?

AminGhafouri: It'sfine.

VirginiaDignum: Okay.

AminGhafouri: Yes.

Talk Session 3 - AI for Urban Planning Page 25 of 30

VirginiaDignum: Okay.

AminGhafouri: Sothepredictor,also,itcouldbeanything,soaslongasyougivemeapredictor,wecandothat.

VirginiaDignum: Okay.

AminGhafouri: But,ofcourse,foramovingsensor,designingapredictorisachallengeitself.SoperhapsIneedtodoanotherPhDtojustworkon...

VirginiaDignum: Okay.Sothankyouand...thankyouagain.

KarenJuddS.: Doyouhavea[inaudible01:15:33].

Okay,soIguessI'mon.I'mnotsurethatI'minurbanplanning,butI'mhere.So,thankyouverymuch.Anyway,Virginiasaidthatshebeganhersessionbysayingweneedtotakeastepback,wellIwanttotakeastepeven,perhaps,furtherback.Becausewhenyou'relookingattheUnitedNations,youkindof...theassumptionis,actually,thatyou'relookingataglobalsphere,youknow,aglobalpicture,butIdowanttonotethatit'sperhapsnotquiteallthatitseems.ButIwanttojustalsoexplaintoyoualittlebit,becauseIdocomefromsureadifferentbackground,I'mnotlookingatthegranulelevelsofallofthealgorithms,andmathematics,althoughonceuponatimeIlookedatthatstuff.

Mybubble'sbeenalittlebitdifferent,sophysics,andhistory,andphilosophy,ofscienceafewyearsago.ButsincethenI'vebeenintheinternationalNGOarena,and20ofthosehavebeenworkingattheUnitedNations,Ihavemyownversionofanalgorithm,foradvocacyattheUnitedNations.I'veoftenfocusedonchangeleadership,bothatthepersonallevel,andattheorganizationallevel,becausethetwoareinterconnected,andI'dhavetosaythatoneofthethingsthatisoftenunderstood,andyetsomehowforgotten,intheprocessofengaging,andchange,isthatthere'sthepersonallevel,thenthere'stheorganizationallevelsofchange.

Andorganizationalchangeisoftenmuchmorechallenging,andoften...weareallinsomekindoforganization,butweseethroughthatfilteroften.Butweoftenforget,also,thatwhatweneedtoengageinisupgrading,oradapting,thatorganization,itsstructuresandprograms,aswellasourownattheindividuallevel,orteamlevels.Anyway,increasinglyIthinkit'spretty...noneofthisisrocketscience,butIwanttogiveittoyouasabackgroundtowherewe'regoing.

Wheremymainpointisvery,verysimple,andIthinkit'skindof...rightatthetopitsays,it'samissingconversation.IthinktheAIcommunity,thetechcommunity,needstobemuchmoredirectlyengagedwiththeglobalconversationsthattakeplaceattheUnitedNations,butthat'snotalwaysasimpleprocess.Ilikenit...Ispentanumberofyearsonthewater,andIlikenit

Talk Session 3 - AI for Urban Planning Page 26 of 30

totryingtotransfersomebodyfromasmallvessel,ontoanoceanliner,oranaircraftcarrier,notsosimple.Niceidea,itcouldbutsafeupthere,buttogetfromthetinylittlevehicleuptothemajorone,isactuallyquiteachallengingprocess.

Butanyway,we'veseenthis,everybodycanseethis,it'sanexperiencethatprettymucheveryoneisgoingthrough.Technology,anditsevolution,isallowingandenablingallkindsofnewactivities,andpossibilities,andwe'rehearingthatallthetimeineveryoneofthesesessions.Butoursocialstructures,andourlegalsystems,arekindofslowatchanging,andinasense,rightlyso.Butthefactiswe'vegotthishugeareawhereit'smorelikeagray,Idon'tknowwhetherit's50shadesofgray,butit's...mightbemore.

Butit'salargeareawherewestillareoperating,buttherearenoabsolutelystrictguidelinesforthat,sowerelyonourownstandards,ourownethics,ourownmorals,ourownorganizations.Andtosomeextentwerelyon...

Speaker8: [inaudible01:19:40]

KarenJuddS.: What'sthenumber?Five.

Speaker8: Five.

KarenJuddS.: Great.Werelyonallthosekindsofthings,forhelpingusfigureouthowtogetthroughthisarea,andthat'spartofwhatwe'rewrestlingwithhere,ishowdowedothat?Howdowerelyon?WhatdowerelyonAIfor,andwhataretheethicsofAI,andhowdoesthatgetintegratedintooursystems?AndhowdowethenevenconvincethosewhodonotknowanythingaboutAI,andareperhapssuspiciousofit,thatmaybeitisgood,ormaybeit'snotgood?Wedon'tknow.Butthere'sthisjusthugemassiveareathatwe'reneedingtodealwith.

Sooneofthethings,andI'mveryquicklygoingtogooninto...thisis,whodowetrust?Justsomeinterestinglittletidbits.Fear,motivatedabout67%there.45motivated...youknowit....Thethingis,oneofthethingsthat'sgoingon,whenwe'reinthisareaofuncertainty,iswedo,ashumanbeings,andthisiswhereIwanttogetintoakindof...mylittleviewofchangeleadership,andsomeofthedimensionsthatIthinkdoinformourconceptofthesocialgood,alittlebit.AndI'dliketojustquicklygooverthem.Andthat'sdifference,ourdrivers,andthewdomain,orscope.

Again,notrocketscience,justawaytohelpframeoursenseofwhatthesocialgoodis.Sothedriversacknowledgesthatwedohavethreebrains.Evolutionarilywehavethreebrains.Sointheprevioussection,whatwewerelookingatishowfearisdrivingalotofoursocialreality.Andthatisverysimplybecausewedohavethreebrains.Wearehumanbeingswithareptiliancomplex,wedohaveourpaleo-mammalian,andourneo-mammalian

Talk Session 3 - AI for Urban Planning Page 27 of 30

complexes.Andthat'swonderful,butwecannotignoretherealityofthoseunderlyingdrivingforcesthatare,inasense,pre-rational.

Soanyway,difference,justverysimply,islookingathistoricallyinlargerorganizations,incorporations,alotofthemanagement.Sustainingtheorganization,movingforwardincrementallyinacontrollableway,isyouknowprobably,thedifferencesarenotbigthehere.Atthemomentalotofthedifferencesthatyou'reengagedin,comparedtowhatthemajorityofsociety,areinnovative,they'reabigdifference,andthatbigdifferencecreatesvectors,sotospeak,insociety.

Sothenyou'vegotthedomainorscope,soyou'vegotindividuallevel,youknowfamiliesortribes,dependingonwhereyouare.You'vegotyou'recommunitylevel,you'vegotyournationallevel,andyou'vegotyourgloballevel.Andourorganizationsneedtobestructurallyappropriatetothelevelsatwhichweintendtohavethatimpact.Butsometimes,inthatstructuralappropriateness,it'snotexactly...wedon'talwaystellourselvesexactlythewayitis,andwe'vegot...

Speaker8: Twominutes.

KarenJuddS.: Twominutes.So,okay.Soyouhavetoreadthis.Okay,sothenbacktothis.AndIjustwanttosaythatthisisanessentiallypartofus,ithelpsusunderstand...WhenIwentbacktoAustralia,throughouttheelectionprocess,ofcourseeverybodywantedtoknowwhatwashappeninginAmericanpolitics,andfirstoutitwas,"Isitpossiblehecould?"Then,"Youmeanhe'sreallygonnarun?"To,"Youthinkhecanwin?"Therewasa...andIhadtogiveaveryshortexplanation,andmyexplanationwas,hespeaksfromhislizardbrain,tothelizardbrains.Thatdoesn'tsoundsogood,doesit?

But,anyway,thepointisthere,wedohave...thispartisustoo,eachoneofus.Soit'snotbad,it'sjustthatwhenwedon'tbalanceitout,andincludetheothercomponents,andbringthoseintowhateveritmightbethatwe'redoing.Okay,soanyway,realworldproblems,I'msogladthatwe'resolvingthem,engaging'em.Viewitasverycomplex.Ican'tgointoallofthedetailsabouthowtheUNworks,butwhenIwasmeetingwiththeundersecretarygeneralofIT,justtwoweeksagoIdiscoveredthatthereisanewinitiativethatshehasbegun,andit'scalledtheDigitalBlueHelmets,andthisiswhereitcomesintowhat'spossiblefortheAI,andthetechcommunity,asawhole.

Again,goingbacktothatbigship,oneofthewaystogetfromwhatyouaredoing,thegranulelevel,theequations,thestudies,theresearchthat'sgoingon,totheapplicationthatisusefulforourworld,isnotsuchasimpleconnectingpoint.ButtheDigitalBlueHelmetplatform,thatisnewlyestablished,allowsforthat.Thecaveatbeing,ofcourse,theUN's...anybody?DoyouknowUNstructure?Howitworks?Okay,no.

Talk Session 3 - AI for Urban Planning Page 28 of 30

AudienceMember: No.

KarenJuddS.: Okay,believeme.Yeah,likeeverybody'sgoingtobelievemenow.Butanyway.Okay,sothesecretariat,ofwhatsheispart,islikethebigsecretarytothememberstates,thegeneralassembly,andthesecuritycouncil,andtheeconomicandsocialcouncils,wherethegovernmentalrepresentativesdotheirwork.Sothesecretariatoftenhasthelargestglobalvisionoftheworld,becausethey'reattendingall193nations,they'renotjustlookingattheindividualnationalinterestsofonenation.Sotheytendtohaveaglobalview.Butthat'snottheirjobtodothat.Theirjobistoservicethememberstates.

SoinestablishingtheDigitalBlueHelmets,thisisprimarilytoservicetheneedsoftheUnitedNationsmemberstates,luckilyI'vebeensogladtoseethateverybody'sbeen...oh,puttingthatupthere.Andtheseareactuallyawaythattheplatformcanbeused,sothatshouldacrisisarise,shouldassessmentsneedtobemadeinanyoftheseareas,thereisthepotential,aswegodowntheroad,todoso,utilizingtheDigitalBlueHelmetplatform,whereyoucangetinthereanddosomeofyourgranuleworkonspecificissues.Ijustwanttoputthatouttheresothatyouareawareofit.It'sonlyjustbeginning,andsoit'snotupandrunningperse,andtherewillneedtobestepsinbetween,becauseofthissecretariatservicingthememberstatesfirst,kindofconcept.

Butthepointis,thatthereisnowaplatform.Awaytogetfromourlittleboats,upontothebigship,andifanybodywantstotalktomemore,orhasquestions,thankyouverymuch.

VirginiaDignum: It'sagreattopicforacoffeebreak,butifthereareafewshortquestionsbeforewegoforabreak?

KarenJuddS.: Yes.

Speaker3: Soisitthecase,ImeanI'vetalkedtosomepeoplethatworkattheUN,andtheyusuallytellmethat,Imean,peopletherearenotthey'reresilienttousingAI,becausetheybelievethatit'snot...it'sgoodwhereitisbeingused,anditisnotasusefulwhenyoulookatcountrylevelmatters,andmaybethey'reright.Imeanmaybewhenwetalkofmachinelearningsystems,Imeanafalsepositive,asSyedwasjustmentioning,ittakesacompletelydifferentdimensioninwhenyou'redealingwithcountrynegotiations.

KarenJuddS.: Right.

Speaker3: Whatdoesthefalsepositivemean?Andsomaytheirsystemsarenotreadyyet,andwhatdoyouthinkwouldbeagoodwaytoconvincethem,thepeopleattheUN,forthemtobeabletouseAIbasedsystems,more,andmore?BecauserightnowIthink,thepeoplewhoaretheleadershiptheyarenotfromtheagewhenAIwasbuilt.

Talk Session 3 - AI for Urban Planning Page 29 of 30

KarenJuddS.: Right.

Speaker3: Likethey'refromanearlierage,yeah.

KarenJuddS.: There'sprobablythreeanswerstothat.One,tosomeextentwedohavetowaittilltheolderofusdieoff.I'msorry,youknowhistory,andphilosophyofscience,anybody...yeah,okay.SoI'mnotjustbeingterrible.

Numbertwo,thesolutionsthatAIbringsaredefinitelymore,Ithinkatthisstage,atthegranulelevel.Sowhiletherearegovernmentallevel,orstatelevelconversations,thatneedtogoon,andyou'reprobablynotgoingtohaveAIbotsintherehavingthosedialogues,butwhenitgetsdowntowho'sgonnaimplementthemandates,inalloftheseareas,againthat'swhereyoucomein,andthat'swhere,again,whytheDigitalBlueHelmetplatformwillallowtechfolktocomein,eithersecondedbyyourorganization,'causetheUN'saplacefororganizations,ratherthanindividuals,orindividualvolunteersworkingwithagroupofpeople.

Thenwhenthere'saspecificprojectestablished,andhowthatgetsestablished,andthosethingsgetchosen,andpickedup,isstillthegrayarea.ButinmyexperienceattheUnitedNations,thereisalotthatyoucandoifyougetinthereanddoit.Anditwon'thappenifyoudon't.

VirginiaDignum: [inaudible01:29:21]

KarenJuddS.: It'sprettybasic,youknow,it'snot,asIsay,it'snotrocketscience.AndIhadathirdone,butitleft.

VirginiaDignum: So,onemorelastquestion.

AudienceMember: Justacomment,morethanaquestion,Ithink.Inresponsetowhatyou'vesaid.Justforwhatit'sworth,I'manentrepreneur,andI'vebeenrunningahealthcarefoundationthatIstartedinIndiaaboutthreeyearsago,andtheideaishowtouseinnovationtoimprovehealthcareforthepoorestpeople.WeteamedupwiththeWorldBanktoidentifythefouror500mostinterestinghealthcareinnovationsinIndia,andtheissuewashowdoyouactuallygetthemoutintothefield,inremoteareas,andurbanslums,andsoon.

You'reexactlyright,thatalotoftheinnovationsthatlookedgreatatthedevelopmentalstage,attheprototype,andthelabstage,don'tworkthatwellwhenyouactuallyget'emoutintoremote,youknow,highcrime,lowhealthindicator,typeareas,veryhighpoverty,andsoon.Butthereisaprocess,whichorganizationsliketheUN,andtheWorldBank,andothershavedeveloped,stepbystep,toseewhathappens.Inourcase,Imeanwearebeyondjustlookingatinnovations,wenowrunabout300healthclinicsinremotepartsofIndia,we'retreating10,000patientsamonth.

Talk Session 3 - AI for Urban Planning Page 30 of 30

Sowhatthatdoesisgiveusverylarge[desbeds01:30:36]forusingthemostpromisingoftheseinnovations,reallyseeinghowtheywork.Demonstratingtheirefficacy.Andonceyoudemonstratethat,thengovernmentsgetveryinterestedingettingthemin.Sothat'skindoftheprocess,andI'msuretheUNhassomethingsimilar.

KarenJuddS.: Andthere'spartofthechallenge,alsotheadditionalchallengeisthatwhenyoudohavesomethingthatworks,ontheground,evenifitisinremoteareas,andI'veworkedwithorganizationsthatworkoncounteringviolentextremism,atwhichitrequiresalotofeducationworkingwith,youknow,yourkeypartners,yourkeypeersinthereligiouscommunitiesinmadrasas,andthingslikethat.

Evenwhenitworksexceptionallywell,atthoselevels,bringingittotheattentionoftheUnitedNations,thememberstatesintheUnitedNations,andgettingthemtobuyinandscaleupthatkindofworkthatispossible,therearealwaysfundingissues,therearealwaysinterestissues.It'sacomplicatedworld,it'snotasimpleone,butitispossible,andwhenthoseconnectionscanbemade,atkeyandcentrallevels,thenyou'rereallypivotingsomemajorstuff.

Anyway,butitreallyneedsyouguys.Justsaying.

VirginiaDignum: Good.

KarenJuddS.: Justsaying.

VirginiaDignum: Thankyouverymuch.Also,thankyouforallthespeakers.