The case of theThe case of theKiller RobotKiller Robot
byby
Richard EpsteinRichard Epstein
As retold byAs retold by
Netiva CaftoriNetiva CaftoriNEIUNEIU
Oct. 2004Oct. 2004
When software engineering and When software engineering and computer ethics collidecomputer ethics collide
The newsThe news
A robot operator, Bart A robot operator, Bart Matthews, was killed Matthews, was killed by his robot, Robbie by his robot, Robbie CX30, while at work.CX30, while at work.
A programmer, Randy A programmer, Randy Samuels, who wrote Samuels, who wrote the faulty code was the faulty code was indicted for indicted for manslaughter.manslaughter.
Silicon Techchronics is Silicon Techchronics is Randy’s employer.Randy’s employer.
What really happened?What really happened?
The robot The robot malfunctioned & malfunctioned & crushed its crushed its operator to deathoperator to death
Blood all overBlood all over Decapitated headDecapitated head Num-lock key light Num-lock key light
not onnot on Numeric key pad Numeric key pad
bloodybloody
Who is at fault?Who is at fault?
The programmer The programmer who made a who made a careless mistake?careless mistake?
The designer?The designer? The robotics The robotics
division?division? The company?The company? The operator of the The operator of the
robot?robot? The robot?The robot?
Robbie CX 10 & 20 were Robbie CX 10 & 20 were experimental.experimental.
The robotics division chief, Johnson, put The robotics division chief, Johnson, put pressure on the project manager Reynolds pressure on the project manager Reynolds to finish the project by January.to finish the project by January.
Johnson hired 20 new programmers in June Johnson hired 20 new programmers in June against Reynolds’ will, by shifting against Reynolds’ will, by shifting resources. Robbie CX 30 had to succeed.resources. Robbie CX 30 had to succeed.
Johnson only knew manufacturing Johnson only knew manufacturing hardware. One cannot speed software hardware. One cannot speed software coding by adding more programmers.coding by adding more programmers.
Robbie CX 30 was a step ahead in Robbie CX 30 was a step ahead in sophistication.sophistication.
Johnson: Perfect software is an oxymoron.Johnson: Perfect software is an oxymoron.
Programmer personalityProgrammer personality Randy’s home page: Randy’s home page:
freedom for freedom for programmersprogrammers
A hacker typeA hacker type Enormous stressEnormous stress Programmer was a Programmer was a
prima donna: could prima donna: could not accept criticism or not accept criticism or his own fallibilityhis own fallibility
Helpful but arrogantHelpful but arrogant Under pressure to Under pressure to
finish project on timefinish project on time
A good programming teamA good programming team
Needs to have a whole array of Needs to have a whole array of personalities types including:personalities types including:
An interaction-oriented personAn interaction-oriented person Someone who keeps the peace and Someone who keeps the peace and
helps move things in a positive helps move things in a positive directiondirection
Task-oriented people, but not allTask-oriented people, but not all A democratic teamA democratic team Egoless programmingEgoless programming
Team dynamicsTeam dynamics
The project was controversial from the The project was controversial from the beginningbeginning
Jan Anderson, a programmer, was fired Jan Anderson, a programmer, was fired after she attacked project manager after she attacked project manager Reynolds on his decision to use the Reynolds on his decision to use the waterfall methodology versus the waterfall methodology versus the prototype methodology.prototype methodology.
Reynolds was replacing a dead project Reynolds was replacing a dead project manager as a cost saving measure instead manager as a cost saving measure instead of hiring someone from outside the of hiring someone from outside the company. He never worked with robotics company. He never worked with robotics before.before.
More explanationsMore explanations
The earlier Robbie models processed The earlier Robbie models processed transactions. Robbie CX30 was to transactions. Robbie CX30 was to interact with its operator, therefore interact with its operator, therefore the interface cannot be designed as the interface cannot be designed as an afterthought. Thus, a prototyping an afterthought. Thus, a prototyping model is preferable where the users model is preferable where the users can use a prototype robot while in can use a prototype robot while in the design stage.the design stage.
Software development life-cycleSoftware development life-cycleWater fall methodologyWater fall methodology
RequirementsRequirements AnalysisAnalysis DesignDesign CodingCoding TestingTesting ImplementationImplementation MaintenanceMaintenance
No standard No standard practices practices
Quality controlQuality controlor quality assuranceor quality assurance
Dynamic testingDynamic testing Static testingStatic testing Unit testing: Black and white box Unit testing: Black and white box
testingtesting Integration testingIntegration testing
Prof Silber attested that test results Prof Silber attested that test results were inconsistent with actual killer were inconsistent with actual killer code test resultscode test results
When is the software good When is the software good enough?enough?
Errors can happen in any stageErrors can happen in any stage Design stage: 2 alternatives:Design stage: 2 alternatives:Waterfall & prototypingWaterfall & prototyping TestingTesting User interfaceUser interface The complexity of the task of The complexity of the task of
building real-world softwarebuilding real-world software Enormous stressEnormous stress
The company-Sili-TechThe company-Sili-Tech Two environments: Two environments:
Worker-friendly: Chip Worker-friendly: Chip Creek facility. Creek facility. Prevent repetitive Prevent repetitive strain injuries.strain injuries.
Special training for Special training for employees.employees.
Well designed Well designed workstations.workstations.
Frequent breaks.Frequent breaks.
Worker-unfriendly: Worker-unfriendly: Silicon Valley plant.Silicon Valley plant.
Workers criticized Workers criticized for novelties.for novelties.
No exercises or No exercises or training training encouraged.encouraged.
RSI frequentRSI frequent More compensation More compensation
claimsclaims
Law suitsLaw suits
The wife of the The wife of the murdered operator.murdered operator.
The indicted The indicted programmer programmer himself.himself.
More possible More possible indictments.indictments.
Why was Randy Samuels indicted?Why was Randy Samuels indicted?
Sili-Tech promised to deliver robots that would Sili-Tech promised to deliver robots that would cause no bodily harm.cause no bodily harm.
So Randy was not legally responsible for the So Randy was not legally responsible for the death of the robot operator.death of the robot operator.
Waterson, the president of Sili-Tech has Waterson, the president of Sili-Tech has contributed large sums to the re-election of judge contributed large sums to the re-election of judge McMurdock, who indicted Randy.McMurdock, who indicted Randy.
Functional requirements specify the behavior of Functional requirements specify the behavior of the robot under exceptional conditions. Operator the robot under exceptional conditions. Operator intervention may be needed.intervention may be needed.
Exceptional conditions were not mentioned in Exceptional conditions were not mentioned in training of operators.training of operators.
The charactersThe characters
Jane
Cindy
Prof SilberProf Gritty
Bart
E-mail was not secure at Sili-TechE-mail was not secure at Sili-TechIt was found that:It was found that:
Cindy Yardley faked the test results Cindy Yardley faked the test results of the Killer codeof the Killer code
Johnson told Cindy that the robot was Johnson told Cindy that the robot was safe and all will lose their jobs if the safe and all will lose their jobs if the robot were not shipped on timerobot were not shipped on time
Randy stole some of his software, but Randy stole some of his software, but admitted he had bugs in his Killer admitted he had bugs in his Killer codecode
Ethical issuesEthical issues
Email privacyEmail privacy Professional ethics: Professional ethics:
delivering bug-free delivering bug-free softwaresoftware
Plagiarism-Plagiarism-intellectual intellectual propertyproperty
Honor Honor confidentialityconfidentiality
ACM code of ethicsACM code of ethics
Five ethical tests:Five ethical tests:by Kallman & Grilloby Kallman & Grillo
The mom test: would you tell your The mom test: would you tell your mother?mother?
The TV test: would you tell your story The TV test: would you tell your story on national TV?on national TV?
The smell test: does it smell bad?The smell test: does it smell bad? The other person’s shoes test: would The other person’s shoes test: would
you like it if done to you?you like it if done to you? The market test: would your action The market test: would your action
make a good sale pitch?make a good sale pitch?