Recall the classic utilitarian morality puzzle (via Wikipedia):
There is a runaway trolley barrelling down the railway tracks. Ahead, on the tracks, there are five people tied up and unable to move. The trolley is headed straight for them. You are standing some distance off in the train yard, next to a lever. If you pull this lever, the trolley will switch to a different set of tracks. Unfortunately, you notice that there is one person on the side track. You do not have the ability to operate the lever in a way that would cause the trolley to derail without loss of life (for example, holding the lever in an intermediate position so that the trolley goes between the two sets of tracks, or pulling the lever after the front wheels pass the switch, but before the rear wheels do). You have two options: (1) Do nothing, and the trolley kills the five people on the main track. (2) Pull the lever, diverting the trolley onto the side track where it will kill one person. Which is the correct choice?
How should we program robots to answer this question? Specifically, what about self-driving cars? Should they be programmed to injure or kill their driver in order to save many others? The question is raised at minute three of this short video on robots and ethics. The whole video is worth your time.