Background
Expected Utility MaximisersAn expected utility maximiser is a theoretical agent who considers its actions, computes their consequences and then rates them according to a utility function. Next, it performs the action which it thinks is likely to produce the largest utility - and then iterates this process.For an example, consider a computer program that plays the game of go. Such a program considers its possible moves, calculates their possible consequences, and then performs the move that it thinks gives it the best chance of winning. Expected utility maximisation is common framework used in the context of modelling intelligent agents and constructing synthetic intelligences. A utility function can neatly encapsulate many concepts from economics - such as risk aversion and temporal discounting and marginal utility. If the utility function is expressed as in a Turing-complete lanugage, the framework represents a remarkably-general model of intelligent agents - one which is capable of representing any pattern of behavioural responses that can itself be represented computationally. The utility function encodes all the agent's preferences - including:
Self-Improving SystemsPowerful expected utility maximisers can be expected to become self-improving systems. Self-improvement is one of the fundamental strategies expected utility maximisers are likely to use to help them attain their goals.After a certain point, such systems tend to naturally come to share various traits with living organisms - they will resist death, maintain themselves, absorb resources, grow and/or reproduce, eliminate the competition - and so on. These natural tendencies are not necessarily benign. Self-improving systems may wish to change their levels of temporal discounting and risk aversion - depending on their capabilities. One obvious way of doing that is to make these factors depend on your percieved self-confidence. The complex field of utility engineering deals with how to construct utiliy functions which are useful, and don't have too many undesirable side effects.
Pragmatic and ideal goalsSelf-improving systems will often make use of the concepts of ideal goals and pragmatic goals.An ideal goal represents what a system actually wants, pragmatic goals are a cut-down versions of this - which are faster, easier or cheaper to calculate. For example, the synthetic intelligence, Deep Blue had a complex utility function with over 8,000 parts, which contained relative piece values, the worth of central control vs castling, heuristics about pawn promotion - and so on. However, its real aim was to increase IBM's stock price by winning games of chess. A self-improving system will normally only have one ideal utility function - but may derive various pragmatic utility functions from this - depending on the resource constraints it faces. It is not usually a good idea to encode strategies for dealing with resource constraints into the ideal goals of a system - since resource availability may change as time passes. Strategies such as outcome pruning and temporal discounting normally belong in pragmatic utility functions. Proximate and ultimate goalsSimilarly, expected utility maximisers typically have one ultimate goal, but may pursue various proximate goals in service of this.Often long-term projects can be broken down into numerous short-term ones. For example having a child can be decomposed into learning a trade, getting a job, buying a house, finding a mate - and so on. These short-term targets are known as proximate goals. Proximate goals can sometimes be used as pragmatic goals - but these are separate concepts. Ultimate goals and ideal goals, however, are different terms for essentially the same concept.
LinksSelf-Aware SystemsSteve Omohundro The Nature of Self-Improving Artificial Intelligence - a paper by Steve Omohundro The Basic AI drives - a paper by Steve Omohundro Eliezer Yudkowsky Expected utility hypothesis - Wikipedia Rational choice theory - Wikipedia |
