Expected Utility Maximisers
Expected Utility Maximisers
An expected utility maximiser is a theoretical agent who considers its actions, computes
their consequences and then rates them according to a utility function. Next, it
performs the action which it thinks is likely to produce the largest utility - and then
iterates this process.
For an example, consider a computer program that plays
the game of go. Such a program
considers its possible moves, calculates their possible consequences, and then performs
the move that it thinks gives it the best chance of winning.
Expected utility maximisation is common framework used in the context of
modelling intelligent agents and constructing synthetic intelligences.
A utility function can neatly encapsulate many concepts from economics - such as risk
aversion and temporal discounting and marginal utility.
If the utility function is expressed as in a Turing-complete lanugage,
the framework represents a remarkably-general model of intelligent
agents - one which is capable of representing any pattern of behavioural
responses that can itself be represented computationally.
The utility function encodes all the agent's preferences - including:
discounting refers to how an agent values utility now, compared to
utility later. Is ten dollars now better than twenty dollars tomorrow?
An agent's temporal discounting preferences specify such things.
refers to the reluctance of an agent to accept a bargain with an
uncertain payoff rather than another bargain with a more certain, but
Powerful expected utility maximisers can be expected to
systems. Self-improvement is one of the fundamental strategies
expected utility maximisers are likely to use to help
them attain their goals.
After a certain point, such systems tend to naturally come to share
various traits with living organisms - they will resist death,
maintain themselves, absorb resources, grow and/or reproduce,
eliminate the competition - and so on. These natural
tendencies are not necessarily benign.
Self-improving systems may wish to change their levels of temporal
discounting and risk aversion - depending on their
capabilities. One obvious way of doing that is to make these factors
depend on your percieved self-confidence.
The complex field of utility engineering deals with how to
construct utiliy functions which are useful, and don't have too many
undesirable side effects.
Pragmatic and ideal goals
Self-improving systems will often make use of the concepts of
ideal goals and pragmatic goals.
An ideal goal represents what a system actually wants,
pragmatic goals are a cut-down versions of this - which are
faster, easier or cheaper to calculate.
For example, the synthetic intelligence,
Deep Blue had a complex utility function with over 8,000
parts, which contained relative piece values, the worth of central
control vs castling, heuristics about pawn promotion - and so on.
However, its real aim was to increase IBM's stock price by
winning games of chess.
A self-improving system will normally only have one ideal
utility function - but may derive various pragmatic utility
functions from this - depending on the resource constraints it
It is not usually a good idea to encode strategies for dealing with
resource constraints into the ideal goals of a system - since
resource availability may change as time passes. Strategies such as
outcome pruning and temporal discounting normally
belong in pragmatic utility functions.
Proximate and ultimate goals
Similarly, expected utility maximisers typically have one ultimate
goal, but may pursue various proximate goals in service
Often long-term projects can be broken down into numerous short-term
ones. For example having a child can be decomposed into learning a
trade, getting a job, buying a house,
finding a mate - and so on. These short-term targets are known as
Proximate goals can sometimes be used as
pragmatic goals - but these are separate concepts.
Ultimate goals and ideal goals, however, are
different terms for essentially the same concept.
The Nature of Self-Improving Artificial Intelligence - a paper by Steve Omohundro
The Basic AI drives - a paper by Steve Omohundro
Expected utility hypothesis - Wikipedia
Rational choice theory - Wikipedia