Expected Utility Maximisers


Expected Utility Maximisers

An expected utility maximiser is a theoretical agent who considers its actions, computes their consequences and then rates them according to a utility function. Next, it performs the action which it thinks is likely to produce the largest utility - and then iterates this process.

For an example, consider a computer program that plays the game of go. Such a program considers its possible moves, calculates their possible consequences, and then performs the move that it thinks gives it the best chance of winning.

Expected utility maximisation is common framework used in the context of modelling intelligent agents and constructing synthetic intelligences.

A utility function can neatly encapsulate many concepts from economics - such as risk aversion and temporal discounting and marginal utility.

If the utility function is expressed as in a Turing-complete lanugage, the framework represents a remarkably-general model of intelligent agents - one which is capable of representing any pattern of behavioural responses that can itself be represented computationally.

The utility function encodes all the agent's preferences - including:

  • Temporal discounting

    Temporal discounting refers to how an agent values utility now, compared to utility later. Is ten dollars now better than twenty dollars tomorrow? An agent's temporal discounting preferences specify such things.

  • Risk aversion

    Risk aversion refers to the reluctance of an agent to accept a bargain with an uncertain payoff rather than another bargain with a more certain, but lower, payoff.

Self-Improving Systems

Powerful expected utility maximisers can be expected to become self-improving systems. Self-improvement is one of the fundamental strategies expected utility maximisers are likely to use to help them attain their goals.

After a certain point, such systems tend to naturally come to share various traits with living organisms - they will resist death, maintain themselves, absorb resources, grow and/or reproduce, eliminate the competition - and so on. These natural tendencies are not necessarily benign.

Self-improving systems may wish to change their levels of temporal discounting and risk aversion - depending on their capabilities. One obvious way of doing that is to make these factors depend on your percieved self-confidence.

The complex field of utility engineering deals with how to construct utiliy functions which are useful, and don't have too many undesirable side effects.

Pragmatic and ideal goals

Self-improving systems will often make use of the concepts of ideal goals and pragmatic goals.

An ideal goal represents what a system actually wants, pragmatic goals are a cut-down versions of this - which are faster, easier or cheaper to calculate.

For example, the synthetic intelligence, Deep Blue had a complex utility function with over 8,000 parts, which contained relative piece values, the worth of central control vs castling, heuristics about pawn promotion - and so on. However, its real aim was to increase IBM's stock price by winning games of chess.

A self-improving system will normally only have one ideal utility function - but may derive various pragmatic utility functions from this - depending on the resource constraints it faces.

It is not usually a good idea to encode strategies for dealing with resource constraints into the ideal goals of a system - since resource availability may change as time passes. Strategies such as outcome pruning and temporal discounting normally belong in pragmatic utility functions.

Proximate and ultimate goals

Similarly, expected utility maximisers typically have one ultimate goal, but may pursue various proximate goals in service of this.

Often long-term projects can be broken down into numerous short-term ones. For example having a child can be decomposed into learning a trade, getting a job, buying a house, finding a mate - and so on. These short-term targets are known as proximate goals.

Proximate goals can sometimes be used as pragmatic goals - but these are separate concepts.

Ultimate goals and ideal goals, however, are different terms for essentially the same concept.


Self-Aware Systems
Steve Omohundro
The Nature of Self-Improving Artificial Intelligence - a paper by Steve Omohundro
The Basic AI drives - a paper by Steve Omohundro
Eliezer Yudkowsky
Expected utility hypothesis - Wikipedia
Rational choice theory - Wikipedia
Tim Tyler | Contact