The value function of an optimization problem gives the value attained by the objective function at a solution, while only depending on the parameters of the problem.[1][2] In a controlled dynamical system, the value function represents the optimal payoff of the system over the interval [t, t1] when started at the time-t state variable x(t)=x.[3] If the objective function represents some cost that is to be minimized, the value function can be interpreted as the cost to finish the optimal program, and is thus referred to as "cost-to-go function."[4][5] In an economic context, where the objective function usually represents utility, the value function is conceptually equivalent to the indirect utility function.[6][7]
In a problem of optimal control, the value function is defined as the supremum of the objective function taken over the set of admissible controls. Given , a typical optimal control problem is to
subject to
with initial state variable .[8] The objective function is to be maximized over all admissible controls , where is a Lebesgue measurable function from to some prescribed arbitrary set in . The value function is then defined as
with , where is the "scrap value". If the optimal pair of control and state trajectories is , then . The function that gives the optimal control based on the current state is called a feedback control policy,[4] or simply a policy function.[9]
Bellman's principle of optimality roughly states that any optimal policy at time , taking the current state as "new" initial condition must be optimal for the remaining problem. If the value function happens to be continuously differentiable,[10] this gives rise to an important partial differential equation known as Hamilton–Jacobi–Bellman equation,
where the maximand on the right-hand side can also be re-written as the Hamiltonian, , as
with playing the role of the costate variables.[11] Given this definition, we further have , and after differentiating both sides of the HJB equation with respect to ,
which after replacing the appropriate terms recovers the costate equation
where is Newton notation for the derivative with respect to time.[12]
The value function is the unique viscosity solution to the Hamilton–Jacobi–Bellman equation.[13] In an online closed-loop approximate optimal control, the value function is also a Lyapunov function that establishes global asymptotic stability of the closed-loop system.[14]
^Fleming, Wendell H.; Rishel, Raymond W. (1975). Deterministic and Stochastic Optimal Control. New York: Springer. pp. 81–83. ISBN 0-387-90155-8.
^Caputo, Michael R. (2005). Foundations of Dynamic Economic Analysis : Optimal Control Theory and Applications. New York: Cambridge University Press. p. 185. ISBN 0-521-60368-4.
^Weber, Thomas A. (2011). Optimal Control Theory : with Applications in Economics. Cambridge: The MIT Press. p. 82. ISBN 978-0-262-01573-8.
^ abBertsekas, Dimitri P.; Tsitsiklis, John N. (1996). Neuro-Dynamic Programming. Belmont: Athena Scientific. p. 2. ISBN 1-886529-10-8.
^"EE365: Dynamic Programming" (PDF).
^Mas-Colell, Andreu; Whinston, Michael D.; Green, Jerry R. (1995). Microeconomic Theory. New York: Oxford University Press. p. 964. ISBN 0-19-507340-1.
^Corbae, Dean; Stinchcombe, Maxwell B.; Zeman, Juraj (2009). An Introduction to Mathematical Analysis for Economic Theory and Econometrics. Princeton University Press. p. 145. ISBN 978-0-691-11867-3.
^Kamien, Morton I.; Schwartz, Nancy L. (1991). Dynamic Optimization : The Calculus of Variations and Optimal Control in Economics and Management (2nd ed.). Amsterdam: North-Holland. p. 259. ISBN 0-444-01609-0.
^Ljungqvist, Lars; Sargent, Thomas J. (2018). Recursive Macroeconomic Theory (Fourth ed.). Cambridge: MIT Press. p. 106. ISBN 978-0-262-03866-9.
^Benveniste and Scheinkman established sufficient conditions for the differentiability of the value function, which in turn allows an application of the envelope theorem, see Benveniste, L. M.; Scheinkman, J. A. (1979). "On the Differentiability of the Value Function in Dynamic Models of Economics". Econometrica. 47 (3): 727–732. doi:10.2307/1910417. JSTOR 1910417. Also see Seierstad, Atle (1982). "Differentiability Properties of the Optimal Value Function in Control Theory". Journal of Economic Dynamics and Control. 4: 303–310. doi:10.1016/0165-1889(82)90019-7.
^Kirk, Donald E. (1970). Optimal Control Theory. Englewood Cliffs, NJ: Prentice-Hall. p. 88. ISBN 0-13-638098-0.
^Zhou, X. Y. (1990). "Maximum Principle, Dynamic Programming, and their Connection in Deterministic Control". Journal of Optimization Theory and Applications. 65 (2): 363–373. doi:10.1007/BF01102352. S2CID 122333807.
^Theorem 10.1 in Bressan, Alberto (2019). "Viscosity Solutions of Hamilton-Jacobi Equations and Optimal Control Problems" (PDF). Lecture Notes.
^Kamalapurkar, Rushikesh; Walters, Patrick; Rosenfeld, Joel; Dixon, Warren (2018). "Optimal Control and Lyapunov Stability". Reinforcement Learning for Optimal Feedback Control: A Lyapunov-Based Approach. Berlin: Springer. pp. 26–27. ISBN 978-3-319-78383-3.
The valuefunction of an optimization problem gives the value attained by the objective function at a solution, while only depending on the parameters...
Functionvalue may refer to: In mathematics, the value of a function when applied to an argument. In computer science, a closure. This disambiguation page...
mathematics, a multivalued function (also known as a multiple-valuedfunction) is a function that has two or more values in its range for at least one...
the absolute valuefunction is idempotent (meaning that the absolute value of any absolute value is itself). The absolute valuefunction of a real number...
A hash function is any function that can be used to map data of arbitrary size to fixed-size values, though there are some hash functions that support...
theorem, as stated, is false if a differentiable function is complex-valued instead of real-valued. For example, define f ( x ) = e x i {\displaystyle...
a continuous function is a function such that a small variation of the argument induces a small variation of the value of the function. This implies...
In mathematics, a Boolean function is a function whose arguments and result assume values from a two-element set (usually {true, false}, {0,1} or {-1...
complex-valuedfunction f on an arbitrary set X (is isomorphic to, and therefore, in that sense, it) can be considered as an ordered pair of two real-valued functions:...
The gamma function is an important special function in mathematics. Its particular values can be expressed in closed form for integer and half-integer...
value v ( S ) {\displaystyle v(S)} , which is not already accounted for by its subsets. The Shapley values are given in terms of the synergy function...
Unless otherwise specified, the term generally refers to the positive-valuedfunction of a real variable, although it can be extended to the complex numbers...
of change of a function's output with respect to its input. The derivative of a function of a single variable at a chosen input value, when it exists...
In mathematics, the sign function or signum function (from signum, Latin for "sign") is a function that has the value −1, +1 or 0 according to whether...
the quantile function outputs the value of a random variable such that its probability is less than or equal to an input probability value. Intuitively...
two main approaches for achieving this are valuefunction estimation and direct policy search. Valuefunction approaches attempt to find a policy that maximizes...
In mathematics, the Weierstrass function is an example of a real-valuedfunction that is continuous everywhere but differentiable nowhere. It is an example...
same value of utility. Individual utility and social utility can be construed as the value of a utility function and a social welfare function respectively...
decision theory, a loss function or cost function (sometimes also called an error function) is a function that maps an event or values of one or more variables...
definite integral of the function over the real numbers to equal 1 (whereas the same integral of the unnormalized sinc function has a value of π). As a further...
partial sums would grow indefinitely large. The zeta functionvalues listed below include functionvalues at the negative even numbers (s = −2, −4, etc.),...
the mean-value theorem hold for the symmetric derivative; some similar but weaker statements have been proved. For the absolute valuefunction f ( x )...
function, or the unit step function, usually denoted by H or θ (but sometimes u, 1 or 𝟙), is a step function named after Oliver Heaviside, the value...