A population can be described as being in an evolutionarily stable state when that population's "genetic composition is restored by selection after a disturbance, provided the disturbance is not too large" (Maynard Smith, 1982).^[1] This population as a whole can be either monomorphic or polymorphic.^[1] This is now referred to as convergent stability. ^[2]

While related to the concept of an evolutionarily stable strategy (ESS), evolutionarily stable states are not identical and the two terms cannot be used interchangeably.

An ESS is a strategy that, if adopted by all individuals within a population, cannot be invaded by alternative or mutant strategies.^[1] This strategy becomes fixed in the population because alternatives provide no fitness benefit that would be selected for. In comparison, an evolutionarily stable state describes a population that returns as a whole to its previous composition even after being disturbed.^[1] In short: the ESS refers to the strategy itself, uninterrupted and supported through natural selection, while the evolutionarily stable state refers more broadly to a population-wide balance of one or more strategies that may be subjected to temporary change.^[3]

The term ESS was first used by John Maynard Smith in an essay from the 1972 book On Evolution.^[4] Maynard Smith developed the ESS drawing in part from game theory and Hamilton's work on the evolution of sex ratio.^[5][6] The ESS was later expanded upon in his book Evolution and the Theory of Games in 1982, which also discussed the evolutionarily stable state.^[1]

There has been variation in how the term is used and exploration of under what conditions an evolutionarily stable state might exist. In 1984, Benhard Thomas compared "discrete" models in which all individuals use only one strategy to "continuous" models in which individuals employ mixed strategies.^[3] While Maynard Smith had originally defined an ESS as being a single "uninvadable strategy," Thomas generalized this to include a set of multiple strategies employed by individuals.^[1][3] In other words, a collection of simultaneously present strategies could be considered uninvadable as a group. Thomas noted that evolutionary stability can exist in either model, allowing for an evolutionarily stable state to exist even when multiple strategies are used within the population.^[3]

The strategy employed by individuals (or ESS) is thought to depend on fitness: the better the strategy is at supporting fitness, the more likely the strategy is to be used.^[5] When it comes to an evolutionarily stable state, all of the strategies used within the population must have equal fitness.^[7] While the equilibrium may be disturbed by external factors, the population is considered to be in an evolutionarily stable state if it returns to the equilibrium state after the disturbance.^[7]

One of the base mathematical models for identifying an evolutionarily stable state was outlined by Taylor & Jonker in 1978.^[7] Their base equilibrium model for ES states stipulates that ^[3][7]

A state p is called an ESS (evolutionary stable state) if for every state q ≠ p, if we let p̅ =(1-ε)p + εq (the perturbed state), then F(q|p) < F(p|p̅) for sufficiently small ε>0.

In greater detail, the Taylor & Jonker model can be understood this way ^[7]

In a game of individuals in competition with each other there are (N) possible strategies available. Thus each individual is using one of these (N) strategies. If we denote each strategy as I we let S_i be the proportion of individuals who are currently using strategy I. Then S=(S_1 -> S_n) is a probability vector (i.e. S ≥ 0 and S_1 + S_2... + S_n = 1) this is called the state vector of the population. Using this the function F(i|s) can be made, F(i|s) refers to the fitness of I in state S. The state vector of the population (S) is not static. The idea behind it is that the more fit a strategy at the moment the more likely it is to be employed in the future, thus the state vector (S) will change. Using game theory we can look how (S) changes over time and try to figure out in what state it has reached an equilibrium. Let K be the set of all probability vectors  of length N, this is the state space of the population. Thus element P in K represents a possible strategy mix. A state P in K is called an equilibrium state if F(i|p) is equal for all pure strategies i for which P_i > 0, That is, supp(p) = {i :p,≠0}. If Q is in K: F(q|p) + (ΣQ_1 x F(i|p). We can see F(q|p) as the expected fitness of an individual using mixed strategy Q against the population in state P. If P is an equilibrium state and the supp(q) is contained in supp(p) then F(q|p) = F(q|p).(supp(p) are the I's for which P_i > 0). Thus a state p is called an ESS (evolutionary stable state) if for every state Q ≠ P, if we let p̅=(1-ε)p + εq (the perturbed state), then F(q|p) < F(p|p̅) for sufficiently small ε>0 ^[7]

In summary, a state P is evolutionarily stable whenever a small change from P to state p̅ the expected fitness in the perturbed state is less than the expected fitness of the remaining population.

It has been suggested by Ross Cressman that criteria for evolutionary stability include strong stability, as it would describe evolution of both frequency and density (whereas Maynard Smith's model focused on frequency).^[8] Cressman further demonstrated that in habitat selection games modeling only a single species, the ideal free distribution (IFD) is itself an evolutionarily stable state containing mixed strategies.^[9]

Evolutionary game theory as a whole provides a theoretical framework examining interactions of organisms in a system where individuals have repeated interactions within a population that persists on an evolutionarily relevant timescale.^[10] This framework can be used to better understand the evolution of interaction strategies and stable states, though many different specific models have been used under this framework. The Nash Equilibrium (NE) and folk theorem are closely related to the evolutionarily stable state. There are various potential refinements proposed to account for different theory games and behavioral models.^[11]

For the purpose of predicting evolutionary outcomes, the replicator equation is also a frequently utilized tool. ^[12][13] Evolutionarily stable states are often taken as solutions to the replicator equation, here in linear payoff form:

${\displaystyle {\dot {x_{i}}}=x_{i}\left(\left(Ax\right)_{i}-x^{T}Ax\right),}$

The state ${\displaystyle {\hat {x}}}$ is said to be evolutionarily stable if for all ${\displaystyle x\neq {\hat {x}}}$ in some neighborhood of ${\displaystyle {\hat {x}}}$.

${\displaystyle x^{T}Ax<{\hat {x}}^{T}Ax}$