The mathematical description of biological evolution by means of natural selection is a multi-faceted subject. Disciplines as widely divergent as population genetics, quantitative genetics, evolutionary ecology, game theory and life-history theory provide various methods to tackle evolutionary questions. In recent decades, evolutionary game theory and the analysis of invasibility, which predict the fate of rare mutant types that are numerically dominated by resident types, have gained solid ground for analysing frequency- and density-dependent selection.
Evolution can be seen as the ongoing ousting of resident types by successful novel mutant strategies. Mutants can invade if their fitness is positive, that is, if they can increase in numbers when introduced in an environment that is set by the resident type. Hence, fitness by definition depends on the prevailing environment, which in its turn is determined by the phenotypic composition of the resident population. Recent developments such as the emerging theory of adaptive dynamics integrate classification schemes of evolutionarily stable strategies in dynamical systems theory. By restricting the attention to clonal genetics, and separating the time-scales of convergence to a population-dynamical attractor and of the generation of new successful mutations, the environmental condition can be expressed in terms of the resident genotype. In this way, invasion functions can be derived that depend on the type of the mutant and on the composition of the resident population. These invasion functions serve, together with mutation, as the generators of the dynamical systems describing the evolutionary process.
This thesis deals with three problems in the framework of evolutionary game dynamics and complex population dynamics, with invasibility analysis as a unifying principle. It consists of three parts, and each part in turn consists of two chapters. The first part deals with evolutionary game dynamics, where the model Battle of the Sexes is taken as an example. It shows that the evolutionary predictions of traditional game-dynamic models, in which the costs and benefits of a strategy are summarized by simple entries from a payoff matrix, may be quite different from models that explicitly consider the mechanisms controlling mating in a structured population. This part stresses the importance of incorporating life-history information in game-dynamic model building. In part two, the semelparous life-history of Pacific salmon is used to illustrate an analysis of invasion in periodically fluctuating populations. Contrary to the steady-state situation, successful invasion can inevitably lead to an attractor switch, causing the expulsion of the former invader. This intriguing example may prove to be the exception to the rule, but in any case underlines the need for precision when defining invasibility of strategies in the presence of multiple attractors. The third part argues that the strategies brought forth by evolution should be defined as strategies that are both non-invasible and attainable. It raises and answers the question whether (and, if so, how) they can also be found by maximizing simple optimization criteria. This part makes yet another plea for more mechanistic model building, emphasizing the significance of the precise way in which density dependence limits population growth.