## 60J27 Continuous-time Markov processes on discrete state spaces

### Refine

#### Document Type

- ZIB-Report (2)
- Habilitation (1)

#### Language

- English (3)

#### Has Fulltext

- yes (3)

#### Is part of the Bibliography

- no (3)

#### Keywords

- Hamiltonian system (1)
- Markov chain (1)
- Markov operator (1)
- Perron root (1)
- Perron- (1)
- almost invariant set (1)
- bellmann equation (1)
- biochemical conformation (1)
- chemical master equation (1)
- diagnostic frequency (1)

#### Institute

- Numerical Mathematics (2)
- ZIB Allgemein (1)

Accurate modeling and numerical simulation of reaction kinetics is a topic of steady interest. We consider the spatiotemporal chemical master equation (ST-CME) as a model for stochastic reaction-diffusion systems that exhibit properties of metastability. The space of motion is decomposed into metastable compartments and diffusive motion is approximated by jumps between these compartments. Treating these jumps as first-order reactions, simulation of the resulting stochastic system is possible by the Gillespie method. We present the theory of Markov state models (MSM) as a theoretical foundation of this intuitive approach. By means of Markov state modeling, both the number and shape of compartments and the transition rates between them can be determined. We consider the ST-CME for two reaction-diffusion systems and compare it to more detailed models. Moreover, a rigorous formal justification of the ST-CME by Galerkin projection methods is presented.

Markov Decision Processes (MDP) or Partially Observable MDPs (POMDP) are
used for modelling situations in which the evolution of a process is partly random and
partly controllable. These MDP theories allow for computing the optimal control policy
for processes that can continuously or frequently be observed, even if only partially.
However, they cannot be applied if state observation is very costly and therefore rare
(in time). We present a novel MDP theory for rare, costly observations and derive the
corresponding Bellman equation. In the new theory, state information can be derived
for a particular cost after certain, rather long time intervals. The resulting information
costs enter into the total cost and thus into the optimization criterion. This approach
applies to many real world problems, particularly in the medical context, where the
medical condition is examined rather rarely because examination costs are high. At the
same time, the approach allows for efficient numerical realization. We demonstrate the
usefulness of the novel theory by determining, from the national economic perspective,
optimal therapeutic policies for the treatment of the human immunodefficiency virus
(HIV) in resource-rich and resource-poor settings. Based on the developed theory and
models, we discover that available drugs may not be utilized efficiently in resource-poor
settings due to exorbitant diagnostic costs.

The function of many important biomolecules comes from their dynamic properties and their ability to switch between different {\em conformations}. In a conformation, the large scale geometric structure of the molecule is understood to be conserved, whereas on smaller scales the system may well rotate, oscillate or fluctuate. In a recent article [J. Comp. Phys., 151,1 (1999)], the present author and coworkers demonstrated that (a) conformations can be understood as almost invariant sets of some Markov chain being defined via the Hamiltonian system governing the molecular dynamics and that (b) these sets can efficiently be computed via eigenvectors of the corresponding Markov operator. The persent manuscript reviews the mathematical modelling steps behind the novel concept, includes a rigorous analytical justification of this approach and especially of the numerical details of the algorithm, and illustrates its performance when applied to realistic molecular systems.