60J28 Applications of continuous-time Markov processes on discrete state spaces
Refine
Year of publication
- 2013 (2)
Document Type
- ZIB-Report (2)
Language
- English (2)
Has Fulltext
- yes (2)
Is part of the Bibliography
- no (2)
Keywords
- Markov state models (1)
- bellmann equation (1)
- diagnostic frequency (1)
- hidden state (1)
- information costs (1)
- long timescales (1)
- optimal control (1)
- optimal therapeutic policies (1)
- rare events (1)
- resource-poor (1)
Institute
Rare but important transition events between long lived states are a key feature of many molecular systems. In many cases the computation of rare event statistics by direct molecular dynamics (MD) simulations is infeasible even on the most powerful computers because of the immensely long simulation timescales needed. Recently a technique for spatial discretization of the molecular state space designed to help overcome such problems, so-called Markov State Models (MSMs), has attracted a lot of attention. We review the theoretical background and algorithmic realization of MSMs and illustrate their use by some numerical examples. Furthermore we introduce a novel approach to using MSMs for the efficient solution of optimal control problems that appear in applications where one desires to optimize molecular properties by means of external controls.
Markov Decision Processes (MDP) or Partially Observable MDPs (POMDP) are
used for modelling situations in which the evolution of a process is partly random and
partly controllable. These MDP theories allow for computing the optimal control policy
for processes that can continuously or frequently be observed, even if only partially.
However, they cannot be applied if state observation is very costly and therefore rare
(in time). We present a novel MDP theory for rare, costly observations and derive the
corresponding Bellman equation. In the new theory, state information can be derived
for a particular cost after certain, rather long time intervals. The resulting information
costs enter into the total cost and thus into the optimization criterion. This approach
applies to many real world problems, particularly in the medical context, where the
medical condition is examined rather rarely because examination costs are high. At the
same time, the approach allows for efficient numerical realization. We demonstrate the
usefulness of the novel theory by determining, from the national economic perspective,
optimal therapeutic policies for the treatment of the human immunodefficiency virus
(HIV) in resource-rich and resource-poor settings. Based on the developed theory and
models, we discover that available drugs may not be utilized efficiently in resource-poor
settings due to exorbitant diagnostic costs.