TY - GEN A1 - Winkelmann, Stefanie A1 - Schütte, Christof A1 - von Kleist, Max T1 - Markov Control Processes with Rare State Observation: Theory and Application to Treatment Scheduling in HIV-1 N2 - Markov Decision Processes (MDP) or Partially Observable MDPs (POMDP) are used for modelling situations in which the evolution of a process is partly random and partly controllable. These MDP theories allow for computing the optimal control policy for processes that can continuously or frequently be observed, even if only partially. However, they cannot be applied if state observation is very costly and therefore rare (in time). We present a novel MDP theory for rare, costly observations and derive the corresponding Bellman equation. In the new theory, state information can be derived for a particular cost after certain, rather long time intervals. The resulting information costs enter into the total cost and thus into the optimization criterion. This approach applies to many real world problems, particularly in the medical context, where the medical condition is examined rather rarely because examination costs are high. At the same time, the approach allows for efficient numerical realization. We demonstrate the usefulness of the novel theory by determining, from the national economic perspective, optimal therapeutic policies for the treatment of the human immunodefficiency virus (HIV) in resource-rich and resource-poor settings. Based on the developed theory and models, we discover that available drugs may not be utilized efficiently in resource-poor settings due to exorbitant diagnostic costs. T3 - ZIB-Report - 13-34 KW - information costs KW - hidden state KW - bellmann equation KW - optimal therapeutic policies KW - diagnostic frequency KW - resource-poor KW - resource-rich Y1 - 2013 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-41955 SN - 1438-0064 ER - TY - GEN A1 - Winkelmann, Stefanie T1 - Markov Control with Rare State Observation: Average Optimality N2 - This paper investigates the criterion of long-term average costs for a Markov decision process (MDP) which is not permanently observable. Each observation of the process produces a fixed amount of \textit{information costs} which enter the considered performance criterion and preclude from arbitrarily frequent state testing. Choosing the \textit{rare} observation times is part of the control procedure. In contrast to the theory of partially observable Markov decision processes, we consider an arbitrary continuous-time Markov process on a finite state space without further restrictions on the dynamics or the type of interaction. Based on the original Markov control theory, we redefine the control model and the average cost criterion for the setting of information costs. We analyze the constant of average costs for the case of ergodic dynamics and present an optimality equation which characterizes the optimal choice of control actions and observation times. For this purpose, we construct an equivalent freely observable MDP and translate the well-known results from the original theory to the new setting. T3 - ZIB-Report - 16-59 KW - Markov decision process KW - partial observability KW - average optimality KW - information costs Y1 - 2016 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-60981 SN - 1438-0064 ER - TY - GEN A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - The spatiotemporal master equation: approximation of reaction-diffusion dynamics via Markov state modeling N2 - Accurate modeling and numerical simulation of reaction kinetics is a topic of steady interest. We consider the spatiotemporal chemical master equation (ST-CME) as a model for stochastic reaction-diffusion systems that exhibit properties of metastability. The space of motion is decomposed into metastable compartments and diffusive motion is approximated by jumps between these compartments. Treating these jumps as first-order reactions, simulation of the resulting stochastic system is possible by the Gillespie method. We present the theory of Markov state models (MSM) as a theoretical foundation of this intuitive approach. By means of Markov state modeling, both the number and shape of compartments and the transition rates between them can be determined. We consider the ST-CME for two reaction-diffusion systems and compare it to more detailed models. Moreover, a rigorous formal justification of the ST-CME by Galerkin projection methods is presented. T3 - ZIB-Report - 16-60 KW - reaction-diffusion KW - stochastic chemical kinetics KW - chemical master equation Y1 - 2016 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-60999 SN - 1438-0064 ER - TY - JOUR A1 - Winkelmann, Stefanie T1 - Markov Control with Rare State Observation: Average Optimality JF - Markov Processes and Related Fields N2 - This paper investigates the criterion of long-term average costs for a Markov decision process (MDP) which is not permanently observable. Each observation of the process produces a fixed amount of information costs which enter the considered performance criterion and preclude from arbitrarily frequent state testing. Choosing the rare observation times is part of the control procedure. In contrast to the theory of partially observable Markov decision processes, we consider an arbitrary continuous-time Markov process on a finite state space without further restrictions on the dynamics or the type of interaction. Based on the original Markov control theory, we redefine the control model and the average cost criterion for the setting of information costs. We analyze the constant of average costs for the case of ergodic dynamics and present an optimality equation which characterizes the optimal choice of control actions and observation times. For this purpose, we construct an equivalent freely observable MDP and translate the well-known results from the original theory to the new setting. Y1 - 2017 VL - 23 SP - 1 EP - 34 ER - TY - JOUR A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - The Spatiotemporal Master Equation: Approximation of Reaction-Diffusion Dynamics via Markov State Modeling JF - Journal of Chemical Physics N2 - Accurate modeling and numerical simulation of reaction kinetics is a topic of steady interest.We consider the spatiotemporal chemical master equation (ST-CME) as a model for stochastic reaction-diffusion systems that exhibit properties of metastability. The space of motion is decomposed into metastable compartments and diffusive motion is approximated by jumps between these compartments. Treating these jumps as first-order reactions, simulation of the resulting stochastic system is possible by the Gillespie method. We present the theory of Markov state models (MSM) as a theoretical foundation of this intuitive approach. By means of Markov state modeling, both the number and shape of compartments and the transition rates between them can be determined. We consider the ST-CME for two reaction-diffusion systems and compare it to more detailed models. Moreover, a rigorous formal justification of the ST-CME by Galerkin projection methods is presented. Y1 - 2016 U6 - https://doi.org/10.1063/1.4971163 VL - 145 IS - 21 ER - TY - JOUR A1 - Duwal, Sulav A1 - Winkelmann, Stefanie A1 - Schütte, Christof A1 - von Kleist, Max T1 - Optimal Treatment Strategies in the Context of 'Treatment for Prevention' against HIV/1 in Resource-Poor Settings JF - PloS Computational Biology N2 - An estimated 2.7 million new HIV-1 infections occurred in 2010. `Treatment-for-prevention’ may strongly prevent HIV-1 transmission. The basic idea is that immediate treatment initiation rapidly decreases virus burden, which reduces the number of transmittable viruses and thereby the probability of infection. However, HIV inevitably develops drug resistance, which leads to virus rebound and nullifies the effect of `treatment-for-prevention’ for the time it remains unrecognized. While timely conducted treatment changes may avert periods of viral rebound, necessary treatment options and diagnostics may be lacking in resource-constrained settings. Within this work, we provide a mathematical platform for comparing different treatment paradigms that can be applied to many medical phenomena. We use this platform to optimize two distinct approaches for the treatment of HIV-1: (i) a diagnostic-guided treatment strategy, based on infrequent and patient-specific diagnostic schedules and (ii) a pro-active strategy that allows treatment adaptation prior to diagnostic ascertainment. Both strategies are compared to current clinical protocols (standard of care and the HPTN052 protocol) in terms of patient health, economic means and reduction in HIV-1 onward transmission exemplarily for South Africa. All therapeutic strategies are assessed using a coarse-grained stochastic model of within-host HIV dynamics and pseudo-codes for solving the respective optimal control problems are provided. Our mathematical model suggests that both optimal strategies (i)-(ii) perform better than the current clinical protocols and no treatment in terms of economic means, life prolongation and reduction of HIV-transmission. The optimal diagnostic-guided strategy suggests rare diagnostics and performs similar to the optimal pro-active strategy. Our results suggest that ‘treatment-for-prevention’ may be further improved using either of the two analyzed treatment paradigms. Y1 - 2015 U6 - https://doi.org/10.1371/journal.pcbi.1004200 VL - 11 IS - 4 ER - TY - JOUR A1 - Schütte, Christof A1 - Winkelmann, Stefanie A1 - Hartmann, Carsten T1 - Optimal control of molecular dynamics using Markov state models JF - Math. Program. (Series B) Y1 - 2012 UR - http://publications.imp.fu-berlin.de/1107/ U6 - https://doi.org/10.1007/s10107-012-0547-6 VL - 134 IS - 1 SP - 259 EP - 282 ER - TY - JOUR A1 - Djurdjevac Conrad, Natasa A1 - Helfmann, Luzie A1 - Zonker, Johannes A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - Human mobility and innovation spreading in ancient times: a stochastic agent-based simulation approach JF - EPJ Data Science N2 - Human mobility always had a great influence on the spreading of cultural, social and technological ideas. Developing realistic models that allow for a better understanding, prediction and control of such coupled processes has gained a lot of attention in recent years. However, the modeling of spreading processes that happened in ancient times faces the additional challenge that available knowledge and data is often limited and sparse. In this paper, we present a new agent-based model for the spreading of innovations in the ancient world that is governed by human movements. Our model considers the diffusion of innovations on a spatial network that is changing in time, as the agents are changing their positions. Additionally, we propose a novel stochastic simulation approach to produce spatio-temporal realizations of the spreading process that are instructive for studying its dynamical properties and exploring how different influences affect its speed and spatial evolution. Y1 - 2018 U6 - https://doi.org/10.1140/epjds/s13688-018-0153-9 VL - 7 IS - 1 PB - EPJ Data Science ET - EPJ Data Science ER - TY - BOOK A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - Stochastic Dynamics in Computational Biology T3 - Frontiers in Applied Dynamical Systems: Reviews and Tutorials Y1 - 2020 SN - 978-3-030-62386-9 U6 - https://doi.org/10.1007/978-3-030-62387-6 VL - 8 PB - Springer International Publishing ER - TY - JOUR A1 - Niemann, Jan-Hendrik A1 - Winkelmann, Stefanie A1 - Wolf, Sarah A1 - Schütte, Christof T1 - Agent-based modeling: Population limits and large timescales JF - Chaos: An Interdisciplinary Journal of Nonlinear Science N2 - Modeling, simulation and analysis of interacting agent systems is a broad field of research, with existing approaches reaching from informal descriptions of interaction dynamics to more formal, mathematical models. In this paper, we study agent-based models (ABMs) given as continuous-time stochastic processes and their pathwise approximation by ordinary and stochastic differential equations (ODEs and SDEs, respectively) for medium to large populations. By means of an appropriately adapted transfer operator approach we study the behavior of the ABM process on long time scales. We show that, under certain conditions, the transfer operator approach allows to bridge the gap between the pathwise results for large populations on finite timescales, i.e., the SDE limit model, and approaches built to study dynamical behavior on long time scales like large deviation theory. The latter provides a rigorous analysis of rare events including the associated asymptotic rates on timescales that scale exponentially with the population size. We demonstrate that it is possible to reveal metastable structures and timescales of rare events of the ABM process by finite-length trajectories of the SDE process for large enough populations. This approach has the potential to drastically reduce computational effort for the analysis of ABMs. Y1 - 2021 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-77309 SN - 1438-0064 VL - 31 IS - 3 ER -