TY - GEN A1 - Winkelmann, Stefanie T1 - Markov Control with Rare State Observation: Average Optimality N2 - This paper investigates the criterion of long-term average costs for a Markov decision process (MDP) which is not permanently observable. Each observation of the process produces a fixed amount of \textit{information costs} which enter the considered performance criterion and preclude from arbitrarily frequent state testing. Choosing the \textit{rare} observation times is part of the control procedure. In contrast to the theory of partially observable Markov decision processes, we consider an arbitrary continuous-time Markov process on a finite state space without further restrictions on the dynamics or the type of interaction. Based on the original Markov control theory, we redefine the control model and the average cost criterion for the setting of information costs. We analyze the constant of average costs for the case of ergodic dynamics and present an optimality equation which characterizes the optimal choice of control actions and observation times. For this purpose, we construct an equivalent freely observable MDP and translate the well-known results from the original theory to the new setting. T3 - ZIB-Report - 16-59 KW - Markov decision process KW - partial observability KW - average optimality KW - information costs Y1 - 2016 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-60981 SN - 1438-0064 ER - TY - GEN A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - The spatiotemporal master equation: approximation of reaction-diffusion dynamics via Markov state modeling N2 - Accurate modeling and numerical simulation of reaction kinetics is a topic of steady interest. We consider the spatiotemporal chemical master equation (ST-CME) as a model for stochastic reaction-diffusion systems that exhibit properties of metastability. The space of motion is decomposed into metastable compartments and diffusive motion is approximated by jumps between these compartments. Treating these jumps as first-order reactions, simulation of the resulting stochastic system is possible by the Gillespie method. We present the theory of Markov state models (MSM) as a theoretical foundation of this intuitive approach. By means of Markov state modeling, both the number and shape of compartments and the transition rates between them can be determined. We consider the ST-CME for two reaction-diffusion systems and compare it to more detailed models. Moreover, a rigorous formal justification of the ST-CME by Galerkin projection methods is presented. T3 - ZIB-Report - 16-60 KW - reaction-diffusion KW - stochastic chemical kinetics KW - chemical master equation Y1 - 2016 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-60999 SN - 1438-0064 ER - TY - JOUR A1 - Winkelmann, Stefanie T1 - Markov Control with Rare State Observation: Average Optimality JF - Markov Processes and Related Fields N2 - This paper investigates the criterion of long-term average costs for a Markov decision process (MDP) which is not permanently observable. Each observation of the process produces a fixed amount of information costs which enter the considered performance criterion and preclude from arbitrarily frequent state testing. Choosing the rare observation times is part of the control procedure. In contrast to the theory of partially observable Markov decision processes, we consider an arbitrary continuous-time Markov process on a finite state space without further restrictions on the dynamics or the type of interaction. Based on the original Markov control theory, we redefine the control model and the average cost criterion for the setting of information costs. We analyze the constant of average costs for the case of ergodic dynamics and present an optimality equation which characterizes the optimal choice of control actions and observation times. For this purpose, we construct an equivalent freely observable MDP and translate the well-known results from the original theory to the new setting. Y1 - 2017 VL - 23 SP - 1 EP - 34 ER - TY - JOUR A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - The Spatiotemporal Master Equation: Approximation of Reaction-Diffusion Dynamics via Markov State Modeling JF - Journal of Chemical Physics N2 - Accurate modeling and numerical simulation of reaction kinetics is a topic of steady interest.We consider the spatiotemporal chemical master equation (ST-CME) as a model for stochastic reaction-diffusion systems that exhibit properties of metastability. The space of motion is decomposed into metastable compartments and diffusive motion is approximated by jumps between these compartments. Treating these jumps as first-order reactions, simulation of the resulting stochastic system is possible by the Gillespie method. We present the theory of Markov state models (MSM) as a theoretical foundation of this intuitive approach. By means of Markov state modeling, both the number and shape of compartments and the transition rates between them can be determined. We consider the ST-CME for two reaction-diffusion systems and compare it to more detailed models. Moreover, a rigorous formal justification of the ST-CME by Galerkin projection methods is presented. Y1 - 2016 U6 - https://doi.org/10.1063/1.4971163 VL - 145 IS - 21 ER - TY - BOOK A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - Stochastic Dynamics in Computational Biology T3 - Frontiers in Applied Dynamical Systems: Reviews and Tutorials Y1 - 2020 SN - 978-3-030-62386-9 U6 - https://doi.org/10.1007/978-3-030-62387-6 VL - 8 PB - Springer International Publishing ER - TY - JOUR A1 - Niemann, Jan-Hendrik A1 - Winkelmann, Stefanie A1 - Wolf, Sarah A1 - Schütte, Christof T1 - Agent-based modeling: Population limits and large timescales JF - Chaos: An Interdisciplinary Journal of Nonlinear Science N2 - Modeling, simulation and analysis of interacting agent systems is a broad field of research, with existing approaches reaching from informal descriptions of interaction dynamics to more formal, mathematical models. In this paper, we study agent-based models (ABMs) given as continuous-time stochastic processes and their pathwise approximation by ordinary and stochastic differential equations (ODEs and SDEs, respectively) for medium to large populations. By means of an appropriately adapted transfer operator approach we study the behavior of the ABM process on long time scales. We show that, under certain conditions, the transfer operator approach allows to bridge the gap between the pathwise results for large populations on finite timescales, i.e., the SDE limit model, and approaches built to study dynamical behavior on long time scales like large deviation theory. The latter provides a rigorous analysis of rare events including the associated asymptotic rates on timescales that scale exponentially with the population size. We demonstrate that it is possible to reveal metastable structures and timescales of rare events of the ABM process by finite-length trajectories of the SDE process for large enough populations. This approach has the potential to drastically reduce computational effort for the analysis of ABMs. Y1 - 2021 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-77309 SN - 1438-0064 VL - 31 IS - 3 ER - TY - THES A1 - Kostré, Margarita T1 - Hybrid Models and Simulations of Reaction-Diffusion Processes N2 - In this thesis,we are interested in multiscale models for particle-based reaction diffusion (PBRD) simulations,where we focus on coupling particle-based simulations to macroscopic chemical reservoirs. These reservoirs are given by a mean concentration of chemical species that can vary in time and space. We model these reservoirs as reaction-diffusion partial differential equations (PDEs). The goal of this work is to achieve a mathematically consistent coupling between the PBRD simulations and the reaction-diffusion PDEs. Y1 - 2019 ER - TY - JOUR A1 - Dibak, Manuel A1 - del Razo, Mauricio J. A1 - de Sancho, David A1 - Schütte, Christof A1 - Noé, Frank T1 - MSM/RD: Coupling Markov state models of molecular kinetics with reaction-diffusion simulations JF - Journal of Chemical Physics N2 - Molecular dynamics (MD) simulations can model the interactions between macromolecules with high spatiotemporal resolution but at a high computational cost. By combining high-throughput MD with Markov state models (MSMs), it is now possible to obtain long time-scale behavior of small to intermediate biomolecules and complexes. To model the interactions of many molecules at large length scales, particle-based reaction-diffusion (RD) simulations are more suitable but lack molecular detail. Thus, coupling MSMs and RD simulations (MSM/RD) would be highly desirable, as they could efficiently produce simulations at large time and length scales, while still conserving the characteristic features of the interactions observed at atomic detail. While such a coupling seems straightforward, fundamental questions are still open: Which definition of MSM states is suitable? Which protocol to merge and split RD particles in an association/dissociation reaction will conserve the correct bimolecular kinetics and thermodynamics? In this paper, we make the first step toward MSM/RD by laying out a general theory of coupling and proposing a first implementation for association/dissociation of a protein with a small ligand (A + B ⇌ C). Applications on a toy model and CO diffusion into the heme cavity of myoglobin are reported. Y1 - 2018 U6 - https://doi.org/10.1063/1.5020294 VL - 148 IS - 21 ER - TY - GEN A1 - Gelß, Patrick A1 - Matera, Sebastian A1 - Schütte, Christof T1 - Solving the master equation without kinetic Monte Carlo: tensor train approximations for a CO oxidation model N2 - In multiscale models of heterogeneous catalysis, one crucial point is the solution of a Markovian master equation describing the stochastic reaction kinetics. This usually is too high-dimensional to be solved with standard numerical techniques and one has to rely on sampling approaches based on the kinetic Monte Carlo method. In this study we break the curse of dimensionality for the direct solution of the Markovian master equation by exploiting the Tensor Train Format for this purpose. The performance of the approach is demonstrated on a first principles based, reduced model for the CO oxidation on the RuO_2(110) surface. We investigate the complexity for increasing system size and for various reaction conditions. The advantage over the stochastic simulation approach is illustrated by a problem with increased stiffness. T3 - ZIB-Report - 15-41 KW - heterogeneous catalysis KW - master equation KW - kinetic Monte Carlo KW - tensor decompositions KW - tensor train format KW - alternating linear scheme Y1 - 2015 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-55743 SN - 1438-0064 ER - TY - JOUR A1 - Klus, Stefan A1 - Schütte, Christof T1 - Towards tensor-based methods for the numerical approximation of the Perron-Frobenius and Koopman operator JF - Journal of Computational Dynamics N2 - The global behavior of dynamical systems can be studied by analyzing the eigenvalues and corresponding eigenfunctions of linear operators associated with the system. Two important operators which are frequently used to gain insight into the system's behavior are the Perron-Frobenius operator and the Koopman operator. Due to the curse of dimensionality, computing the eigenfunctions of high-dimensional systems is in general infeasible. We will propose a tensor-based reformulation of two numerical methods for computing finite-dimensional approximations of the aforementioned infinite-dimensional operators, namely Ulam's method and Extended Dynamic Mode Decomposition (EDMD). The aim of the tensor formulation is to approximate the eigenfunctions by low-rank tensors, potentially resulting in a significant reduction of the time and memory required to solve the resulting eigenvalue problems, provided that such a low-rank tensor decomposition exists. Typically, not all variables of a high-dimensional dynamical system contribute equally to the system's behavior, often the dynamics can be decomposed into slow and fast processes, which is also reflected in the eigenfunctions. Thus, the weak coupling between different variables might be approximated by low-rank tensor cores. We will illustrate the efficiency of the tensor-based formulation of Ulam's method and EDMD using simple stochastic differential equations. Y1 - 2016 U6 - https://doi.org/10.3934/jcd.2016007 VL - 3 IS - 2 SP - 139 EP - 161 ER - TY - JOUR A1 - Rüdrich, S. A1 - Sarich, Marco A1 - Schütte, Christof T1 - Utilizing hitting times for finding metastable sets in non-reversible Markov chains JF - Journal of Comp. Dynamics Y1 - 2017 ER - TY - GEN A1 - Winkelmann, Stefanie A1 - Schütte, Christof T1 - Hybrid Models for Chemical Reaction Networks: Multiscale Theory and Application to Gene Regulatory Systems N2 - Well-mixed stochastic chemical kinetics are properly modelled by the chemical master equation (CME) and associated Markov jump processes in molecule number space. If the reactants are present in large amounts, however, corresponding simulations of the stochastic dynamics become computationally expensive and model reductions are demanded. The classical model reduction approach uniformly rescales the overall dynamics to obtain deterministic systems characterized by ordinary differential equations, the well-known mass action reaction rate equations. For systems with multiple scales there exist hybrid approaches that keep parts of the system discrete while another part is approximated either using Langevin dynamics or deterministically. This paper aims at giving a coherent overview of the different hybrid approaches, focusing on their basic concepts and the relation between them. We derive a novel general description of such hybrid models that allows to express various forms by one type of equation. We also check in how far the approaches apply to model extensions of the CME for dynamics which do not comply with the central well-mixed condition and require some spatial resolution. A simple but meaningful gene expression system with negative self-regulation is analysed to illustrate the different approximation qualities of some of the hybrid approaches discussed. T3 - ZIB-Report - 17-29 Y1 - 2017 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-64264 ER - TY - GEN A1 - Bittracher, Andreas A1 - Koltai, Péter A1 - Klus, Stefan A1 - Banisch, Ralf A1 - Dellnitz, Michael A1 - Schütte, Christof T1 - Transition manifolds of complex metastable systems: Theory and data-driven computation of effective dynamics N2 - We consider complex dynamical systems showing metastable behavior but no local separation of fast and slow time scales. The article raises the question of whether such systems exhibit a low-dimensional manifold supporting its effective dynamics. For answering this question, we aim at finding nonlinear coordinates, called reaction coordinates, such that the projection of the dynamics onto these coordinates preserves the dominant time scales of the dynamics. We show that, based on a specific reducibility property, the existence of good low-dimensional reaction coordinates preserving the dominant time scales is guaranteed. Based on this theoretical framework, we develop and test a novel numerical approach for computing good reaction coordinates. The proposed algorithmic approach is fully local and thus not prone to the curse of dimension with respect to the state space of the dynamics. Hence, it is a promising method for data-based model reduction of complex dynamical systems such as molecular dynamics. T3 - ZIB-Report - 17-22 KW - metastability KW - slow dynamics KW - effective dynamics KW - transition manifold KW - embedding KW - transfer operator KW - reaction coordinate Y1 - 2017 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-63822 SN - 1438-0064 ER -