C. W. Gear, Ioannis Kevrekidis
Using existing, forward-in-time integration schemes, we demonstrate that it is possible to compute unstable, saddle-type fixed points of stiff systems of ODEs when the stable compenents are fast (i.e., rapidly damped) while the unstable components are slow. The approach has implications for the reverse (backward in time) integration of such stiff systems, and for the coarse reverse integration of microscopic/stochastic simulations.
Jaime Cisternas, C. William Gear, Simon Levin, Ioannis G. Kevrekidis
We demonstrate how direct simulation of stochastic, individual-based models can be combined with continuum numerical analysis techniques to study the dynamics of evolving diseases. % Sidestepping the necessity of obtaining explicit population-level models, the approach analyzes the (unavailable in closed form) `coarse' macroscopic equations, estimating the necessary quantities through appropriately initialized, short `bursts' of individual-based dynamic simulation. % We illustrate this approach by analyzing a stochastic and discrete model for the evolution of disease agents caused by point mutations within individual hosts. % Building up from classical SIR and SIRS models, our example uses a one-dimensional lattice for variant space, and assumes a finite number of individuals. % Macroscopic computational tasks enabled through this approach include stationary state computation, coarse projective integration, parametric continuation and stability analysis.
C. W. Gear, Ioannis G. Kevrekidis
This note reports on a scheme for interpolating the boundary conditions be- tween non-adjacent modeling regions when the model is based on Monte-Carlo computations of a collection of particles. The scheme conserves particles in a natural way, and thereby can be made to conserve other quantities.
C. William Gear, Ju Li, Ioannis G. Kevrekidis
We explore the gap-tooth method for multiscale modeling of systems represented by microscopic physics-based simulators, when coarse-grained evolution equations are not available in closed form. A biased random walk particle simulation, motivated by the viscous Burgers equation, serves as an example. We construct macro-to-micro (lifting) and micro-to-macro (restriction) operators, and drive the coarse time-evolution by particle simulations in appropriately coupled microdomains (teeth) separated by large spatial gaps. A macroscopically interpolative mechanism for communication between the teeth at the particle level is introduced. The results demonstrate the feasibility of a closure-on-demand approach to solving hydrodynamics problems.
Antonios Armaou, Ioannis G. Kevrekidis
We present a computer-assisted approach to approximating coarse optimal switching policies for systems described by microscopic/stochastic evolution rules. The coarse timestepper constitutes a bridge between the underlying kinetic Monte Carlo simulation and traditional, continuum numerical optimization techniques formulated in discrete time. The approach is illustrated through a simplified kinetic Monte Carlo simulation of NO reduction on a Pt catalyst: a switch between two coexisting stable steady states is implemented by minimal manipulation of a system parameter.
Alexei G. Makeev, Dimitrios Maroudas, Ioannis G. Kevrekidis
We implement a computer-assisted approach that, under appropriate conditions, allows the bifurcation analysis of the coarse dynamic behavior of microscopic simulators without requiring the explicit derivation of closed macroscopic equations for this behavior. The approach is inspired by the so-called time-step per based numerical bifurcation theory. We illustrate the approach through the computation of both stable and unstable coarsely invariant states for Kinetic Monte Carlo models of three simple surface reaction schemes. We quantify the linearized stability of these coarsely invariant states, perform pseudo-arclength continuation, detect coarse limit point and coarse Hopf bifurcations and construct two-parameter bifurcation diagrams.
Panagiotis Papaioannou, Ronen Talmon, Ioannis Kevrekidis, Constantinos Siettos
We address a three-tier numerical framework based on manifold learning for the forecasting of high-dimensional time series. At the first step, we embed the time series into a reduced low-dimensional space using a nonlinear manifold learning algorithm such as Locally Linear Embedding and Diffusion Maps. At the second step, we construct reduced-order regression models on the manifold, in particular Multivariate Autoregressive (MVAR) and Gaussian Process Regression (GPR) models, to forecast the embedded dynamics. At the final step, we lift the embedded time series back to the original high-dimensional space using Radial Basis Functions interpolation and Geometric Harmonics. For our illustrations, we test the forecasting performance of the proposed numerical scheme with four sets of time series: three synthetic stochastic ones resembling EEG signals produced from linear and nonlinear stochastic models with different model orders, and one real-world data set containing daily time series of 10 key foreign exchange rates (FOREX) spanning the time period 03/09/2001-29/10/2020. The forecasting performance of the proposed numerical scheme is assessed using the combinations of manifold learning, modelling and lifting approaches. We also provide a comparison with the Principal Component Analysis algorithm as well as with the naive random walk model and the MVAR and GPR models trained and implemented directly in the high-dimensional space.
Dongbin Xiu, Ioannis Kevrekidis
We present an ``equation-free'' multiscale approach to the simulation of unsteady diffusion in a random medium. The diffusivity of the medium is modeled as a random field with short correlation length, and the governing equations are cast in the form of stochastic differential equations. A detailed fine-scale computation of such a problem requires discretization and solution of a large system of equations, and can be prohibitively time-consuming. To circumvent this difficulty, we propose an equation-free approach, where the fine-scale computation is conducted only for a (small) fraction of the overall time. The evolution of a set of appropriately defined coarse-grained variables (observables) is evaluated during the fine-scale computation, and ``projective integration'' is used to accelerate the integration. The choice of these coarse variables is an important part of the approach: they are the coefficients of pointwise polynomial expansions of the random solutions. Such a choice of coarse variables allows us to reconstruct representative ensembles of fine-scale solutions with "correct" correlation structures, which is a key to algorithm efficiency. Numerical examples demonstrating accuracy and efficiency of the approach are presented.
Hector Vargas Alvarez, Dimitrios G. Patsatzis, Lucia Russo, Ioannis Kevrekidis, Constantinos Siettos
Bridging the microscopic and macroscopic modelling scales in crowd dynamics constitutes an open challenge for systematic numerical analysis, optimization, and control. Here, we propose a manifold-informed machine learning approach to learn the discrete evolution operator for the emergent/collective crowd dynamics in latent spaces from high-fidelity individual/agent-based simulations. The proposed framework is a four-stage one, \textit{explicitly conserving the mass} of the reconstructed dynamics in the high-dimensional space. In the first step, we derive continuous macroscopic fields (densities) from discrete microscopic data (pedestrians' positions) using Kernel Density Estimation. In the second step, we construct a map from the density-field space into an appropriate latent space parametrized by a few coordinates based on Proper-Orthogonal Decomposition (POD) of the corresponding density distributions. The third step involves learning reduced-order surrogate models in the latent space using machine learning techniques, particularly Long Short-Term Memory networks and Multivariate Autoregressive models. Finally, we reconstruct the crowd dynamics in the high-dimensional space with POD, demonstrating that the POD reconstruction conserves the mass. Thus, with this ``embed -> learn in latent space -> lift back to the high-dimensional space'' pipeline, we create an effective solution operator of the unavailable (at the macroscopic scale) PDE for the evolution of the density distribution. For our illustrations, we used the Social Force Model to generate data in a corridor with an obstacle, imposing periodic boundary conditions in two scenarios: (i) a unidirectional flow, and (ii) a counterflow. The numerical results demonstrate high accuracy, robustness, and generalizability, thus allowing for fast and accurate modelling/simulation of crowd dynamics from agent-based simulations.
Aaron Mahler, Tyrus Berry, Tom Stephens, Harbir Antil, Michael Merritt, Jeanie Schreiber, Ioannis Kevrekidis
This work provides a computable, direct, and mathematically rigorous approximation to the differential geometry of class manifolds for high-dimensional data, along with nonlinear projections from input space onto these class manifolds. The tools are applied to the setting of neural network image classifiers, where we generate novel, on-manifold data samples, and implement a projected gradient descent algorithm for on-manifold adversarial training. The susceptibility of neural networks (NNs) to adversarial attack highlights the brittle nature of NN decision boundaries in input space. Introducing adversarial examples during training has been shown to reduce the susceptibility of NNs to adversarial attack; however, it has also been shown to reduce the accuracy of the classifier if the examples are not valid examples for that class. Realistic "on-manifold" examples have been previously generated from class manifolds in the latent of an autoencoder. Our work explores these phenomena in a geometric and computational setting that is much closer to the raw, high-dimensional input space than can be provided by VAE or other black box dimensionality reductions. We employ conformally invariant diffusion maps (CIDM) to approximate class manifolds in diffusion coordinates, and develop the Nyström projection to project novel points onto class manifolds in this setting. On top of the manifold approximation, we leverage the spectral exterior calculus (SEC) to determine geometric quantities such as tangent vectors of the manifold. We use these tools to obtain adversarial examples that reside on a class manifold, yet fool a classifier. These misclassifications then become explainable in terms of human-understandable manipulations within the data, by expressing the on-manifold adversary in the semantic basis on the manifold.
Yu Zou, Ioannis Kevrekidis, Roger Ghanem
We present an equation-free dynamic renormalization approach to the computational study of coarse-grained, self-similar dynamic behavior in multidimensional particle systems. The approach is aimed at problems for which evolution equations for coarse-scale observables (e.g. particle density) are not explicitly available. Our illustrative example involves Brownian particles in a 2D Couette flow; marginal and conditional Inverse Cumulative Distribution Functions (ICDFs) constitute the macroscopic observables of the evolving particle distributions.
Anastasia Georgiou, Daniel Jungen, Luise Kaven, Verena Hunstig, Constantine Frangakis, Ioannis Kevrekidis, Alexander Mitsos
Bayesian Optimization (BO) with Gaussian Processes relies on optimizing an acquisition function to determine sampling. We investigate the advantages and disadvantages of using a deterministic global solver (MAiNGO) compared to conventional local and stochastic global solvers (L-BFGS-B and multi-start, respectively) for the optimization of the acquisition function. For CPU efficiency, we set a time limit for MAiNGO, taking the best point as optimal. We perform repeated numerical experiments, initially using the Muller-Brown potential as a benchmark function, utilizing the lower confidence bound acquisition function; we further validate our findings with three alternative benchmark functions. Statistical analysis reveals that when the acquisition function is more exploitative (as opposed to exploratory), BO with MAiNGO converges in fewer iterations than with the local solvers. However, when the dataset lacks diversity, or when the acquisition function is overly exploitative, BO with MAiNGO, compared to the local solvers, is more likely to converge to a local rather than a global ly near-optimal solution of the black-box function. L-BFGS-B and multi-start mitigate this risk in BO by introducing stochasticity in the selection of the next sampling point, which enhances the exploration of uncharted regions in the search space and reduces dependence on acquisition function hyperparameters. Ultimately, suboptimal optimization of poorly chosen acquisition functions may be preferable to their optimal solution. When the acquisition function is more exploratory, BO with MAiNGO, multi-start, and L-BFGS-B achieve comparable probabilities of convergence to a globally near-optimal solution (although BO with MAiNGO may require more iterations to converge under these conditions).
Evangelos Galaris, Gianluca Fabiani, Ioannis Gallos, Ioannis Kevrekidis, Constantinos Siettos
We address a three-tier data-driven approach to solve the inverse problem in complex systems modelling from spatio-temporal data produced by microscopic simulators using machine learning. In the first step, we exploit manifold learning and in particular parsimonious Diffusion Maps using leave-one-out cross-validation (LOOCV) to both identify the intrinsic dimension of the manifold where the emergent dynamics evolve and for feature selection over the parametric space. In the second step, based on the selected features, we learn the right-hand-side of the effective partial differential equations (PDEs) using two machine learning schemes, namely shallow Feedforward Neural Networks (FNNs) with two hidden layers and single-layer Random Projection Networks(RPNNs) which basis functions are constructed using an appropriate random sampling approach. Finally, based on the learned black-box PDE model, we construct the corresponding bifurcation diagram, thus exploiting the numerical bifurcation analysis toolkit. For our illustrations, we implemented the proposed method to construct the one-parameter bifurcation diagram of the 1D FitzHugh-Nagumo PDEs from data generated by $D1Q3$ Lattice Boltzmann simulations. The proposed method was quite effective in terms of numerical accuracy regarding the construction of the coarse-scale bifurcation diagram. Furthermore, the proposed RPNN scheme was $\sim$ 20 to 30 times less costly regarding the training phase than the traditional shallow FNNs, thus arising as a promising alternative to deep learning for solving the inverse problem for high-dimensional PDEs.
Hassan Arbabi, Ioannis Kevrekidis
Equations governing physico-chemical processes are usually known at microscopic spatial scales, yet one suspects that there exist equations, e.g. in the form of Partial Differential Equations (PDEs), that can explain the system evolution at much coarser, meso- or macroscopic length scales. Discovering those coarse-grained effective PDEs can lead to considerable savings in computation-intensive tasks like prediction or control. We propose a framework combining artificial neural networks with multiscale computation, in the form of equation-free numerics, for efficient discovery of such macro-scale PDEs directly from microscopic simulations. Gathering sufficient microscopic data for training neural networks can be computationally prohibitive; equation-free numerics enable a more parsimonious collection of training data by only operating in a sparse subset of the space-time domain. We also propose using a data-driven approach, based on manifold learning and unnormalized optimal transport of distributions, to identify macro-scale dependent variable(s) suitable for the data-driven discovery of said PDEs. This approach can corroborate physically motivated candidate variables, or introduce new data-driven variables, in terms of which the coarse-grained effective PDE can be formulated. We illustrate our approach by extracting coarse-grained evolution equations from particle-based simulations with a priori unknown macro-scale variable(s), while significantly reducing the requisite data collection computational effort.
Stefan Klus, Feliks Nüske, Péter Koltai, Hao Wu, Ioannis Kevrekidis, Christof Schütte, Frank Noé
In this review paper, we will present different data-driven dimension reduction techniques for dynamical systems that are based on transfer operator theory as well as methods to approximate transfer operators and their eigenvalues, eigenfunctions, and eigenmodes. The goal is to point out similarities and differences between methods developed independently by the dynamical systems, fluid dynamics, and molecular dynamics communities such as time-lagged independent component analysis (TICA), dynamic mode decomposition (DMD), and their respective generalizations. As a result, extensions and best practices developed for one particular method can be carried over to other related methods.
Erik M. Bollt, Qianxiao Li, Felix Dietrich, Ioannis Kevrekidis
Matching dynamical systems, through different forms of conjugacies and equivalences, has long been a fundamental concept, and a powerful tool, in the study and classification of nonlinear dynamic behavior (e.g. through normal forms). In this paper we will argue that the use of the Koopman operator and its spectrum is particularly well suited for this endeavor, both in theory, but also especially in view of recent data-driven algorithm developments. We believe, and document through illustrative examples, that this can nontrivially extend the use and applicability of the Koopman spectral theoretical and computational machinery beyond modeling and prediction, towards what can be considered as a systematic discovery of "Cole-Hopf-type" transformations for dynamics.
Gianmaria Viola, Alessandro Della Pia, Lucia Russo, Ioannis Kevrekidis, Constantinos Siettos
We propose a three-tier machine learning framework based on the next-generation Equation-Free algorithm for learning the spatio-temporal dynamics of mass-constrained complex systems with hidden states, whose dynamics can in principle be described by PDEs, but lack explicit models. In the first step, we employ Diffusion Maps (DMs), a nonlinear manifold learning algorithm, to extract low-dimensional latent representations of the complex spatio-temporal evolution. In the second step, we learn manifold-informed reduced-order models (ROMs) with Sparse Identification of Nonlinear Dynamics (SINDy) and standard linear Multivariate Autoregressive models (MVARs) to approximate the solution operator on the latent space. In the final step, the latent dynamics are lifted back to the original high-dimensional space by solving an (ill-posed) pre-image problem via a convex interpolation based on the k-NN algorithm. In doing so, the proposed framework reconstructs the solution operator of the unknown mass-constrained PDE, without explicitly identifying the PDE itself. For comparison purposes, we also evaluated the performance of the scheme for constructing ROMs based on Proper Orthogonal Decomposition (POD) and prove that both POD and the k-NN lifting operators preserve the mass. We illustrate the approach using two benchmark problems: (a) the Hughes model of crowd dynamics, which minimizes walking time while avoiding obstacles and high-density regions, and (b) a CFD problem involving the spatio-temporal evolution of a passive tracer advected by a periodic Navier--Stokes velocity field. We show that ROMs informed by DMs yield parsimonious models that consistently outperform the best POD-informed ROMs, yielding stable and accurate approximations of the solution operator in the latent space and, via reconstruction, in the original high-dimensional space over long time horizons.
Hassan Arbabi, Felix P. Kemeth, Tom Bertalan, Ioannis Kevrekidis
We explore the derivation of distributed parameter system evolution laws (and in particular, partial differential operators and associated partial differential equations, PDEs) from spatiotemporal data. This is, of course, a classical identification problem; our focus here is on the use of manifold learning techniques (and, in particular, variations of Diffusion Maps) in conjunction with neural network learning algorithms that allow us to attempt this task when the dependent variables, and even the independent variables of the PDE are not known a priori and must be themselves derived from the data. The similarity measure used in Diffusion Maps for dependent coarse variable detection involves distances between local particle distribution observations; for independent variable detection we use distances between local short-time dynamics. We demonstrate each approach through an illustrative established PDE example. Such variable-free, emergent space identification algorithms connect naturally with equation-free multiscale computation tools.
Michail Kavousanakis, Gianluca Fabiani, Anastasia Georgiou, Constantinos Siettos, Panagiotis Kevrekidis, Ioannis Kevrekidis
In the past, we have presented a systematic computational framework for analyzing self-similar and traveling wave dynamics in nonlinear partial differential equations (PDEs) by dynamically factoring out continuous symmetries such as translation and scaling. This is achieved through the use of time-dependent transformations -- what can be viewed as dynamic pinning conditions -- that render the symmetry-invariant solution stationary or slowly varying in rescaled coordinates. The transformation process yields a modified evolution equation coupled with algebraic constraints on the symmetry parameters, resulting in index-2 differential-algebraic equation (DAE) systems. The framework accommodates both first-kind and second-kind self-similarity, and directly recovers the self-similarity exponents or wave speeds as part of the solution, upon considering steady-state solutions in the rescaled coordinate frame. To solve the resulting high-index DAE systems, we employ Physics-Informed Neural Networks (PINNs), which naturally integrate PDE residuals and algebraic constraints into a unified loss function. This allows simultaneous inference of both the invariant solution and the transformation properties (such as the speed or the scaling rate without the need for large computational domains, mesh adaptivity, or front tracking. We demonstrate the effectiveness of the method on four canonical problems: (i) the Nagumo equation exhibiting traveling waves, (ii) the diffusion equation (1D and 2D) with first-kind self-similarity, (iii) the 2D axisymmetric porous medium equation showcasing second-kind self-similarity, and (iv) the Burgers equation, which involves both translational and scaling invariance. The results demonstrate the capability of PINNs to effectively solve these complex PDE-DAE systems, providing a promising tool for studying nonlinear wave and scaling phenomena.
Zahra Monfared, Saksham Malhotra, Sekiya Hajime, Ioannis Kevrekidis, Felix Dietrich
For continuous-time dynamical systems with reversible trajectories, the nowhere-vanishing eigenfunctions of the Koopman operator of the system form a multiplicative group. Here, we exploit this property to accelerate the systematic numerical computation of the eigenspaces of the operator. Given a small set of (so-called ``principal'') eigenfunctions that are approximated conventionally, we can obtain a much larger set by constructing polynomials of the principal eigenfunctions. This enriches the set, and thus allows us to more accurately represent application-specific observables. Often, eigenfunctions exhibit localized singularities (e.g. in simple, one-dimensional problems with multiple steady states) or extended ones (e.g. in simple, two-dimensional problems possessing a limit cycle, or a separatrix); we discuss eigenfunction matching/continuation across such singularities. By handling eigenfunction singularities and enabling their continuation, our approach supports learning consistent global representations from locally sampled data. This is particularly relevant for multistable systems and applications with sparse or fragmented measurements.