Metric entropy limits on recurrent neural network learning of linear dynamical systems
(Submitted to Applied and Computational Harmonic Analysis, Apr 2021)
Keywords: Recurrent neural networks, linear dynamical systems, metric entropy, universal approximation, system identification
Abstract: One of the most influential results in neural network theory is the universal approximation theorem [1, 2, 3] which states that continuous functions can be approximated to within arbitrary accuracy by single-hidden-layer feedforward neural networks. The purpose of this paper is to establish a result in this spirit for the approximation of general discrete-time linear dynamical systems—including time-varying systems—by recurrent neural networks (RNNs). For the subclass of linear time-invariant (LTI) systems, we devise a quantitative version of this statement. Speci cally, measuring the complexity of the considered class of LTI systems through metric entropy according to [4], we show that RNNs can optimally learn—or identify in system-theory parlance—stable LTI systems. For LTI systems whose input-output relation is characterized through a difference equation, this means that RNNs can learn the difference equation from input-output traces in a metric-entropy optimal manner.
https://arxiv.org/abs/2105.02556
Knowledge transfer across cell lines using Hybrid Gaussian Process models with entity embedding vectors
(Submitted to Biotechnology and Bioengineering)
Keywords: Gaussian Process Regression, Embedding Vector, Transversal Data Analysis, Hybrid semi-parametric modeling, bioprocess development, cell culture
Abstract: To date, a large number of experiments are performed to develop a biochemical process. The generated data is used only once, to take decisions for development. Could we exploit data of already developed processes to make predictions for a novel process, we could significantly reduce the number of experiments needed. Processes for different products exhibit differences in behaviour, typically only a subset behave similar. Therefore, effective learning on multiple product spanning process data requires a sensible representation of the product identity. We propose to represent the product identity (a categorical feature) by embedding vectors that serve as input to a Gaussian Process regression model. We demonstrate how the embedding vectors can be learned from process data and show that they capture an interpretable notion of product similarity. The improvement in performance is compared to traditional one-hot encoding on a simulated cross product learning task. All in all, the proposed method could render possible significant reductions in wet-lab experiments.
https://arxiv.org/abs/2011.13863
Border Ownership Representation and Polychronisation in a SpikingNeural Network Model (Bachelor Thesis)
OFTNAI – University of Oxford (Oct 2017 – Mar 2018)
Abstract: In this computer simulation study we investigated a hierarchical neural network model of the ventral visual pathway in the primate brain. It consists of four layers of spiking leaky integrate and fire neurons that are connected by bottom-up, top-down and lateral synapses. These synapses are modified through Spike Time Dependent Plasticity while visual stimuli of simple shapes are presented to the network. Similarly to Eguchi and Stringer (2016), we observed the emergence of cells that seem to represent border owner- ship, that is they seemed to encode the presence of a border in their receptive field and which side of the shape it is located on. However, detailed investigations later indicated that these cells instead encode the presence of some contour in rough regions of the retina and that they can therefore not be considered border ownership cells. This might also be the case for some of the cells reported in Eguchi and Stringer (2016). Further, we investi- gated the emergence of reliable spatio-temporal patterns of spiking activity (polychonous groups). We initially observed large numbers of informative polychronous groups consist- ing of two neurons, which is in line with the findings of Eguchi et al. (2018). Additionally, we found that the high values of information in polychronous groups are a side effect of information carried by firing rates of cells. Thus, the precise times of spikes do not seem to carry information in the investigated network. All in all, it might be that the problem with measuring polychronous group information has also occurred in Eguchi et al. (2018), which raises doubts about the reported high information carried by polychronous groups.