Learning Dynamical Systems using Recurrent Networks

Abstract:

We investigate mechanisms for learning a dynamical system (DS) using a recurrent neural network (RNN). For affine neural dynamical systems (A-NDSs), which constitute an important class in DSs produced by RNNs, we have constructed a unique parameteric representation.

Introduction

Units of an RNN are divided into visible units and hidden units. An RNN with no hidden units always produces a DS on the visible state space [1]. However, an RNN with hidden units, which has a greater potential for representing a DS than an RNN with no hidden units, does not produce a DS on the visible state space unless a mapping from the visible state space to the hidden state space is successfully specified (cf. Figure 1). Therefore, it is necessary to investigate what DS is produced by an RNN to approximate a target DS.

Affine Neural Dynamical Systems

We have proposed a neural dynamial system (NDS) as a DS produced by an RNN with hidden units, and constructed A-NDSs as the concrete examples [2]. We can prove that any DS on a Euclidean space is finitely approximated by some A-NDS with any precision [3]. Therefore, we consider adopting an A-NDS as a DS that an RNN with hidden units produces on the visible state space to approximate a target DS.

A Unique Representation of Affine Neural Dynamical Systems

An n-dimensional A-NDS is parametrically represented by a suitable pair of an RNN with n visible units and r hidden units, and an affine mapping from the n-dimensinal space to the r-dimensional space. However, this parametric representation has redundancy. Concerning the learning of a DS using an A-NDS, we have constructed a unique parametric representation of an A-NDS with the aim of building efficient learning algorithms [3].

  

Figure 1: Examples where DSs are not produced

Contact: Masahiro Kimura, Email: kimura@cslab.kecl.ntt.co.jp

Bibliography

1

Kimura, M. and Nakano, R.: Learning dynamical systems from trajectories by continuous time recurrent neural networks, Proc. of IEEE International Conference on Neural Networks (ICNN'95), pp. 2992-2997 (1995).

2

Kimura, M. and Nakano, R.: Learning dynamical systems produced by recurrent neural networks, Proc. of the 6th International Conference on Artificial Neural Networks (ICANN'96), pp. 133-138 (1996).

3

Kimura, M. and Nakano, R.: Unique representations of dynamical systems produced by recurrent neural networks, Proc. of the 7th International Conference on Artificial Neural Networks (ICANN'97), pp. 403-408 (1997).