Files in this item



application/pdfSHIVAM-THESIS-2021.pdf (22MB)Restricted Access
(no description provided)PDF


Title:Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system
Author(s):Shivam, Keshav
Advisor(s):Gazzola, Mattia; Chowdhary, Girish
Department / Program:Computer Science
Discipline:Computer Science
Degree Granting Institution:University of Illinois at Urbana-Champaign
Subject(s):Soft robotics
reservoir computing
reinforcement learning
neuromorphic computing
spiking neural networks
Abstract:Modern approaches in machine learning and artificial intelligence are dominated by deep learning. Although inspired by the brain, these network architectures are not biologically plausible. In contrast, the reservoir computing paradigm has a single sparsely and recurrently connected hidden layer, with the linear readout layer being the only learned parameter. We apply reservoir computing in a reinforcement learning context to actuate a soft, slender muscular arm. The arm must track a moving target in a partially observable environment. Soft robots present an especially challenging test bed for reinforcement learning due to their nonlinear, continuum dynamics. We propose learning strategies for two classes of reservoirs: Echo State Networks (ESNs) built using tanh neurons and Liquid State Machines (LSMs) built using Leaky Integrate and Fire (LIF) neurons. Unlike traditional activation functions, LIF neurons provide discrete spike trains with respect to time, and LSM-like structures are found in vivo. Crucially, we prohibit any additional feedforward layers, making the reservoir the sole neural computing unit. Our ESN policy significantly outperforms the state-of-the-art algorithm Proximal Policy Optimization (PPO) and our LSM policy matches PPO. Finally, we deploy a LSM directly on neuromorphic hardware, opening up opportunities for energy efficient reinforcement learning and robotic control. End-to-end, our soft robot controlled with a spiking reservoir is a novel bio-inspired system.
Issue Date:2021-12-09
Rights Information:Copyright 2021 Keshav Shivam
Date Available in IDEALS:2022-04-29
Date Deposited:2021-12

This item appears in the following Collection(s)

Item Statistics