Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system
Shivam, Keshav
Loading…
Permalink
https://hdl.handle.net/2142/114113
Description
Title
Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system
Author(s)
Shivam, Keshav
Issue Date
2021-12-09
Director of Research (if dissertation) or Advisor (if thesis)
Gazzola, Mattia
Chowdhary, Girish
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Date of Ingest
2022-04-29T21:58:42Z
Keyword(s)
Computer science
Language
eng
Abstract
Modern approaches in machine learning and artificial intelligence are dominated by deep learning. Although inspired by the brain, these network architectures are not biologically plausible. In contrast, the reservoir computing paradigm has a single sparsely and recurrently connected hidden layer, with the linear readout layer being the only learned parameter. We apply reservoir computing in a reinforcement learning context to actuate a soft, slender muscular arm. The arm must track a moving target in a partially observable environment. Soft robots present an especially challenging test bed for reinforcement learning due to their nonlinear, continuum dynamics. We propose learning strategies for two classes of reservoirs: Echo State Networks (ESNs) built using tanh neurons and Liquid State Machines (LSMs) built using Leaky Integrate and Fire (LIF) neurons. Unlike traditional activation functions, LIF neurons provide discrete spike trains with respect to time, and LSM-like structures are found in vivo. Crucially, we prohibit any additional feedforward layers, making the reservoir the sole neural computing unit. Our ESN policy significantly outperforms the state-of-the-art algorithm Proximal Policy Optimization (PPO) and our LSM policy matches PPO. Finally, we deploy a LSM directly on neuromorphic hardware, opening up opportunities for energy efficient reinforcement learning and robotic control. End-to-end, our soft robot controlled with a spiking reservoir is a novel bio-inspired system.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
