Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system

Shivam, Keshav

Permalink

https://hdl.handle.net/2142/114113 Copy

Description

Title

Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system

Author(s)

Shivam, Keshav

Issue Date

2021-12-09

Director of Research (if dissertation) or Advisor (if thesis)

• Gazzola, Mattia
• Chowdhary, Girish

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Soft robotics
• reservoir computing
• reinforcement learning
• neuromorphic computing
• spiking neural networks

Abstract

Modern approaches in machine learning and artificial intelligence are dominated by deep learning. Although inspired by the brain, these network architectures are not biologically plausible. In contrast, the reservoir computing paradigm has a single sparsely and recurrently connected hidden layer, with the linear readout layer being the only learned parameter. We apply reservoir computing in a reinforcement learning context to actuate a soft, slender muscular arm. The arm must track a moving target in a partially observable environment. Soft robots present an especially challenging test bed for reinforcement learning due to their nonlinear, continuum dynamics. We propose learning strategies for two classes of reservoirs: Echo State Networks (ESNs) built using tanh neurons and Liquid State Machines (LSMs) built using Leaky Integrate and Fire (LIF) neurons. Unlike traditional activation functions, LIF neurons provide discrete spike trains with respect to time, and LSM-like structures are found in vivo. Crucially, we prohibit any additional feedforward layers, making the reservoir the sole neural computing unit. Our ESN policy significantly outperforms the state-of-the-art algorithm Proximal Policy Optimization (PPO) and our LSM policy matches PPO. Finally, we deploy a LSM directly on neuromorphic hardware, opening up opportunities for energy efficient reinforcement learning and robotic control. End-to-end, our soft robot controlled with a spiking reservoir is a novel bio-inspired system.

Graduation Semester

2021-12

Type of Resource

Thesis

Permalink

http://hdl.handle.net/2142/114113

Copyright and License Information

Copyright 2021 Keshav Shivam

Owning Collections