Reinforcement Learning (RL) is a sampling based approach to optimization, where learning agents rely on scalar reward signals to discover optimal solutions. The specification of learning objectives as scalar rewards is tedious and error prone, and more so for real-time systems with complex time- critical requirements. This paper advocates the use of Duration Calculus (DC)—a highly expressive real-time logic with duration and length modalities—in expressing the learning objectives in model-free RL for stochastic real-time systems. On the other hand, to model stochastic real-time environments, we consider probabilistic timed automata (PTA)—Markov decision processes extended with clock variables—that provide an expressive yet computationally decidable formalism to capture real-time con- straints over nondeterministic and probabilistic behaviors. The key hurdle in developing a convergent RL algorithm for DC specifications is the undecidability of the synthesis problem for PTA against general DC specifications. Inspired by the di- chotomy between event-triggered and time-triggered approaches to the design of real-time systems, we present two variants of DC logic—that we dub event-triggered duration calculus (EDC) and time-triggered duration calculus (TDC)—and identify their subclasses with appealing theoretical properties. We study the decidability (and exact complexity) of the satisfiability of these calculi as well as the controller synthesis against PTA models. Based on these results, we propose a reward scheme for RL agents in such a way that guarantees that any RL algorithm maximizing rewards is guaranteed to maximize the probability of satisfaction for the given DC specification. The effectiveness of the proposed approach is demonstrated via grid-world benchmarks and a proof-of-concept case study for synthesizing control for simple cardiac pacemaker directly from a set of DC specifications.

Event-Triggered and Time-Triggered Duration Calculus for Model-Free Reinforcement Learning

**Session chair:** Luis Almeida

- 25 min: [**Vulnerability of Controller Area Network to Schedule-Based Attacks**](https://conferences.computer.org/rtsspub/pdfs/RTSS2021-23beZleJJDdiozCiQ9rzXD/280200a495/280200a495.pdf), *Sena Hounsinou, Mark Stidd, Uchenna Ezeobi, Habeeb Olufowobi, Mitra Nasri and Gedare Bloom*
- 25 min: [**AegisDNN: Dependable and Timely Execution of DNN Tasks with SGX**](https://conferences.computer.org/rtsspub/pdfs/RTSS2021-23beZleJJDdiozCiQ9rzXD/280200a068/280200a068.pdf), *Yecheng Xiang, Yidi Wang, Hyunjong Choi, Mohsen Karimi and Hyoseung Kim*
- 25 min: [**Event-Triggered and Time-Triggered Duration Calculus for Model-Free Reinforcement Learning (Artifact Evaluated)**](https://conferences.computer.org/rtsspub/pdfs/RTSS2021-23beZleJJDdiozCiQ9rzXD/280200a240/280200a240.pdf), *Kalyani Dole, John Komp, Ashutosh Gupta, Krishna S. and Ashutosh Trivedi*
- 5 min: [**(WiP) Enabling Secure Boot for Real-Time Restart-Based Cyber-Physical Systems**](https://conferences.computer.org/rtsspub/pdfs/RTSS2021-23beZleJJDdiozCiQ9rzXD/280200a524/280200a524.pdf), *Sena Hounsinou, Vijay Kumar Banerjee, Chunhao Peng, Monowar Hasan and Gedare Bloom*
- 5 min: [**(WiP) Improving Resilience of Distributed Real-Time Applications via Security and Fault Tolerance Co-Design**](https://conferences.computer.org/rtsspub/pdfs/RTSS2021-23beZleJJDdiozCiQ9rzXD/280200a536/280200a536.pdf), *Wei Jiang, Xinke Liao, Jinyu Zhan and Ke Jiang*



4.22 - Verification, Security, and Trust + 2 BPs

technical paper

 **RTSS 2021** has successfully concluded. Hope you have enjoyed it, and look forward to seeing you next year in person.
    Announcement for the Awards:
 * Best Paper Award goes to: Monte Carlo Response-Time Analysis, Sergey Bozhko, Georg von der Brüggen, and Björn Brandenburg
 * Best Student Paper Award goes to: Automatic Energy-Hotspot Detection and Elimination in Real-Time Deeply Embedded Systems, Mohsen Shekarisaz, Lothar Thiele, and Mehdi Kargahi.
 * Best Solution Award for Industry Session goes to: Toward Real-Time Guaranteed Scheduling for Autonomous Driving Systems. Jinghao Sun, Tianyi Wang, Kailu Duan, Bin Lu, Jiankang Ren, Zhishan Guo, and Guozhen Tan.



IEEE RTSS 2021

The IEEE Real-Time Systems Symposium (RTSS) is the premier conference in the field of real-time systems, and is a venue for researchers and practitioners to showcase innovations covering all aspects of real-time systems, including theory, design, analysis, implementation, evaluation, and experience. 

Kalyani Dole

1

Presentations

Event-Triggered and Time-Triggered Duration Calculus for Model-Free Reinforcement Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES