Recently, improving the robustness of policies across different environments attracts increasing attention in the reinforcement learning (RL) community. Existing robust RL methods mostly aim to achieve the max-min robustness by optimizing the policy’s performance in the worst-case environment. However, in practice, a user that uses an RL policy may have different preferences over its performance across environments. Clearly, the aforementioned max-min robustness is oftentimes too conservative to satisfy user preference. Therefore, in this paper, we integrate user preference into policy learning in robust RL, and propose a novel User-Oriented Robust RL (UOR-RL) framework. Specifically, we define a new User-Oriented Robustness (UOR) metric for RL, which allocates different weights to the environments according to user preference and generalizes the max-min robustness metric. To optimize the UOR metric, we develop two different UOR-RL training algorithms for the scenarios with or without a priori known environment distribution, respectively. Theoretically, we prove that our UOR-RL training algorithms converge to near-optimal policies even with inaccurate or completely no knowledge about the environment distribution. Furthermore, we carry out extensive experimental evaluations in 6 MuJoCo tasks. The experimental results demonstrate that UOR-RL is comparable to the state-of-the-art baselines under the average-case and worst-case performance metrics, and more importantly establishes new state-of-the-art performance under the UOR metric.

User-Oriented Robust Reinforcement Learning

Safety comes first in many real-world applications involving autonomous agents. Despite a large number of reinforcement learning (RL) methods focusing on safety-critical tasks, there is still a lack of high-quality evaluation of those algorithms that adheres to safety constraints at each decision step under complex and unknown dynamics. In this paper, we revisit prior work in this scope from the perspective of state-wise safe RL and categorize them as projection-based, recovery-based, and optimization-based approaches, respectively.  Furthermore, we propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. This novel technique explicitly enforces hard constraints via the deep unrolling architecture and enjoys structural advantages in navigating the trade-off between reward improvement and constraint satisfaction. To facilitate further research in this area,  we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit, a toolkit that provides off-the-shelf interfaces and evaluation utilities for safety-critical tasks. We then perform a comparative study of the involved algorithms on six benchmarks ranging from robotic control to autonomous driving. The empirical results provide an insight into their applicability and robustness in learning zero-cost-return policies without task-dependent handcrafting. The project page is available at \url{https://sites.google.com/view/saferlkit}.

Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks

Cooperative multi-agent reinforcement learning (c-MARL) is widely applied in safety-critical scenarios, thus the analysis of robustness for c-MARL models is profoundly important. However, robustness certification for c-MARLs has not yet been explored in the community. In this paper, we propose a novel certification method, which is the first work to leverage a scalable approach for c-MARLs to determine actions with guaranteed certified bounds. c-MARL certification poses two key challenges compared with single-agent systems: (i) the accumulated uncertainty as the number of agents increases; (ii) the potential lack of impact when changing the action of a single agent into a global team reward. These challenges prevent us from directly using existing algorithms. Hence, we employ the false discovery rate (FDR) controlling procedure considering the importance of each agent to certify per-state robustness and propose a tree-search-based algorithm to find a lower bound of the global reward under the minimal certified perturbation. As our method is general, it can also be applied in single-agent environments. We empirically show that our certification bounds are much tighter than those of the state-of-the-art RL certification solutions. We also run experiments on two popular c-MARL algorithms: QMIX and VDN, in two different environments, with two and four agents. The experimental results show that our method produces a meaningful guaranteed robustness for all models and environments. Our tool CertifyCMARL is available at https://github.com/TrustAI/CertifyCMARL.

Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning

As the complexity of pacemaker devices continues to grow, the importance of capturing its functional correctness requirement formally cannot be overestimated. The pacemaker system specification document by \emph{Boston Scientific} provides a widely accepted set of specifications for pacemakers. 
As these specifications are written in a natural language, they are not amenable for automated verification, synthesis, or reinforcement learning of pacemaker systems. This paper presents a formalization of these requirements for a dual-chamber pacemaker in \emph{duration calculus} (DC), a highly expressive real-time specification language.
The proposed formalization allows us to automatically translate pacemaker requirements into executable specifications as stopwatch automata, which can be used to enable simulation, monitoring, validation, verification and automatic synthesis of pacemaker systems. 
The cyclic nature of the pacemaker-heart closed-loop system results in DC requirements that compile to a decidable subclass of stopwatch automata. We present shield reinforcement learning (shield RL),  a shield synthesis based reinforcement learning algorithm, by automatically constructing safety envelopes from DC specifications. 


Correct-by-Construction Reinforcement Learning of Cardiac Pacemakers from Duration Calculus Requirements

Spin-orbit torque enables electrical control of the magnetic state of ferromagnets or antiferromagnets. The quintessential system exhibiting spin-orbit torque is a bilayer composed of a heavy metal and a ferromagnet. The heterostructure bilayer geometry breaks the inversion symmetry and allows the spin-orbit torque arising from the spin current flow from the heavy metal layer to the ferromagnetic layer. In this work, we consider the intrinsic spin-orbit torque generated in inversion symmetry breaking magnetic materials directly without stacking with other materials. In the first example of ferromagnetic monolayer Fe3GeTe2, we calculate the intrinsic spin-orbit torque which arises due to the global inversion symmetry breaking. In addition, spin-orbit torque in Fe3GeTe2 has the unique property that the lowest order contribution is fieldlike and time-reversal even because of the C3z rotation symmetry. This is in contrast to the conventional case in which the lowest order is time-reversal odd and fieldlike and the next lowest-order is time-reversal even and dampinglike. We next show that the next lowest-order spin-orbit torque in Fe3GeTe2 is time-reversal odd and dampinglike which can play a more important role in switching the magnetization in the ensuring dynamics, as described by the Landau-Lifshitz-Gilbert equation. In the second example of antiferromagnetic bilayer CrI3, we find staggered dampinglike (time-reversal even) torque and uniform fieldlike (time-reversal odd) torque due to local inversion symmetry breaking and global inversion plus time-reversal symmetry. This is similar to the previous studies of CuMnAs and Mn2Au, where the Neel order switching is demonstrated using uniform fieldlike torque. However, CrI3 has a unique property in that the anisotropy energies and exchange energies are similar in magnitude. This leads to qualitatively different results in the switching dynamics: the staggered dampinglike torque is the dominant switching mechanism. The primary role of staggered dampinglike torque applies to other antiferromagnets in which the product of Gilbert damping and exchange field is smaller than the anisotropy field.

Intrinsic spin-orbit torque in van der Waals magnetic materials

Materials with helical crystal structures have been suggested to be a condensed matter analogue of solenoids, where a current flow produces inductive magnetic fields [1]. At the microscopic scale, a finite spin accumulation can be generated in strong spin-orbit coupling and low symmetry systems via the Edelstein effect [2,3]. Recently, chiral non-magnetic materials have received a lot of attention due to their low symmetries and complex radial spin textures, that could be potentially exploited for spintronic device applications [4,5,6]. However, a straightforward magnetotransport study on the relationship between chirality, spin texture and current-induced spin polarization is still missing. Here, we link one-to-one the direction of the spin polarization, induced by a chirality-dependent Edelstein effect, to the handedness of chiral tellurium (Te) nanowires (NWs), through simple angle-dependent magnetotransport experiments. Transmission electron microscopy (TEM) analysis and theoretical calculations corroborate the transport data. Te NWs show a large, chirality-dependent non-reciprocal charge transport when the electrical current and the magnetic field applied are collinear to the chiral axis of the crystal. By applying a gate voltage, we can also manipulate the magnetotransport response, acting on the carrier density and modifying the spin texture at the Fermi surface. The efficiency of the charge-to-spin interconversion process is one order of magnitude larger than what has been reported so far. Our results are a step forward towards the development of chiral spintronics.

![](https://assets.underline.io/uploads/markdown_image/1/image/cb46e659c3ceca2ac8d3e113ec8dbb71.jpg)

Figure 1: Sketch of the chiral structure of right- and left-handed Te NWs where the current-induced spin polarization depends on the crystal handedness

![](https://assets.underline.io/uploads/markdown_image/1/image/6de41c2cd5dcdb4672e53a8c0641cd6e.jpg)

Figure 2: On the leftt, sketch representing the Edelstein effect mechanism. The shift in kz of the Fermi contours lead to the formation of spin accumulations oriented in opposite directions according to the chirality-dependent radial spin texture of Te. On the right, current-dependent angular dependences of the magnetoresistance Rdiff= [R(Iz) - R(-Iz)]/2 measured at 9 T and 10 K for two Te NWs with opposites handedness. The signals show specular features between left- and right-handed NWs.




Chirality-dependent Edelstein effect in elemental Tellurium nanowires

Perpendicular Spin Transfer Torque Magnetic Random Access Memory (p-STT-MRAM) needs to combine high speed and low critical switching current at technological nodes below 30 nm. Higher STT write efficiency is therefore required. Double magnetic tunnel junctions (DMTJ) stacks have been proposed to achieve this goal [1-5]. The DMTJ critical switching current is decreased by the combined effect of the STT exerted from two antiparallel polarizers on the storage layer (SL). The retention is increased by doubling the SL/MgO interfaces contributing to the layer’s effective perpendicular anisotropy (PMA). Despite the efficiency advantage, the development of p-DMTJ presents several complexities. First, ensuring enough PMA for the top polarizer [6] and second, controlling the stray field compensation of both polarizers acting on the SL. A p-DMTJ structure having a switchable reference layer was recently proposed [7], showing through magnetic simulation similar performance as conventional p-DMTJ with static polarizers. In this work, we report experimental evidence of the working principle for this type of novel p-DMTJ. We fabricated and optimized an ASL-DMTJ structure having a switchable polarizer acting as a switching assistance layer (ASL). Fig.1.(a) and (b) show the two switching sequences, from AP to P and from P to AP states respectively. Both reversals show a transition through an intermediate resistance state (IRS). Differing from the working principles proposed in [7], the ASL is designed to switch its magnetic orientation during the write operation. This maximizes the STT efficiency independently of the switching direction, while also increasing the SL retention, since the assistance layer always aligns parallel to the SL in standby Fig.2(a)-(b). Validation of the switching sequences was possible by performing time-resolved resistance measurements Fig.2.(c)-(b).

![](https://assets.underline.io/uploads/markdown_image/1/image/0c7395b35a105c6f992d8d15bd884588.jpg)

Figure 1. (a) AP→P and (b) P→AP reversal sequence through an intermediated resistance state.

![](https://assets.underline.io/uploads/markdown_image/1/image/938d95a5c9fad68f070f40573955c5dd.jpg)

Figure 2. (a) RH of ASL-DMTJ (b) RV loops of ASL-DMTJ. (b) Real time AP→P switching sequence and (d) real time P to AP switching sequence both transiting through the IRS with (I) applied field, (II) at zero field.

Real time investigation of Double magnetic tunnel junction with switchable assistance layer for high efficiency STT-MRAM

The Ising machine is an unconventional computing architecture that can be used to solve NP-hard combinatorial optimization problems more efficiently than traditional von Neumann architectures. GHz spin Hall nano-oscillators (SHNOs) are a particularly attractive technology for building fast, energy-efficient, and scalable Ising machines; however, electrical coupling mechanisms which allow for full programmability among different oscillator nodes have not yet been fully demonstrated in such a network. Here, we develop a general analytical framework that can describe injection locking of SHNOs with arbitrary oscillation orbits at both the fundamental frequency and harmonics. With this compact analytical framework, we integrate the SHNO into a Verilog-A device model that can emulate the oscillator’s injection locking behavior in circuit simulations, where the dynamics of coupled oscillator networks are simulated together with conventional electronic components (Fig. 1). While our abstract circuit simulations achieve similar accuracy as full micromagnetic simulations, the device model we developed leads to more than 100 times improvement in simulation efficiency and allows us to evaluate the performance using realistic circuits. Using circuit-level simulations, we further study the effects of phase noise and scalability in networks of up to hundreds of coupled oscillators to show that the SHNO-based Ising machine can be operated robustly at room temperature. Compared with existing technologies, SHNO networks exhibit orders of magnitude improvement in time, space, and energy efficiency (Table 1). Our results provide analytical tools and design insights that will be useful for the realization of a CMOS-integrated SHNO Ising machine.

![](https://assets.underline.io/uploads/markdown_image/1/image/a0dab57c9a06319f78760697037422fd.jpg)

Fig. 1: (a) Electrical coupling circuit linking two SHNOs in the oscillator-based Ising machine. (b) Phases of oscillators in a 4-node coupled network solving the Ising model in circuit simulations (HSPICE) using our abstract device model and in micromagnetic simulations (MuMax3).

![](https://assets.underline.io/uploads/markdown_image/1/image/56041ec137cf8fc9052b69c276e29915.jpg)

Table 1: Comparison of speed, energy, and space metrics between the SHNO Ising machine and existing technologies. Values are standardized to a 100-node network.

Using Machine Based on Electrically Coupled Spin Hall Nano-Oscillators

Antiferromagnets (AFMs) are magnetically ordered materials without a macroscopic magnetization. There is accelerating interest in using them for memory applications, motivated by the possibility for electrically controlling AFM order via spin-orbit torques, and its read-out via magnetoresistive effects [1-2]. Here we show that micrometer-scale bits of the antiferromagnet PtMn, grown on a Ta or Pt heavy metal (HM) layer, can be switched reversibly between different magnetic states by electric currents, with readout through a second harmonic voltage technique [3]. The device exhibits multiple states, which can be set using different current amplitudes and may find application in neuromorphic computing. Micromagnetic simulations suggest that these metastable states consist of vortex and anti-vortex textures that move in response to current. We also demonstrate, using a modified device layout, direct voltage measurements to read out the electrically-induced switching of noncollinear PtMn3. The modified layout and measurement methodology, which was recently demonstrated using the IrMn3/Pt material system [4], is utilized here to distinguish current-induced magnetic and nonmagnetic switching signals in PtMn3/Pt devices. We also realize switching down to 10 µs pulse width, which shows the potential for fast device applications. The effect of temperature and external magnetic field (up to 15 T) are also studied. Our results pave the way towards practical PtMn and PtMn3 antiferromagnetic memories integrated on silicon, and indicate the importance of micromagnetic textures in the static and dynamic behavior of antiferromagnetic devices.

Electrical manipulation of antiferromagnetic order in PtMn and PtMn3 pillars

The patch antenna is advantageous in many applications because of its simple and inexpensive structure. However, one disadvantage is its large size. One of the methods to reduce the size of an antenna is to utilize ferrite material [1]. While this does effectively miniaturize the antenna, the peak realized gain (PRG) is decreased by 8.3 dBic due to the magnetic loss of the ferrite core. In this paper, we take the multi-layer ferrite and dielectric approach to reduce the patch antenna size. All antennas were simulated using ANSYS HFSS for their performance. Fig. 1 shows the specifications and designs of the patch antennas. The baseline patch antenna has the dimensions given in [2] and uses a 1.4 mm thick FR-4 substrate (permeability (μr) = 1, permittivity (εr) = 4.4). The dielectric and magnetic loss tangents (tanδe and tanδu) are 0.02 and 0.00, respectively, at UHF band frequency, and its volume (V) is 9800 mm3. To realize the multi-layer concept in the antenna size reduction, 0.8 mm thick layers of ferrite (μr = 6, εr =13, tanδu = 0.05, tanδe = 0.02, and V = 2720 mm3) are inserted into the FR-4 substrate. Three topologies are investigated: one ferrite layer, two ferrite layers, and three ferrite layers. For each topology, the volume of the ferrite remains the same. For the cases with more than one ferrite layer, the inner-most layer has a constant V = 600 mm3, and the remaining volume is divided equally among the outer layers. Antenna performance results are summarized in Table 1. For one ferrite layer, 16.7% miniaturization is achieved, but the PRG decreases by 14.1 dBi. However, as the number of layers increases to two, 49.8% miniaturization is achieved with a decrease in PRG of 5.6 dBi. For three layers, 49.3% miniaturization is observed with a 5.6 dBi decrease in PRG. Compared to the case with the all-ferrite substrate (V = 9800 mm3), the patch antenna with two ferrite layers has a 4.4 dBi higher PRG. Ferrite and dielectric substrate contributes remarkably to antenna size reduction.

![](https://assets.underline.io/uploads/markdown_image/1/image/e68b9a4e3e72ffd5c9f56c3ee81cfc08.jpg)

Fig. 1. Design and specifications of (a) baseline patch antenna, (b) ferrite-dielectric layering, and (c) multi-layer patches

![](https://assets.underline.io/uploads/markdown_image/1/image/74ed72c7f2f5ae5991e9a4a0fc105818.jpg)

Table 1. Comparison of antenna performance results

Miniaturizing Patch Antenna with Multi-layer Soft Ferrite and Dielectric Substrate Structure

Neural Machine Translation (NMT) systems have been shown to perform impressively on many language pairs compared to Statistical Machine Translation (SMT). However, these systems are data-intensive, which is problematic for the majority of language pairs, and especially for low-resource languages. In this work, we address this issue in the case of certain Arabic dialects, those variants of Modern Standard Arabic (MSA) that are spelling non-standard, morphologically rich, and yet resource-poor variants. Here, we have experimented with several multitasking learning strategies to take advantage of the relationships between these dialects. Despite the simplicity of this idea, empirical results show that several multitasking learning strategies are capable of achieving remarkable performance compared to statistical machine translation. For instance, we obtained the BLUE scores for the Algerian → Modern-Standard-Arabic and the Moroccan → Palestinian of 35.06 and 27.55, respectively, while the scores obtained with a statistical method are 15.1 and 18.91 respectively. We show that on 42 machine translation experiments, and despite the use of a small corpus, multitasking learning achieves better performance than statistical machine translation in 88% of cases.

Underline digital video library

User-Oriented Robust Reinforcement Learning

Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks

Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning

Correct-by-Construction Reinforcement Learning of Cardiac Pacemakers from Duration Calculus Requirements

Intrinsic spin-orbit torque in van der Waals magnetic materials

Chirality-dependent Edelstein effect in elemental Tellurium nanowires

Real time investigation of Double magnetic tunnel junction with switchable assistance layer for high efficiency STT-MRAM

Using Machine Based on Electrically Coupled Spin Hall Nano-Oscillators

Electrical manipulation of antiferromagnetic order in PtMn and PtMn3 pillars

Miniaturizing Patch Antenna with Multi-layer Soft Ferrite and Dielectric Substrate Structure

Improving Machine Translation of Arabic Dialects through Multi-Task Learning

Knowledge-based neural pre-training for Intelligent Document Management

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES