To contribute to the diversity of the next generation of NIH funded biomedical researchers, ensuring that undergraduate student participants enter and succeed in graduate school is one of the goals of the BUILDing SCHOLARS center at the University of Texas at El Paso. To accomplish this goal, the center implemented the Johnson and Bozeman Asset Bundles model. Using the Student Hallmarks of Success developed by the NIH Diversity Program Consortium as guidance, and mapping each program activity to the model, academic metrics in combination with results from a comprehensive evaluation plan for the first three cohorts of participants will be presented.<

Developing the Biomedical Researchers of the Future at a Hispanic Serving Institution

As the complexity of pacemaker devices continues to grow, the importance of capturing its functional correctness requirement formally cannot be overestimated. The pacemaker system specification document by \emph{Boston Scientific} provides a widely accepted set of specifications for pacemakers. 
As these specifications are written in a natural language, they are not amenable for automated verification, synthesis, or reinforcement learning of pacemaker systems. This paper presents a formalization of these requirements for a dual-chamber pacemaker in \emph{duration calculus} (DC), a highly expressive real-time specification language.
The proposed formalization allows us to automatically translate pacemaker requirements into executable specifications as stopwatch automata, which can be used to enable simulation, monitoring, validation, verification and automatic synthesis of pacemaker systems. 
The cyclic nature of the pacemaker-heart closed-loop system results in DC requirements that compile to a decidable subclass of stopwatch automata. We present shield reinforcement learning (shield RL),  a shield synthesis based reinforcement learning algorithm, by automatically constructing safety envelopes from DC specifications. 


Correct-by-Construction Reinforcement Learning of Cardiac Pacemakers from Duration Calculus Requirements

Recently, improving the robustness of policies across different environments attracts increasing attention in the reinforcement learning (RL) community. Existing robust RL methods mostly aim to achieve the max-min robustness by optimizing the policy’s performance in the worst-case environment. However, in practice, a user that uses an RL policy may have different preferences over its performance across environments. Clearly, the aforementioned max-min robustness is oftentimes too conservative to satisfy user preference. Therefore, in this paper, we integrate user preference into policy learning in robust RL, and propose a novel User-Oriented Robust RL (UOR-RL) framework. Specifically, we define a new User-Oriented Robustness (UOR) metric for RL, which allocates different weights to the environments according to user preference and generalizes the max-min robustness metric. To optimize the UOR metric, we develop two different UOR-RL training algorithms for the scenarios with or without a priori known environment distribution, respectively. Theoretically, we prove that our UOR-RL training algorithms converge to near-optimal policies even with inaccurate or completely no knowledge about the environment distribution. Furthermore, we carry out extensive experimental evaluations in 6 MuJoCo tasks. The experimental results demonstrate that UOR-RL is comparable to the state-of-the-art baselines under the average-case and worst-case performance metrics, and more importantly establishes new state-of-the-art performance under the UOR metric.

User-Oriented Robust Reinforcement Learning

Safety comes first in many real-world applications involving autonomous agents. Despite a large number of reinforcement learning (RL) methods focusing on safety-critical tasks, there is still a lack of high-quality evaluation of those algorithms that adheres to safety constraints at each decision step under complex and unknown dynamics. In this paper, we revisit prior work in this scope from the perspective of state-wise safe RL and categorize them as projection-based, recovery-based, and optimization-based approaches, respectively.  Furthermore, we propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. This novel technique explicitly enforces hard constraints via the deep unrolling architecture and enjoys structural advantages in navigating the trade-off between reward improvement and constraint satisfaction. To facilitate further research in this area,  we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit, a toolkit that provides off-the-shelf interfaces and evaluation utilities for safety-critical tasks. We then perform a comparative study of the involved algorithms on six benchmarks ranging from robotic control to autonomous driving. The empirical results provide an insight into their applicability and robustness in learning zero-cost-return policies without task-dependent handcrafting. The project page is available at \url{https://sites.google.com/view/saferlkit}.

Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks

Cooperative multi-agent reinforcement learning (c-MARL) is widely applied in safety-critical scenarios, thus the analysis of robustness for c-MARL models is profoundly important. However, robustness certification for c-MARLs has not yet been explored in the community. In this paper, we propose a novel certification method, which is the first work to leverage a scalable approach for c-MARLs to determine actions with guaranteed certified bounds. c-MARL certification poses two key challenges compared with single-agent systems: (i) the accumulated uncertainty as the number of agents increases; (ii) the potential lack of impact when changing the action of a single agent into a global team reward. These challenges prevent us from directly using existing algorithms. Hence, we employ the false discovery rate (FDR) controlling procedure considering the importance of each agent to certify per-state robustness and propose a tree-search-based algorithm to find a lower bound of the global reward under the minimal certified perturbation. As our method is general, it can also be applied in single-agent environments. We empirically show that our certification bounds are much tighter than those of the state-of-the-art RL certification solutions. We also run experiments on two popular c-MARL algorithms: QMIX and VDN, in two different environments, with two and four agents. The experimental results show that our method produces a meaningful guaranteed robustness for all models and environments. Our tool CertifyCMARL is available at https://github.com/TrustAI/CertifyCMARL.

Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning

The use of time-varying magnetic field sources is indispensable in numerous applications, such as magnetic resonance imaging, sensors, motors, actuators, magnetometers or magnetic
refrigerators. Halbach cylinders have been used as an alternative to electromagnets, since they have no losses by the Joule effect. However, rod mangle systems are cheap and promising
alternatives to the currently used Halbach cylinders [1-3]. The use of rotation patterns has potential for application in several innovative áreas [4].
In this work, we modeled rod mangle structures with different number of rods, aiming to control the magnitude, direction and homogeneity of the magnetic field. Figure 1 (a) shows the
modeled system for 3 rods. After determining the configuration with the maximum magnetic flux density at the center of each system, it was possible to generate a homogeneous magnetic
field. The homogeneity increases with the number of rods and for a large number of rods, the rod mangle structure begins to resemble an ideal Halbach. We also analyzed two different
rotation patterns to generate a decreasing magnetic field over time. For the first pattern only the direction of the magnetic field of the odd configurations remained constant. For the second
rotation pattern, after the magnetic field is reduced to zero, it starts to increase in the opposite direction and then decreases to zero again. If we rotate all rods counterclockwise, the
magnetic field will rotate clockwise, keeping the magnitude fairly constant. Finally, we validated the results experimentally with structures of 3 and 4 rods (Fig 1 (b)). 

**References:** 

[1] J. M. D. Coey, Permanent magnet applications, Journal of Magnetism and Magnetic Materials 248 (3) (2002) 441–456. 
[2] O. Cugat, P. Hansson, J. M. D.Coey, Permanent magnet variable flux sources, IEEE Transactions on Magnetics 30 (6) (1994) 4602–4604. 
[3] F. Hiptmair, Z. Major, R. Haßlacher, S. Hild, Design and application of permanent magnet flux sources for mechanical testing of magnetoactive elastomers at variable field directions,
Review of Scientific Instruments 86 (8) (2015) 085107. 
[4] H. Sakuma, T. Nakagawara, Optimization of rotation patterns of a mangle-type magnetic field source using covariance matrix adaptation evolution strategy, Journal of Magnetism and
Magnetic Materials 527 (2021) 167752. 

![](https://s3.eu-west-1.amazonaws.com/underline.prod/uploads/markdown_image/1/image/38aef6bfcfd0e0a9d8729e58145a10d4.png)

Rod mangle rotation patterns for magnetic field generation

Magnetorheological fluids (MRFs) can change their viscosity in response to applied magnetic field. Magnetorheological energy absorbers (MREAs) are being investigated for adaptive shock isolation because MREAs can adjust their stroking load to severity of impact when magnetic field is used to control MRF viscosity. Employing MRFs in MREAs requires a highly stable suspension to maintain a uniform concentration, because MREAs would be used only in the event of an impact or crash event, that is, rarely. Suspension stability of MRFs can be studied using an automated vertical axis inductance monitoring system (AVAIMS), where an inductance sensor is translated vertically along a column of MRF to measure particle concentration as a function of column height and time [1,2]. The AVAIMS allows for tracking discontinuities in density or shocks including: the mud-line (boundary between the topmost supernatant (clear fluid) zone and the original concentration zone), the gel-line (boundary between the original concentration zone below mud-line and variable concentration zone), and the cake-line (boundary between the variable concentration zone and the sediment zone at the bottom of the column). MRF sedimentation can be evaluated comprehensively analyzing these boundaries. The AVAIMS developed here (Fig. 1) seeks to improve sedimentation zone boundary localization using an optimized inductance sensor to measure concentration gradient as a function of time. The inductance sensor design, material selection, and geometry were optimized and permalloy and plastic were used advantageously to improve sensor sensitivity. The calibration curve of the sensor was obtained by measuring samples with different particle volume fractions. The sensor design was validated by examining a uniformly mixed MRF column for seven consecutive days after preparation of an MRF with an Fe solids loading of 20 vol%. These tests show that the AVAIMS developed here was able to better localize sedimentation zone boundaries using concentration gradient profile method (Fig. 2).

Sedimentation of Magnetorheological Fluids Measured Using a Automated Vertical Axis Inductance Monitoring System

Magnetorheological fluids (MRFs) consist of micron-sized magnetic particles dispersed in a carrier fluid for which rheological properties can be tuned reversibly with
magnetic field.1 If an MRF is contained in a cellular elastomeric encapsulant (CEE), its mechanical properties, such as stiffness and damping, can be controlled via magnetic field. Such
tunability can be exploited for adaptive vibration control or energy absorption systems. A magnetorheological damper often uses an accumulator to modulate pressure injected by external
forces applied to the damper.2 An accumulator is implemented into the cellular encapsulant to enable essentially incompressible fluid to enter the accumulator and to allow compression of
the cellular material to take place, and also enables a mechanical bulge stiffness to the design.
Here, MRF was prepared with silicon oil (40% vol. fraction). Thermoplastic polyurethane (TPU) was 3-D printed as the CEE system. (Fig. 1 a). The MRF was injected into the TPU-CEE,
followed by the printing of a sealing layer. The wall adjacent to the sealing layer had a circular opening and a TPU membrane was bonded to its perimeter so that the pressurized MRF
could flow through the opening and be accumulated in the resulting pocket.
An external magnetic field (0 and 7 kG) was applied along the sample longitudinal axis at the time of uniaxial dynamic testing with 5% pre-strain. The TPU-CEE samples were
characterized via force-displacement tests at 1 Hz under strain amplitudes. The MR effect on the dynamic stiffness is defined as the ratio of the dynamic stiffness under the maximum
magnetic field to the dynamic stiffness in the absence of a magnetic field. Dynamic stiffness in the presence as opposed to the absence of magnetic field was 264%. (Fig.2a) The area of the
force-displacement curve of the system with an accumulator is about 41% greater than one without an accumulator. The accumulator enhanced the increase in damping (Fig. 2b). The
increase in damping capacity generally increased as strain amplitude increased. 

**References:** 

1. Ashtiani M, Hashemabadi SH, Ghaffari A. A review on the magnetorheological fluid preparation and stabilization. Journal of Magnetism and Magnetic Materials. 374, 716.
(2015) 
2. Yu, Jianqiang & Dong, Xiaomin & Wang, Xuhong & Pan, Chengwang & Zhou, Yaqin. Asymmetric Dynamic Model of Temperature-Dependent Magnetorheological Damper and
Application for Semi-active System. Frontiers in Materials. 6. 227. (2019) 

![](https://s3.eu-west-1.amazonaws.com/underline.prod/uploads/markdown_image/1/image/9e96476131d9b4e1a0c5d4b36a289cf6.png)

Development of Magnetorheological Fluids within 3D Printed Elastomeric Cellular Structures with an Accumulator

Most research on remotely driven magnetic micro-robots focuses on their manipulation within fluids [1]. Actuating their motion in soft media and living tissue raises
challenges along with new opportunities for minimally invasive medicine. In contrast to fluids, moving solids in soft media irreversibly rearranges the material, leading to highly non-linear
and history dependent reaction forces. The present work demonstrates that this behavior enables solutions of certain long-standing problems. One problem stems from using magnetic field
gradients to generate forces which limits their range particularly for sub-millimeter size magnetic robots. Locomotion by the transfer of energy via a time varying uniform magnetic field
provides an alternative. In fluids where locomotion effectively occurs through swimming at low Reynolds numbers, this strategy can be costly, requiring more complex shapes (e.g. a screw
[2]) and control over a non-reciprocal motion gait [3]. In tissues, such complex shapes have difficulties in changing their trajectory direction over short distances. The novel finding in this
work is that soft media enables locomotion of simple solids, such as a pair of spheres (see Fig. 1), without requiring them to execute a complex motion gait. Using an external uniform
magnetic field to change the relative orientations of the magnetic moments, the disconnected spherical solids of slightly different diameters can locomote as one in a soft medium via
ratchet-like movements that are completely reciprocal. Such a configuration also allows the pair to change the direction of locomotion quickly. A demonstration is carried out in this work
via development of a theoretical model and its numerical simulation, although some experimental observations support these findings. 

**References:** 

[1] J. J. Abbott et al., "How should microrobots swim?," The international journal of Robotics Research, vol. 28, no. 11-12, pp. 1434-1447, 2009. 
[2] T. W. Fountain, P. V. Kailat, and J. J. Abbott, "Wireless control of magnetic helical microrobots using a rotating-permanent-magnet manipulator," in 2010 IEEE International
Conference on Robotics and Automation, 2010: IEEE, pp. 576-581. 
[3] E. Lauga, "Life around the scallop theorem," Soft Matter, vol. 7, no. 7, pp. 3060-3065, 2011. 

![](https://s3.eu-west-1.amazonaws.com/underline.prod/uploads/markdown_image/1/image/9995c10406482e83eef181eaa6b200c6.png)

On Locomotion of Magnetized Spherical Solids in Soft Media

Micro helical robots actuated by an external magnetic field have been investigated for a possible application to treat occlusive vascular disease. It requires a high-speed
rotating magnetic field (RMF) which generates fast drilling motion of the micro helical robot to tunnel the clogged lesions in blood vessels.1-3 The RMF is generated by the magnetic
navigation systems (MNSs) composed of multiple coils or electromagnets. However, as the frequency increases, the impedance from the inductance of the coils decreases the current and
the magnetic field. To overcome this problem, prior researchers utilized a resonance by including additional capacitance to their MNS.2,3 The electromagnets with magnetic cores generate
a large magnetic field.4 Unlike the MNS with coils only, the MNS with electromagnets has significant mutual-inductance due to the magnetic flux linkages between electromagnets, but
any resonance method suppressing the self and mutual inductances of the electromagnets of the MNS has not been reported.
We propose a method to suppress both the self and mutual-inductance effect of the 3-phase MNS with multiple electromagnets. First, we set the imaginary part and the real parts except the
resistance in the voltage equation to be zero to express the resonance. After solving the real part of the voltage equation and the actuation matrix equation which relates applied current with
flux density together, we can determine 3-phase currents and corresponding phases at resonance. The capacitance at resonance can be determined by solving the imaginary part in the
voltage equation. Fig. 1 shows the 3-phase MNS developed for the experiments and Fig. 2 shows the calculated and measured 3-phase currents and RMF with the application of the
proposed method to generate a magnetic flux density of 10 mT. It shows that the calculated current matches well with the measured one within 5 % discrepancy. The proposed method in
this research can contribute to improving the tunneling performance of micro helical robots by generating a high-speed RMF even in the MNS with the electromagnets. 

**References:** 

1Q. Wang, X. Du, and L. Zhang, “Real-time ultrasound Doppler tracking and autonomous navigation of a miniature helical robot for accelerating thrombolysis in dynamic
blood flow,” ACS Nano, vol. 16, no. 1 (2022). 
2K. T. Nguyen, B. Kang and J. Park, “High-frequency and High-powered electromagnetic actuation system utilizing Two-stage resonant effects,” IEEE/ASME Transactions on
Mechatronics, vol. 25, no. 5 (2020). 
3J. Nam, W. Lee, and G. Jang, “Magnetic navigation system utilizing resonant effect to enhance magnetic field applied to magnetic robots,” IEEE Transactions on Industrial Electronics,
vol. 64, no. 6 (2017). 
4J. Nam, W. Lee, E. Jung, and G. Jang, “Magnetic navigation system utilizing a closed magnetic circuit to maximize magnetic field and a mapping method to precisely control magnetic
field in real time,” IEEE Transactions on Industrial Electronics, vol. 65, no. 7 (2018). 

![](https://s3.eu-west-1.amazonaws.com/underline.prod/uploads/markdown_image/1/image/30e63ebc29490f9ce87650724a4ded38.png)

Resonant Control Method to Suppress the Self and Mutual Inductances of a 3 phase Magnetic Navigation System for Fast Drilling Motion of Micro Helical Robots

Magnetic catheter (MC) controlled by a magnetic navigation system (MNS) and a feeding device have been developed to overcome the steerability and the pushability of
conventional catheters.1,2 External magnetic field (EMF) produced from the MNS generates magnetic torque at the magnets in the MC to remotely steer the MC’s distal end while the
feeding device pushes the proximal end of the MC to move the MC forward.3,4 Sometimes, it does not work effectively when buckling of a flexible catheter occurs in complex blood
vessels.
We propose a position control method (PCM) of the Robotically Assisted MAgnetic Navigation (RAMAN) system to improve the pushability of the MC by maximizing magnetic force
(MF). Fig 1(a). shows the developed RAMAN system in which the 7-axis stage robot can move the 8 electromagnets (EMs) and the workspace surrounded by them. Fig 1(b). shows the MF
in the region of interest (ROI) which is 80% of the workspace. The PCM takes advantage of the fact that the MF increases as the magnet moves close to the EM. When the MF generated at
the MC is not sufficient, the PCM compares the MF of each vertex in the ROI and controls the 7-axis stage robot to move the ROI in such a way that the MC is located close to one of the
EMs to generate maximum MF. Fig 2. shows calculated maximum MF at the center of the ROI and at the point selected by the PCM where an axially magnetized ring magnet (OD: 2 mm,
ID: 1 mm, Length: 10 mm) of NdFeB 52 is located along θ and Φ directions. It shows that the PCM generates larger MF in any directions and 5 times larger MF on average than the
maximum MF generated at the center. We performed a navigation experiment of the MC along the aorta to the right coronary artery in a cardiac vascular phantom model. We observed a
buckling of the MC, and the maximum MF of 2.61 mN at the center did not overcome the buckling. We applied the PCM which generated the MF of 20.1 mN to release the buckling of the
MC. The PCM may contribute to controlling of the MC for robotic endovascular intervention. 

**References:** 

1B. J. Nelson, I. K. Kaliakatsos, and J. J. Abbott, Annual Review of Biomedical Engineering., Vol. 12, 55 (2010).
2N. Kim, S. Lee, W. Lee, and G. Jang, AIP Advances, Vol. 8, 056708 (2018).
3J. Nam, W. Lee, E. Jung, and G. Jang, IEEE Transactions on Industrial Electronics, Vol. 65, 5673 (2018)..
4E. Jung, W. Lee, N. Kim, J. Kim, J. Park, and G. Jang, AIP Advances, Vol 9, 125230 (2019). 

![](https://s3.eu-west-1.amazonaws.com/underline.prod/uploads/markdown_image/1/image/1c5902cf7e1a3cdb456194b529d812ca.png)

Position Control Method of a Robotically Assisted Magnetic Navigation System to Improve the Pushability of a Magnetic Catheter by Maximizing Magnetic Force

Transcranial magnetic stimulation (TMS) is a non-invasive, effective, and safe neuromodulation technique used to diagnose and treat neurological disorders [1]-[2]. A key
challenge involved with TMS is to determine stimulation strength in critical regions of the brain accurately. The induced electric field is affected by complexity and heterogeneity of the
brain and numerical computation methods like finite element analysis (FEA) can be used to estimate electric field distribution. However, these methods need exceedingly high
computational resources and are time-consuming [3]-[6]. The purpose of this study was to develop and test a machine learning algorithm that can efficiently predict stimulation strength
based on neuroanatomy. Head models were developed from magnetic resonance images of eleven recruited subjects using SimNIBS pipeline [7]-[8]. We used Sim4Life, an FEA software
to compute induced electric fields due to simulated TMS; the induced electric fields served as training data. A deep convolutional neural network (DCNN) inspired encoder-decoder
network (Fig 1) was developed to predict induced electric fields from T1-weighted and T2-weighted magnetic resonance imaging (MRI) based anatomical slices. Three representative cases
of electric fields predicted by the DCNN algorithm, along with the corresponding reference electric fields are shown in Fig 2. For the trained network, the training data and the testing data
peak signal to noise ratios are 32.83dB and 28.01dB respectively. The key contribution of this study is the ability to predict the induced electric fields in real-time as well as accurately and
efficiently predicting TMS strength in the targeted brain regions saving both time and computational resources.
Acknowledgment: This work was supported by VCU CERSE. Further funding was provided by the CCI CVN (Proposal ID #: FP00010500). 

**References:** 

[1] D. J. Stultz, S. Osburn, T. Burns, S. Pawlowska-Wajswol, and R. Walton, Transcranial magnetic stimulation (TMS) safety with respect to seizures: A literature review,
Neuropsychiatric disease and treatment, vol. 16, p. 2989 (2020). 
[2] M. Kobayashi, and A. Pascual-Leone, 2003. Transcranial magnetic stimulation in neurology, The Lancet Neurology, vol. 2, no. 3, p. 145-156 (2003). 
[3] E. G. Lee, W. Duffy, R. L. Hadimani, M. Waris, W. Siddiqui, F. Islam, M. Rajamani, R. Nathan, and D. C. Jiles, Investigational effect of brain-scalp distance on the efficacy of
transcranial magnetic stimulation treatment in depression, IEEE Transactions on Magnetics, vol. 52, no. 7, p. 1–4 (2016). 
[4] L. J. Crowther, R. L. Hadimani, A. G. Kanthasamy, and D. C. Jiles, Transcranial magnetic stimulation of mouse brain using high-resolution anatomical models, Journal of Applied
Physics, vol. 115, no. 17, p. 17B303 (2014). 
[5] F. Syeda, H. Magsood, E. G. Lee, A. A. El-Gendy, D. C. Jiles, R. L. Hadimani, Effect of anatomical variability in brain on transcranial magnetic stimulation treatment, AIP Advances,
vol. 7, no. 5, p. 056711 (2017). 
[6] F. Syeda, K. Holloway, A. A. El-Gendy, and R. L. Hadimani, Computational analysis of transcranial magnetic stimulation in the presence of deep brain stimulation probes, AIP
Advances, vol. 7, no. 5, p. 056709 (2017). 
[7] N. Mittal, C. Lewis, Y. Cho, C. L. Peterson, and R. L. Hadimani, Effect of fiber tracts and depolarized brain volume on resting Motor thresholds during transcranial magnetic
stimulation, IEEE Transactions on Magnetics (2022). 
[8] N. Mittal, B. Thakkar, C. B. Hodges, C. Lewis, Y. Cho, R. L. Hadimani, and C. L. Peterson, Effect of neuroanatomy on corticomotor excitability during and after transcranial magnetic
stimulation and intermittent theta burst stimulation, Human Brain Mapping (2022). 

![](https://s3.eu-west-1.amazonaws.com/underline.prod/uploads/markdown_image/1/image/0651fafb02beea8bbff852de15f40770.png)

Prediction of Stimulation Strength of Transcranial Magnetic Stimulation in the Brain with Deep Convolutional Neural Network based Encoder Decoder Network

Underline digital video library