Research

Research

Google Scholar Page
Publications in this page: Bold font indicates serving as first or corresponding author.

0 Intelligent Robot

Automated and Regulated Robotic Systems

The rapid development of generative technology opens up possibility for higher level of automation, and artificial intelligence (AI) embodiment in robotic systems is imminent. However, due to the blackbox nature of the generative technology, the generation of the knowledge and workflow scheme is uncontrolled, especially in a dynamic environment and a complex scene. This poses challenges to regulations in safety-demanding applications such as medical scenes. We argue that the unregulated generative processes from AI is fitted for low level end tasks, but intervention in the form of manual or automated regulation should happen post-workflow-generation and pre-robotic-execution.

Development: Python

AutoArchitecture
End-to-end automation with AI [1]
regulated
Regulated workflow [1]


Publications:
  1. Liu, Y., & Armand, M. (2024). A Roadmap Towards Automated and Regulated Robotic Systems. arXiv preprint arXiv:2403.14049.
  2. Liu, Y., Kheradmand, A., & Armand, M. (2023). Toward Process Controlled Medical Robotic System. arXiv preprint arXiv:2308.05809.

1 Sensing, Tracking, and Machine Vision

Sensing and Tracking

Shape Estimation of Deformable Linear Object

Realtime shape estimation of continuum objects and manipulators is essential for developing accurate planning and manipulation paradigms. The existing methods that create dense point clouds from camera images, and/or use distinguishable markers on a deformable body have limitations in realtime tracking of large continuum objects/manipulators and the physical occlusion of markers can often compromise accurate shape estimation. We propose a robust method to estimate the shape of linear deformable objects in realtime using scattered and unordered key points.

Development: Python, Matlab, ROS

cable
Shape estimation of TMS cable [1]


Publications:
  1. Zhang, J., Zhang, Z., Liu, Y., Chen, Y., Kheradmand, A., & Armand, M. (2024). Realtime Robust Shape Estimation of Deformable Linear Object. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).

Virtual-to-real calibration method in head mounted displays

We propose a simple virtual-to-real calibration method based on retroreflective optical markers, that combines an external optical tracker and a built-in accurate tracking system for MR-HMDs. [1] To further simplify the process, we also design a new virtual-to-real calibration method combining the off-the-shelf self-location algorithm in HMDs. [2]

Development: C++, Python, ROS, C#, Unity

arcalibration2
Setup [2]


Publications:
  1. Ai, L., Liu, Y., Armand, M., Kheradmand, A., & Martin-Gomez, A. (2024). On the Fly Robotic-Assisted Medical Instrument Planning and Execution using Mixed Reality. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  2. Ai, L., Liu, Y., Armand, M., & Martin-Gomez, A. (2024). Calibration of augmented reality headset with external tracking system using ax=yb. In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE.

Projection Mapping and Tracking

This work presents two system improvements for robot-assisted TMS (RA-TMS) application. Previous systems have used outside-in tracking method where a stationary external infrared (IR) tracker is used as a reference point to track the head and TMS coil positions. This method is prone to losing track of the coil or the head if the IR camera is blocked by the robotic arm during its motion. To address this issue, we implemented an inside-out tracking method by mounting a portable IR camera on the robot end-effector. PPMD can track the head via an IR tracker, and can project a planned contact point of TMS coil on the head or overlay the underlying brain anatomy in real-time.

Development: C++, Python, ROS, Vtk

tms_ppmd
PPMD and its calibration [1]


Publications:
  1. Liu, Y., Liu, S. J., Sefati, S., Jing, T., Kheradmand, A., & Armand, M. (2022, March). Inside-out tracking and projection mapping for robot-assisted transcranial magnetic stimulation. In Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III (Vol. 11931, pp. 57-70). SPIE.

Markerless Tracking

2 Medical Robotics

Intelligent Medical Robots

Process Controlled Medical Robots

"Medical errors, defined as unintended acts either of omission or commission that cause the failure of medical actions, are the third leading cause of death in the United States. Medical errors can include communication breakdowns, diagnostic errors, poor judgment, and inadequate skills. The application of autonomy and robotics can alleviate some causes of medical errors by improving accuracy and providing means to preciously follow planned procedures. However, for the robotic applications to improve safety, they must maintain constant operating conditions in the presence of disturbances, and provide reliable measurements, evaluation, and control for each state of the procedure. " [1]

Development: C++, ROS

PCRMSDesignCycle
Design cycle [2]
state_machine_v4
Hierarchical finite state machine [1]


Publications:
  1. Liu, Y., Kheradmand, A., & Armand, M. (2023). Toward Process Controlled Medical Robotic System. arXiv preprint arXiv:2308.05809.
  2. Liu, Y., & Armand, M. (2024). Toward Process Controlled Medical Robotic System. arXiv preprint arXiv:2403.14049.

Systems for Robot Assisted Procedures

Robotic Transcranial Magnetic Stimulation

Transcranial Magnetic Stimulation (TMS) is a neurostimulation technique that relies on electromagnetic induction to generate an electric field in the brain, with applications in both research and clinical settings. Achieving an optimal neuro-modulatory effect requires accurate placement and orientation of the TMS coil on the head. The use of a robotic method can enhance accuracy and streamline the procedure for placing the TMS coil.

Development: C++, Python, Matlab, ROS, Unity, Vtk, 3D Slicer

tmsnav_architecture
Architecture [1]
preopplan
Preoperative plan methods [1]
sensors
Human subject studies vs sensors [2]


Publications:
  1. Liu, Y., Zhang, J., Ai, L., Tian, J., Sefati, S., Liu, H., Martin-Gomez, M., Kheradmand, A., & A. Armand (n.d.). An image-guided robotic system for transcranial magnetic stimulation: system development and experimental evaluation. IEEE Robotics and Automation Letters (Accepted).
  2. Liu, Y., Tian, J., Ai, L., Zhang, J., Roberts, D., Liu, H., Kheradmand, A., & A. Armand (n.d.). Robotic tms reveals magnetic leakage affecting stimulation outcomes. Under Review.
  3. Liu, Y., Kheradmand, A., & Armand, M. (2023). Toward Process Controlled Medical Robotic System. arXiv preprint arXiv:2308.05809.
  4. Liu, Y., Zhang, J., She, Z., Kheradmand, A., & Armand, M. (2024). GBEC: Geometry-Based Hand-Eye Calibration. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  5. Zhang, J., Zhang, Z., Liu, Y., Chen, Y., Kheradmand, A., & Armand, M. (2024). Realtime Robust Shape Estimation of Deformable Linear Object. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  6. Ai, L., Liu, Y., Armand, M., Kheradmand, A., & Martin-Gomez, A. (2024). On the Fly Robotic-Assisted Medical Instrument Planning and Execution using Mixed Reality. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  7. Liu, Y., Liu, S. J., Sefati, S., Jing, T., Kheradmand, A., & Armand, M. (2022, March). Inside-out tracking and projection mapping for robot-assisted transcranial magnetic stimulation. In Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III (Vol. 11931, pp. 57-70). SPIE.

Robot-Assisted Femoroplasty

Osteoporotic hip augmentation is the medical intervention to reinforce hip bone and reduce the risks of hip fracture. It is commonly performed on patients who suffer from osteoporosis. The procedure uses bone cement inside hip joint. During cementoplasty, surgeons drill the femur from the entry point on the greater trochanter surface, and inject bone cement to the femuroal neck. Robotic system has the potential to enhance accuracy and improve surgery results.

Development: C++, Python, ROS, Vtk, 3D Slicer

FemurSetup
Setup [1]


Publications:
  1. Liu, Y., Kheradmand, A., & Armand, M. (2023). Toward Process Controlled Medical Robotic System. arXiv preprint arXiv:2308.05809.
  2. Liu, Y., Zhang, J., She, Z., Kheradmand, A., & Armand, M. (2024). GBEC: Geometry-Based Hand-Eye Calibration. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  3. Ai, L., Liu, Y., Armand, M., Kheradmand, A., & Martin-Gomez, A. (2024). On the Fly Robotic-Assisted Medical Instrument Planning and Execution using Mixed Reality. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).

Robot Kinematics and Dynamics

Robot Kinematics

Robot kinematics is the study of movements. The robotic systems rely on good modeling and accurate calibration results. Applications such as medical and surgical robotics are especially accuracy-demanding. We conduct these fundamental studies for better development of reliable and robust systems.

Development: C++, ROS



Realtime Robust Shape Estimation of Deformable Linear Object [1]
gbec
GBEC: Geometry-Based Hand-Eye Calibration [2]


Publications:
  1. Zhang, J., Zhang, Z., Liu, Y., Chen, Y., Kheradmand, A., & Armand, M. (2024). Realtime Robust Shape Estimation of Deformable Linear Object. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  2. Liu, Y., Zhang, J., She, Z., Kheradmand, A., & Armand, M. (2024). GBEC: Geometry-Based Hand-Eye Calibration. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).

3 Medical Augmented Reality

Systems for AR Assisted Procedures

Augmented Reality Assisted Transcranial Magnetic Stimulation

Also see TMS. Mixed Reality can provide immersive visualization with crucial information in medical procedures. TMS, a neuromodulation technique, requires accurate tracking data to produce desired modulation results. In addition, the overlay of anatomy, simulated electromagnetic fields, neuro pathways will aid the operator in decision making during the procedure.

Development: Unity, C#, ROS, C++

ARTMSSystemDiagram
System overview [1]
mrtms
Demonstration [1]


Publications:
  1. Ai, L., Liu, Y., Armand, M., Kheradmand, A., & Martin-Gomez, A. (2024). On the Fly Robotic-Assisted Medical Instrument Planning and Execution using Mixed Reality. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).
  2. Liu, Y., Liu, S. J., Sefati, S., Jing, T., Kheradmand, A., & Armand, M. (2022, March). Inside-out tracking and projection mapping for robot-assisted transcranial magnetic stimulation. In Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III (Vol. 11931, pp. 57-70). SPIE.

Augmented Reality Assisted Femoroplasty

Also see femoroplasty. The capability of Mixed Reality, especially virtual object overlay and focus and context, can help with surgical scene understanding. In addition, in robotic surgery, the robot path planning and execution can use Mixed Reality to promote security and mindful human-machine interaction.

Development: Unity, C#, ROS, C++

mrfem
Demonstration [1]


Publications:
  1. Ai, L., Liu, Y., Armand, M., Kheradmand, A., & Martin-Gomez, A. (2024). On the Fly Robotic-Assisted Medical Instrument Planning and Execution using Mixed Reality. The 2024 IEEE International Conference on Robotics and Automation (ICRA2024).

Augmented Reality Assisted Orbital Floor Reconstrution Surgery

Conventionally, the accuracy of implant placement relies on the surgeon’s expertise. Intraoperative imaging and navigation are rarely used due to their cost and setup times, so erroneous implant positioning is often unrecognized until postoperative imaging. This confers risk to the patient’s eyeball, orbital vasculature, optic nerves, and stereotactic vision. In this work, we develop the workflow and user interface of an Augmented Reality (AR) system to aid surgeons with intraoperative placement of an orbital floor implant and ultimately reduce rates of implant malposition.

Development: C#, Python, Unity

Publications:
  1. Liu, Y., Azimi, E., Davé, N., Qiu, C., Yang, R., & Kazanzides, P. (2021, May). Augmented Reality Assisted Orbital Floor Reconstruction. In 2021 IEEE International Conference on Intelligent Reality (ICIR) (pp. 25-30). IEEE.

4 Medical Imaging

Image Segmentation

SAM Integration on 3D Slicer

The Segment Anything Model (SAM) is a new image segmentation tool trained with the largest segmentation dataset at this time. The model has demonstrated that it can create high-quality masks for image segmentation with good promptability and generalizability. However, the performance of the model on medical images requires further validation. To assist with the development, assessment, and utilization of SAM on medical images, this research introduce Segment Any Medical Model (SAMM), an extension of SAM on 3D Slicer, a widely-used open-source image processing and visualization software that has been extensively used in the medical imaging community.

Development: Python, 3D Slicer


samme3d
SAMME 3D Results [1]
evd2
SAMME architecture [1]


Publications:
  1. Liu, Y., Zhang, J., Diaz-Pinto, A., Li, H., Martin-Gomez, A., Kheradmand, A., & Armand, M. (2024, April). Segment any medical model extended. In Medical Imaging 2024: Image Processing (Vol. 12926, pp. 411-422). SPIE.
  2. Liu, Y., Zhang, J., She, Z., Kheradmand, A., & Armand, M. (2023). SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAM. arXiv preprint arXiv:2304.05622.

Spatial Reconstruction

Spatial Reconstruction for Anatomy Tracking

"A tracking pipeline combining retro- reflective markers and point clouds is then proposed for accurate head tracking. The head surface is reconstructed using depth data for spatial registration, avoiding fixing tracking targets rigidly on the patient’s skull." [1]

evd2
Framework [1]
evd2
Validation [1]


Publications:
  1. Li, H., Yan, W., Liu, D., Qian, L., Yang, Y., Liu, Y., Zhao, Z., Ding, H., & Wang, G. (2024). EVD Surgical Guidance with Retro-Reflective Tool Tracking and Spatial Reconstruction using Head-Mounted Augmented Reality Device. IEEE transactions on visualization and computer graphics (Accepted).

5 Neuroscience

Autokinetic Effect

Autokinetic Effect of Human Subjects

"In natural viewing conditions, the brain can optimally integrate retinal and extraretinal signals to maintain a stable visual perception. These mechanisms, however, may fail in circumstances where extraction of a motion signal is less viable such as impoverished visual scenes. This can result in a phenomenon known as autokinesis in which one may experience apparent motion of a small visual stimulus in an otherwise completely dark environment. In this study, we examined the effect of autokinesis on visual perception of motion in human observers. We used a novel method with optical tracking in which the visual motion was reported manually by the observer. " [2]

Development: C++, ROS, Matlab, Python, 3D Slicer

demo_5min_data demo_5min
Healthy subject response to static stimulus in the dark, 20x fast forwarded.


Publications:
  1. Liu, Y., Tian, J., Martin-Gomez, A., Arshad, Q., Armand, M., & Kheradmand, A. (2024). Autokinesis reveals a threshold for perception of visual motion. Neuroscience.
  2. Liu, Y., Tian, J., Arshad, Q., Armand, M., & Kheradmand, A. Autokinesis reveals a threshold for perception of visual motion. Program No. NANO44.13. 2023 Neuroscience Meeting Planner. Washington, D.C.: Society for Neuroscience, 2023. Online.

6 Past Research

Pulsed Eddy Current Data Analysis for the Characterization of the Second-Layer Discontinuities

The pulsed eddy current (PEC) technique has proven effective in identifying concealed flaws within metallic structures. Typically, specific time-domain features are utilized to analyze PEC data, including peak value, intersection lift-off point, rising point, crossing time, and differential time to peak. This study builds upon prior research that focused on identifying radial cracks originating from fastener holes in the second layer of a two-layer mock-up aircraft structure.

Development: Python, Keras

pulseeddy1
Discontinuities captured by sensors [1]


Publications:
  1. Liu, Y., Liu, S., Liu, H., Mandache, C., & Liu, Z. (2019). Pulsed eddy current data analysis for the characterization of the second-layer discontinuities. Journal of Nondestructive Evaluation, 38(1), 1-8.

Lithium Batteries with Bio-waste Derived Activated Carbon

Lithium-selenium (LieSe) batteries hold significant potential as an energy storage solution, thanks to selenium's cathodic advantages such as high electronic conductivity and volumetric energy density. The pivotal factor in optimizing Se cathodes for high-performance Li-Se batteries lies in the design of porous carbon with an adjustable structure at an affordable cost. This research focuses on creating hierarchically microporous activated carbon (AC) derived from discarded coffee grounds through a carbonization and KOH-activation process.

electrolyte1
electrolyte2
SEM images of waste coffee grounds and the electrolyte [1]


Publications:
  1. Zhao, P., Shiraz, M. H. A., Zhu, H., Liu, Y., Tao, L., & Liu, J. (2019). Hierarchically porous carbon from waste coffee grounds for high-performance Li–Se batteries. Electrochimica Acta, 325, 134931.