Author: Maha Sajid

Lets start with all the times I wasn’t doing research. This week has been much busier in terms of courses, had to complete assigned readings for the classes. Met a few potential group members for the Advanced Topics course, now I have to finalize who to work with exactly. The initial assignment for CS5590 courses was also released, I completed them on my end and the office hours have had greater attendance than the last week. Nothing too crazy though, since the due date is still a little far away. The teaching experience for CS1054 is a little interesting because the students are still learning the basic programming such as variables and data structures so when they do come up with “Why does my code not work?” questions one has to come up with creative ways to answer without directly pointing it out. Also completed some of the many GTA training workshops.

As for research, we basically decided there are two main ways to produce a security threat in the collaboration system through occlusion. First, is to run a background app that takes over the foreground on a trigger or runs simultaneously with the target application to create an occlusion. Second, is to create push notifications. I am currently in the process of profiling alternates to unity game engine for this purpose, figuring out push notifications for magic leap 2 and find potential means of creating system notification through an external trigger as a Wizard of OZ approach.

For my reading this week I have the following paper from the CSCW December 2023. It proposes a symmetric MR collaborative model that allows for object manipulation through virtual replicas. The paper is supposed to have open-source github repo but the given link is not working for now. I will keep checking if it becomes visible since the system looks very useful.

SurfShare: Lightweight Spatially Consistent Physical Surface and
Virtual Replica Sharing with Head-mounted Mixed-Reality

The first week of classes was occupied with a lot of things. As far as the courses go, both Machine Learning and Advanced Topics in HCI had a relatively slow start. I did not have much course work to tend to besides just understanding the flow of the classes in the upcoming semester and setting my expectations accordingly. On the other hand, the two courses( CS 1054: Intro to Programming in Java, CS/ECE 5990: System and Software Security) that I am a GTA for, did demand some time and attention. I had to schedule office hours of up to 7 hours per week and figure out grading duties with the instructor. I haven’t taken the grad course myself so I am required to complete the upcoming assignments for the students myself before I can guide them. I have set up the environment for that but have yet to solve the assignment.

For research, currently the Magic Leap 2 Remote Assist is pretty much finalized for our user study. I had to evaluate a few features of the application to understand its role in the study. Firstly, the sessions had expiration time and I had to confirm what that time is and if it can be altered. The control of expiration time might have helped to involve zero-trust authentication in the study. However, that expiration time has recently been removed in the latest update and now the sessions don’t expire. I tested the app for up to 1 hr 15 min to verify this. I also looked into the possibility of simulating an obfuscation attack in the AR users view on magic leap 2. For this I created a unity code that made a cube appear and disappear at random times in the AR users view. However, this application could not run in the background and override the view of remote assist. I also confirmed the support for interactive 3d objects (glb files) in the remote assist through their assemble functionality. This can potentially act as additional resources for guidance such as anatomy diagrams etc.

Furthermore, I read the following papers to understand the kind of security risks associated with the use of AR in various use cases and brainstorm replication of any such security attack in my own user study.
“Exploring User Reactions and Mental Models Towards Perceptual Manipulation
Attacks in Mixed Reality”
“When the User Is Inside the User Interface:
An Empirical Study of UI Security Properties in Augmented Reality”

“Accuracy of routine external ventricular drain placement following a mixed reality–guided twist-drill craniostomy”. I also read this paper, which is a good example of how the use of AR can contribute to the health-care industry. This particular paper helps locate the drilling point in a skull surgery more accurately.

I looked through possible off-the-shelf AR/VR collaboration solutions available. Most of the ones I found were paid products targeting big businesses to get their demo. I also looked into Microsoft Dynamics products’ demo including co-pilot, guides and remote assist, couldn’t request that with the VT email either. Currently reading another paper on such a collaboration and hoping to finalize a non-depth-kit setup for our experiment this week. We also decided to examine the user perception of an AI led model for medical advice, which in general is a direction I am excited about. As for courses have to compile and analyze results for social computing course, blockchain development is almost complete so we’ll just be left with paper writing in the following week.

 
Little slow on the research this week, looked into the documentation for live feed on depth kit studio. Looking forward to get it running in the unity over the weekend and look into spout documentation to understand how to stream over network.  Apart from that I caught up on my Friday seminar reflections and conducted user studies for the social computing course. Also done with the final paper summaries for blockchain course this week which is a relief.

This week, I mostly worked on the ICAT presentation. I did try to work on the integration of depthkit studio video and unity, but it wasn’t exactly how we wanted it to be during the presentation. The system still needs work which I will be putting in through the following weeks. I hope to get a live stream on the headset soon. Apart from that, I conducted user studies for my social computing course with my group.  

This week research I had more progress on the lit review part than the technical part. I mainly read two more papers to add to our lit review table namely: Using Virtual Replicas to Improve Mixed Reality Remote Collaboration  and Avatars for Teleconsultation: Effects of Avatar Embodiment Techniques on User Perception in 3D Asymmetric Telepresence.  I have added the analysis of these two to the previous table and simultaneously worked with Brendan to finalize columns for the more precise table. Although, the columns are now finalized I have to make that table for all 6 research papers. Will hopefully do it over the weekend. Apart from that I also found a relevant survey paper published 2023 with a very comprehensive review of technology being used for Remote Assistance and Training in MR: A survey of Remote Assistance and Training in MR. The paper analyzed remote collaboration researches along the following 8 dimensions most of which overlap with the literature review table we are trying to make: 

1. Degree of collaboration. 
2. Perspective sharing 
3. MR space symmetry 
4. Time 
5. Input 
6. Output Modality 
7. Visual Display 
8. Application Domain 

Furthermore, on the technical side I made two short visits to 4020 studio, couldn’t login to depth kit studio the first time because I didn’t have credentials. The second time although I logged into the software the previous project that the documentation refers to was not in my local account so couldn’t make any progress.  I have to consult with Cherelle on how to go about setting up the unity live stream. But there have not been much progress on the technical side 

As for courses, my group submitted the mid-term report for social computing this Thursday and is supposed to with user studies in the coming week (using Meta Quest 3 which I just setup and is definitely a device to look forward to ? ). On the other hand, for the blockchain course I have my paper presentation due in the coming week along with 4 paper summaries (not something to look forward to as much) 

This week the weekly meeting was delayed till Thursday so the following task is yet to be completed but essentially I am supposed to organize the information I found in the papers I read last week in such a form that their implementation can be compared with each other and also looked at with the lens of privacy, I have started on the table and hope to complete it before Monday.  For coursework, there had a mid-term presentation for social computing this week and hoping to submit the mid-term report for both courses in the next week. Also, will start the participants recruitment process for the user study in social computing next week. 

This week I looked into the tech stack used by Cornell tech study on remote healthcare through AR/VR devices. While their paper doesn’t reveal exact tools used, the blogs of one of the author does have a diagram giving insights into how they were able to achieve the setup.  
 

I tried dissecting the tech stack to understand what parts can and should we replicate for our own prototype. While the use of depth kit studio stays constant, the MRTK-WebRTC library used to integrate peer-to-peer real-time audio and video communication has been deprecated by Microsoft so an alternate needs to be explored. I am yet to have my hands on the library machine but as far as transmitting the video mesh from depth kit studio to the VR device is concerned that tech stack can also be adapted to an extent since Spout can send real-time video stream between windows applications and NDI aids that transfer through encoding and decoding of video within a LAN. The next problem I searched up on was how to make this truly remote and transfer outside the LAN and came across this implementation of Janus which can also be experimented with. https://janus.conf.meetecho.com/docs/index.html 

Apart from that this week I had a paper presentation for my social computing class which took some time. Blockchain course was not very demanding this week but is about to get super demanding as the mid-term report is due soon and the rest of the course requires us to submit paper summaries of 4 papers every week.   

The goal I had for this week was to get hands-on coding experience with unity in VR. Since I was still in the middle of unity essential’s course at the start of the week, I essentially just skimmed through a couple of basic projects in junior programming course by unity and started setting up the VR headset with the PC. That unfortunately didn’t happen till Friday evening after my research seminar so the progress there has been slow.  I was also assigned a paper on Zero-Trust authentication for this week’s meeting which I could only start until now but hopefully read completely over the weekend. 

I also had a set-back in my social computing course where the instructor raised concerns about the topic of research my group had chosen and presented in our project proposal. This was a point of concern since we were already done with like 6 pages of the report and a new topic meant doing all of the research again. However, after spending another day of research on the works being done to bring short-form content to VR environment, we were able to convince him about the significance of the topic and let us continue with the same topic. 
Since the blockchain report is also due next week. I spent some time brainstorming the possible methodology and system design for our project along with structuring of related works field.  

Overall,  I feel this week despite being swamped hasn’t been that productive. 

This week I again presented two research papers in the weekly Monday meeting on tele-health and privacy of 3D home tours. This led to some discussion on the ICAT project. In regards to that I have started on the unity essentials tutorial. It has been fairly easy to follow right now but the constraint of sitting in the lab is kind of making the progress slow. Nonetheless I am trying to increase my learning pace on it. I skim through the proposal but plan on having a detailed look over the weekend.  

The Social Computing proposal was due yesterday and that took up most of my time this week. It required a lot of reading especially for the introduction and related works section of the paper. I am glad that submission is over but the similar work for Blockchain Technologies is to be done next week.