Author: Maha Sajid

Starting off with GTA work as usual, there has been a lot due lately. For CS 1054 students had a hard time completing Project 3 and my office hours were completely swamped with queries through out the week. I also had to grade the Quiz 3 for this class. CS5590 students also had their project 3 due this week. Although, not as many students needed help with the assignment , grading the assignment did take quite some time. Apart from that I did some more summary grading and am hoping to complete them by this week alongside grading Project 3 for CS1054. Furthermore, I was working on the reflection essay for GTA training course.

For courses, Machine Learning is taking a lot of time to catch up with there is always an assignment due for this course. I submitted the Homework 3 last week and am currently working on Project due this week followed by Homework 4. All in all its a time taking course. For advanced topics in HCI, things are much more smooth with only one last set of reading summaries left and me being over with my discussant presentation, the project is all there is to do now.

For research, there has been a bunch of blockers along the way with the networking issues of occlusion attack, applying transparency shader in run time, tracking run time virtual object etc. These blockers have also led to some interesting changes in my project such as the addition of two unknown attacks running locally in users headsets. Anyway, I have been able to implement all three attacks on SurfShare out of which two run locally on the user’s device. The next step was to port the project to my laptop and create different application packages for each attack condition running locally on the device which led to another et of porting errors. The current status is that, I have resolved those and created final builds on the two available HoloLens devices, just need a final test run before the demo and user studies to begin. Speaking of user studies the IRB has been approved so we will start with the user studies this week. Since we are aiming for only 10 participants in this preliminary study, I hope the data collection phase isn’t very long. I have also update the data collection forms for today.

In other news my Masters Thesis Committee is filled with Dr. McMahan on board and I am hoping to start working on Plan of Study after Eid.

Last week I wasn’t able to do much GTA work mainly because of course work. I found myself under the weather in the middle of the week and ended up missing office hours for CS1054 that I have to make up for this week. The only GTA work I did complete this week was summary grading for CS5590.

For coursework, the preparation for Machine Learning mid-term exam took some time since there was a lot of syllabus to cover. I gave the actual exam on Friday. The implementation presentation or Advanced Topic in HCI was also due this Friday. I was able to meet most of the goals I had in mind for that submission including basic attack implementation, IRB submission and Study + Protocol Design.

For research work, I am still waiting on IRB’s response so that I can update the proposal with a few changes including the modification of study design and addition of a subjective interview question at the end of the study. While I was able to implement the basic security attacks last week I still plan on improving them till IRB replies and the revision is approved. More specifically, I have to test latency with a another user on the headset itself and see if any changes in the intensity or type of latency (reliable/unreliable with packet loss) is required. For occlusion on top of basic object erasure I am working on an attack that keeps the remote object in the scene but obstructs its view through another game object. I have already written a script for it but since the remote object can not be created in unity play mode , I need to test it out on the headset with a partner. I have set time with Sabaat to help me test this attack on Wednesday. I hope to have better insights on this after the actual testing on the headset.

Once again starting with GTA work. The class of CS1054 had a project due on mainly file based reading concepts in Java, it kept me busy in the office hours and through emails. I was also supposed to grade it over the weekend since Project 3 is also out now. For CS5590 did some more paper summary grading, they had another assignment on Ransomware due this week but a few students showed up in the office hours probably because the assignment was pretty straight forward to solve

The courses have started to become more work intensive especially with Machine Learning having a major component due every week. It has its mid on this Friday, followed by a Home Work due next week, followed by a Project submission. We also have the implementation plan presentation due for Advanced Topics in HCI on the same day as the Machine Learning mid.

For research, last week I finally got the SurfShare to work but it did come with its caveats. This week was about working around those and molding my task to fit into the software’s capabilities along with filing for submitting an IRB for the preliminary study. I started the week with trying to execute a simple tower building task using SurfShare by giving the remote objects physical properties (especially) gravity and allowing users to stack blocks on top of each other. Due to a number of reason starting with the handle for extrusion attached as a child, low performance of the tool to detect horizontal surface and high reliance on participants to create the portal flat on the surface. I tweaked the task a bit into recreating a floating structure (no use of gravity) from colored paper cut-outs provided to the participants. In this case both users will be participants with pieces for structure divided among them. For the attacks I have decided to use the Mirror’s latency simulator transport. As for the occlusion attack I added transform gizmos to gameplay such that I could move the remote objects from the server in run time, unfortunately it did not work for networked object. As an alternative I plan on testing occlusion attack by making Server the host and manipulating object in the session through the unity app.
One consistent problem is that I can’t test things alone as soon as I take off one headset the it loses its orientation. I have been asking for people’s help but it does slow things down a little.

The week of Spring Break! No classes.

Starting off with GTA work, didn’t have to keep any office hours this week which was nice. However, a lot of grading came my way. Had to grade paper summaries and Assignment 2 for CS5590. Assignment 2 was an interesting one based on XSS attack. Had to grade Quiz 2 for CS1054 which took longer than I expected mainly because student submit code snippets as text in canvas quiz and their solutions can come in many forms. Dry running each of them and deciding how to grade unexpected errors took most of the time.

For coursework, although there were no classes, the homework for Machine Learning course was due right after spring break. The last few lectures were provided in form of asynchronous recordings that I caught up on during this week and then started on the homework. The homework itself involved a lot of long mathematical calculations so it took its sweet time being completed.

For research work, starting with the good news, I have been able to make Surf Share run completely with both HoloLens 2 and tested out its functionality and performance. I found out that it is capable of creating multiple virtual object out of the same real world object. A user can use the whit handle to change the height of the object making it shorter or taller. It also has audio support. However, there are some limitations. Firstly the portals need to be created on a solid background color and according to my distance the hue difference of the background and the object in question should be a lot to create virtual objects, it works on very bright objects. The 3D representation of the object is basically taking the top view of the real world object and extending it’s height, it can mainly represent complicated objects as blobs. The movement of portal is possible along X-Y axis only. However I believe we can edit the source code to enable Z-axis movement too. I have recorded a demo to present in Weekly Meeting. Understanding these limitations I have come with another potential task for the user study that plan to discuss in the Weekly Meeting. It is not a medical task per say but one can argue for how it can be representative.

Alright, so for GTA Work fewer students came to the office hours this week, probably because of the Spring Break which was a relief. However, I still had to respond to CS5950 students via email for help on the assignment even after the deadline, because not all students were able to submit on time. I also caught up on the GTA training courses and completed grading of Project 1 for CS1054.

The coursework, this week consisted of a few reading summaries and a proof-of-life presentation for the Advanced Topics project. I am still a few lectures behind on Machine Learning course and am hoping to catch up on them over the spring break.

My research work was not too kind on me this week. I did get the second HoloLens 2 to start testing Surf Share so that’s the good part. However, I have been running into a lot of technical issues ever since. I was able to deploy the build on both devices but getting them to communicate was complicated mainly because Surf Share had no configuration documentation and I was initially just going through their code to figure out how their networking had been implemented. Through this exercise I understood that the application is using two types of networking, Web RTC for video sharing using node-dss signaler and Mirror Networking over KCP protocol for object sharing. Since the flow of the application is such that both users need to configure the portal (object) first before video streaming starts, it is essential that the Mirror Network works fine before Web-RTC can be tested. The architecture of the application was also up for interpretation due to no documentation so initially I was assuming the first user to be Host (Server + Client) and the second User to be (Client). However, the IP configuration was not working out this way. Later I found out that the PC is supposed to serve as the server (for Mirror Network only, since the PC doesn’t share video it has its Node DSS Signaler and Web Cam object disabled) and both HoloLens 2 will connect as a client to it. Although, this configuration works fine when I run it in 3 unity instances and the logs are displayed in node-dss terminal too as far as the mere connection of these devices work. When I build them to the HoloLens 2 devices the connection fails. I ended up emailing the creator of Surf Share with my queries and got a positive response in their willingness to help me out with the IP configuration and explain how the networking component has been designed. They have set up a meeting with me in the coming week so I am hopeful that will help us move forward.

Testing out the technology is very important for my study to proceed as the task is dependent on what the tool is capable of. Nonetheless, I started to jot down the structure for my introduction/motivation, I had done that 2 weeks before but at that time I had structured it mainly around Telemedicine. After discussion with Dr. Ji and Dr. David-John I revamped the structure into a general Remote Guidance through Collaborative AR kind of a structure. I plan on having the new structure reviewed again before I start phrasing it out for a paper.

Lastly, I read another interesting paper in the same domain:

Sharing gaze rays for visual target identification tasks in collaborative
augmented reality
Author(s): Austin Erickson1 · Nahal Norouzi1 · Kangsoo Kim1 · Ryan Schubert1 · Jonathan Jules1 · Joseph J. LaViola Jr.1 · Gerd Bruder1 · Gregory F. Welc

Not very far from what we are trying to achieve with our project, this paper focus on the effects of different types of errors on user’s collaborative performance in AR. However, they specifically focus on shared gaze environment where errors introduced directly influences the shared gaze. They also determine the thresholds for error before the user performance deteriorates and compare the user’s subjective assessment of this threshold’s with the objective results.

Starting with GTA work again this week office hours were busy for both courses. For CS5590 the Cross Site Scripting Attack Assignment is due this week so students were frequently asking questions. For CS1054 the students had their Project 1 due which I have to grade before the Spring break and a Quiz today which I had to proctor too. This meant office hours last week were pretty occupied for this course too.

For coursework, Machine Learning lectures are being posted asynchronously as recordings which are going in the back log for me. I have to catch up on the lectures since Home Work 2 is out and we there’s a mid for this course after Spring Break. For Advanced Topics in HCI, I had three readings due this week related to different side channel attacks. So coursework wise this week was a little easier.

For research, I started off the week with the goal to port the Surf Share project to Magic Leap 2 device due to hardware availability. Since the project is in Unity 2019 version as the first step I had to upgrade the project to 2022 version for the porting to be made possible. This required some work with the existing packages. The next thing was to change the project settings to make the project compatible with Magic Leap 2 this require hand tracking enablement and reconfiguration of MRTK profile to Magic Leap default with some other tweaks. After configuring the project, the project could still not be ported since the OpenCV package used in the original package had been deprecated and is no longer supported by the 2022 version. Having reached this point I switched back to HoloLens 2, searched for the availability of a second device in library but their device is occupied. While I still look at other avenues of getting the second device for the project, I started implementing a MVP for the occlusion attack on the single Surf Share user. For now I have implemented a basic occlusion attack, I will probably work on making that more sophisticated and Denial of Service Attack for the next week.

For GTA work of CS5590 I completed XSS Seed lab myself. I noticed students have already started coming into my office hours right after the assignment was released maybe because they struggled with the last one. Whatever, the cause it is good for them to start early and good for me since I hope that will lead to lesser amount of emails a day before submission. Also graded more paper summaries for this class. For Intro to Java, students got their project assignment so that is mainly what has been occupying my office hours.

For courses, Machine Learning classes are async for a while so have to play catchup with that. I haven’t started on writing Friday Seminar reflections yet so that is also something on my list. for Advanced Topics in HCI, got the project proposal out of the way, and been reading some papers about differential privacy. The formal proofs in those papers is not something I have a great grip over but the discussant presentations really helped understand the paper those papers better.

For research, there were 3 different tasks I was focusing on. Starting with Surf Share, I did get access to one HoloLens 2 which is a start, spent one whole day debugging the code playing with project settings and unity versions to finally get the code running on the one device. The surf share however requires another client to connect to go beyond its first interface of drawing the portal. So then I shifted to understanding the networking mechanism of this code and unity in general. After one research on Network Manager and WebRTC I would like to think I understand how the networking is happening but still need another HoloLens2 device to connect to verify it and basically start looking into security threats. So that is the progress on technical end.

The other important progress for this week was the meeting with the medical advisor on the ICAT project Dr. Andy Muelenaer. Hi insights really help understand the motivation of this project, the current status quo in the medical field n telehealth and the true value we can provide with this study. Brendan suggested I should pen these down in the paper format to ease paper writing down the road. I couldn’t exactly find the time to write the paragraph itself but I did create a structure for it so I remember what to write when I get to it. I made sure to read through the motivation sections of the last year’s publications of the targeted venue and cover the major points according to them.

Lastly, continuing my literature review of the project I read the following paper:

That Doesn’t Go There: Attacks on Shared State in Multi-User Augmented Reality Applications – Slocum et al

This paper talks about how AR devices use camera feed input as the key for authentication in multiuser environment. And how a picture of the physical environment can be used to generate the key and create Read and Write attacks in Augmented Reality

This week GTA work was less teaching more grading. Graded Assignment 1 on Buffer Overflow Attack and Paper Summaries for CS5590 and also the first Quiz for CS1054. Also helped with the preparation of Assignment 3 on Ransomware for CS5590. Now I have to attempt and understand Assignment 2 on XSS attacks for the next week before it is released for the students.

For coursework, I submitted the Machine Learning Homework it was mainly theory based but the questions were very well designed with tricks that made you think harder. For Advanced Topics in HCI read a few papers and submitted their paper summaries, some were related to my own research that talked about perceptual manipulation in MR devices. I am currently working on the project proposal for this class with my roommate to be presented tomorrow.

My research work is kind of overlapping with the Advanced Topics class, so while I was explaining the project idea on security threats in collaborative mixed reality systems to my project partner, I got a chance to zoom out from my work and evaluate it at the higher level and plan out its contributions. This led to a fruitful discussion with Brendan on our aim for the study and somewhat formulation of a research question with a user study that seems technically feasible and now has defined experimental conditions and evaluation metrics. All of this although assumes the correct working of SurfShare. Unfortunately, the current unavailability of Microsoft HoloLens 2 has become a blocker in testing this out. The user study design however, led to me brainstorming on possible experimental tasks that are not directly medical in nature but measure important metrics for carrying out successful medical tasks. These metrics include precision, timeliness and accurate understanding of expert instructions.

This week was yet again swarmed with GTA work. Students have their first programming assignment for CS5590 this Thursday which means my office hours and inbox have been super crowded with queries ranging from the installation, setup and the actual concept and implementation of the assignment. I myself, ran into issues while doing this assignment so it was kind of expected. Also, started grading the paper summaries for students for each class. Speaking of paper summaries had to do loads of them for my own courses as well. For CS1054 students had their first quiz which once again called for a lot of pre quiz questions in and out of office hours. But I do enjoy being a part of someone’s first interaction with computer programming and them being fascinated by it.

For my own courses, another person reached out for Advanced Topics in HCI project, they have their own set of unique skills and I can think of ways they can be beneficial too but at the same time justifying more than two students on my project might be hard. I have however, left the ball in their court to decide if they want to extend this project in there area of expertise. Apart from that the Machine Learning Homework is due this week but the syllabus was not covered till yesterday’s class so I still have questions left to attempt before submission.

For research, Brendan had reached out to the authors of SurfShare to get access to their open-source repository. Which they did share with us, so that is good news. We now have a potential alternative to ML Remote Assist that has an AR device on both ends. It also has some other cool features such as duplicating object from the real world and manipulating the virtual replicas that can be of great assistance to our user study. However, I need two HoloLens 2 to run that code. The HoloLens 2 in our lab has been occupied for this week so that has been a blocker in actually testing the surfshare out. Furthermore, in the search of potential security threats while using VR devices in the medical context or otherwise, I read the following paper:

OVRSEEN: Auditing Network Traffic and Privacy Policies in Oculus VR

This paper evaluates the network traffic of applications from the oculus store and analyze the kinds of data collected, the purpose of it and finally how they disclose this in their privacy policies. The way they collect this network data and decrypt it specifically for oculus can also be of benefit to our study. I have the git repository for their code but again the headsets in the lab have been occupied so will hopefully be able to test this out as soon as there is an oculus available for use. The SurfShare will still be higher on priority list though.