Month: March 2024

Last week I wasn’t able to do much GTA work mainly because of course work. I found myself under the weather in the middle of the week and ended up missing office hours for CS1054 that I have to make up for this week. The only GTA work I did complete this week was summary grading for CS5590.

For coursework, the preparation for Machine Learning mid-term exam took some time since there was a lot of syllabus to cover. I gave the actual exam on Friday. The implementation presentation or Advanced Topic in HCI was also due this Friday. I was able to meet most of the goals I had in mind for that submission including basic attack implementation, IRB submission and Study + Protocol Design.

For research work, I am still waiting on IRB’s response so that I can update the proposal with a few changes including the modification of study design and addition of a subjective interview question at the end of the study. While I was able to implement the basic security attacks last week I still plan on improving them till IRB replies and the revision is approved. More specifically, I have to test latency with a another user on the headset itself and see if any changes in the intensity or type of latency (reliable/unreliable with packet loss) is required. For occlusion on top of basic object erasure I am working on an attack that keeps the remote object in the scene but obstructs its view through another game object. I have already written a script for it but since the remote object can not be created in unity play mode , I need to test it out on the headset with a partner. I have set time with Sabaat to help me test this attack on Wednesday. I hope to have better insights on this after the actual testing on the headset.

Once again starting with GTA work. The class of CS1054 had a project due on mainly file based reading concepts in Java, it kept me busy in the office hours and through emails. I was also supposed to grade it over the weekend since Project 3 is also out now. For CS5590 did some more paper summary grading, they had another assignment on Ransomware due this week but a few students showed up in the office hours probably because the assignment was pretty straight forward to solve

The courses have started to become more work intensive especially with Machine Learning having a major component due every week. It has its mid on this Friday, followed by a Home Work due next week, followed by a Project submission. We also have the implementation plan presentation due for Advanced Topics in HCI on the same day as the Machine Learning mid.

For research, last week I finally got the SurfShare to work but it did come with its caveats. This week was about working around those and molding my task to fit into the software’s capabilities along with filing for submitting an IRB for the preliminary study. I started the week with trying to execute a simple tower building task using SurfShare by giving the remote objects physical properties (especially) gravity and allowing users to stack blocks on top of each other. Due to a number of reason starting with the handle for extrusion attached as a child, low performance of the tool to detect horizontal surface and high reliance on participants to create the portal flat on the surface. I tweaked the task a bit into recreating a floating structure (no use of gravity) from colored paper cut-outs provided to the participants. In this case both users will be participants with pieces for structure divided among them. For the attacks I have decided to use the Mirror’s latency simulator transport. As for the occlusion attack I added transform gizmos to gameplay such that I could move the remote objects from the server in run time, unfortunately it did not work for networked object. As an alternative I plan on testing occlusion attack by making Server the host and manipulating object in the session through the unity app.
One consistent problem is that I can’t test things alone as soon as I take off one headset the it loses its orientation. I have been asking for people’s help but it does slow things down a little.

The week of Spring Break! No classes.

Starting off with GTA work, didn’t have to keep any office hours this week which was nice. However, a lot of grading came my way. Had to grade paper summaries and Assignment 2 for CS5590. Assignment 2 was an interesting one based on XSS attack. Had to grade Quiz 2 for CS1054 which took longer than I expected mainly because student submit code snippets as text in canvas quiz and their solutions can come in many forms. Dry running each of them and deciding how to grade unexpected errors took most of the time.

For coursework, although there were no classes, the homework for Machine Learning course was due right after spring break. The last few lectures were provided in form of asynchronous recordings that I caught up on during this week and then started on the homework. The homework itself involved a lot of long mathematical calculations so it took its sweet time being completed.

For research work, starting with the good news, I have been able to make Surf Share run completely with both HoloLens 2 and tested out its functionality and performance. I found out that it is capable of creating multiple virtual object out of the same real world object. A user can use the whit handle to change the height of the object making it shorter or taller. It also has audio support. However, there are some limitations. Firstly the portals need to be created on a solid background color and according to my distance the hue difference of the background and the object in question should be a lot to create virtual objects, it works on very bright objects. The 3D representation of the object is basically taking the top view of the real world object and extending it’s height, it can mainly represent complicated objects as blobs. The movement of portal is possible along X-Y axis only. However I believe we can edit the source code to enable Z-axis movement too. I have recorded a demo to present in Weekly Meeting. Understanding these limitations I have come with another potential task for the user study that plan to discuss in the Weekly Meeting. It is not a medical task per say but one can argue for how it can be representative.

Alright, so for GTA Work fewer students came to the office hours this week, probably because of the Spring Break which was a relief. However, I still had to respond to CS5950 students via email for help on the assignment even after the deadline, because not all students were able to submit on time. I also caught up on the GTA training courses and completed grading of Project 1 for CS1054.

The coursework, this week consisted of a few reading summaries and a proof-of-life presentation for the Advanced Topics project. I am still a few lectures behind on Machine Learning course and am hoping to catch up on them over the spring break.

My research work was not too kind on me this week. I did get the second HoloLens 2 to start testing Surf Share so that’s the good part. However, I have been running into a lot of technical issues ever since. I was able to deploy the build on both devices but getting them to communicate was complicated mainly because Surf Share had no configuration documentation and I was initially just going through their code to figure out how their networking had been implemented. Through this exercise I understood that the application is using two types of networking, Web RTC for video sharing using node-dss signaler and Mirror Networking over KCP protocol for object sharing. Since the flow of the application is such that both users need to configure the portal (object) first before video streaming starts, it is essential that the Mirror Network works fine before Web-RTC can be tested. The architecture of the application was also up for interpretation due to no documentation so initially I was assuming the first user to be Host (Server + Client) and the second User to be (Client). However, the IP configuration was not working out this way. Later I found out that the PC is supposed to serve as the server (for Mirror Network only, since the PC doesn’t share video it has its Node DSS Signaler and Web Cam object disabled) and both HoloLens 2 will connect as a client to it. Although, this configuration works fine when I run it in 3 unity instances and the logs are displayed in node-dss terminal too as far as the mere connection of these devices work. When I build them to the HoloLens 2 devices the connection fails. I ended up emailing the creator of Surf Share with my queries and got a positive response in their willingness to help me out with the IP configuration and explain how the networking component has been designed. They have set up a meeting with me in the coming week so I am hopeful that will help us move forward.

Testing out the technology is very important for my study to proceed as the task is dependent on what the tool is capable of. Nonetheless, I started to jot down the structure for my introduction/motivation, I had done that 2 weeks before but at that time I had structured it mainly around Telemedicine. After discussion with Dr. Ji and Dr. David-John I revamped the structure into a general Remote Guidance through Collaborative AR kind of a structure. I plan on having the new structure reviewed again before I start phrasing it out for a paper.

Lastly, I read another interesting paper in the same domain:

Sharing gaze rays for visual target identification tasks in collaborative
augmented reality
Author(s): Austin Erickson1 · Nahal Norouzi1 · Kangsoo Kim1 · Ryan Schubert1 · Jonathan Jules1 · Joseph J. LaViola Jr.1 · Gerd Bruder1 · Gregory F. Welc

Not very far from what we are trying to achieve with our project, this paper focus on the effects of different types of errors on user’s collaborative performance in AR. However, they specifically focus on shared gaze environment where errors introduced directly influences the shared gaze. They also determine the thresholds for error before the user performance deteriorates and compare the user’s subjective assessment of this threshold’s with the objective results.