20240109 Inami Lab LT

• Nice to meet you, hello! 🙌

• 

• Bluemo @blu3mo / Shutaro Aoyama

  • I was under the care of Professor Inami during the “未踏2020” program.
  • I am a sophomore at Columbia University, studying Computer Science and Philosophy.
  • I had my coming-of-age ceremony the day before yesterday.
    • 

• What will I talk about?

  • Introducing specific development and research projects.
  • Connecting them with abstract stories.

• Why?

  • I want to receive feedback.
    • It’s okay to interrupt me during the discussion.
  • I would be happy if the members of Inami Lab remember me.

• It is closer to a casual format than a research presentation, but I would be delighted if you find it interesting.

• 2020 IPA未踏 (Inami PM)

  • Online lecture environment balancing real-time and on-demand advantages.
    • 
    • Advantages of “real-time”: student interaction.
    • Advantages of “on-demand”: optimizing speed through time control.
  • Key point: “Elastic Synchronization” that distorts time imperceptibly.
    • 

• Columbia CGUI Lab (Prof. Steven Feiner)

  • An interface where moving objects in a VR space indirectly control factory robots.
  • Paper to be presented at IEEE VR 2024.
  • Video: https://drive.google.com/file/d/1GTalVXEei-yV65Anmlx_149NEIdSOjbw/view?usp=sharing
  • Key point: Asynchronous Human-Robot Interaction.
    • The robot executes actions in VR with a delay after the user in the VR world.
    • Misalignment between “VR present” and “real-world present.”
    • Two proposed interfaces for control speed, preferences, etc.

• Hobby Development

  • 
  • Side story: I created this project as an escape just before the final exam of a relativity class.
    • 
    • Please don’t ask about the exam results.
  • GitHub: https://github.com/Godlysk/SRVR

• 

  • “Monitoring your past self as if they were someone else.”
  • “Being monitored by your future self as if they were someone else.”
  • Q: How far apart in time makes someone else?
  • Demo: https://time-mirror.vercel.app/

• ↑↑↑ Specific development and research discussions

• ↓↓↓ Abstract discussions

• Something I have been thinking about for the past two years: By considering “virtual time,” Removing the constraints of physical time

  • Western modern time concept
    • Train schedules, school timetables, factory schedules.
    • “Appointments,” “Deadlines.”
    • Synchronous economic transactions
      • Synchronized to microseconds.

    • “This socially constructed artificial time has come to encompass everything, and as a result, it seems as if it is time itself, as if there were no other time.”

      • Time is thought to be “one-dimensional,” “objectively shared globally.”
  • In the reality shared through computers (Computer-mediated reality), it is possible to remove the assumption that “everyone shares the same structured time.”
  • Without that assumption, various structures of time are conceivable.
    • Circular time, branching and merging time, etc.
    • Sharing time with others in the past => Kineto (online classes).
    • Sharing time with your past self => Task delivery to your future self.
    • Structures of time and space with discrepancies => Relativity theory VR.
    • Sharing reality asynchronously between humans and robots => Factory robot control research.
  • => Removing the constraints of physical time- Recent Insight: This discussion can be expanded beyond the “structure of time.”
  • The timeline is just one dimension for understanding reality.
  • In general, it is possible to consider a mechanism where subjective elements of various realities such as time, space, language, personality, sensory information, etc., are not shared by everyone, creating an “objective reality.”

• Questioning the dichotomy of Subjective <----------------------------------> Objective

  • Far left (Subjective Reality): Individual VR games,
    • Everyone shares different realities.
    • The red pill from The Matrix (individually optimized simulated world).
  • Far right (Objective Reality): Traditional reality, Cluster, The blue pill from The Matrix
    • Everyone shares the same reality.
    • The blue pill from The Matrix (objective reality).
  • Can we consider something between these two?
    • Asymmetric Reality

• For example, a VR restaurant

  • You see a Chinese restaurant, while your companion sees a French one.
  • If you can hear each other and see avatars, the experience of “dining together” can be shared.

• For example, space

  • For instance, considering a scenario where multiple people collaborate in a VR space.
    • The room contains building blocks and monitors.
    • The top image represents the original reality.
      • However, the red/blue/green individuals see slightly different, individually customized realities.
      • The world seen by the red person (bottom left):
        • Avatars and monitors of others are placed in different positions from the original.
        • Customization intent: wanted to rearrange for better monitor visibility.
      • The world seen by the blue person (bottom center):
        • Objects on the table are larger than in the original.
        • Customization intent: wanted to see details of the objects.
      • The world seen by the green person (bottom right):
        • Avatars and monitors of others are transparent.
        • Customization intent: wanted to focus on the objects on the table.
    • Thus, even in collaborative spaces, in VR, one can experience different realities customized individually.
    • However, such individual customization of reality can lead to communication discrepancies due to “misalignment” in reality.
      • Issues like “an object I see is not visible to the other person” or “an object I’m pointing at appears in a different position for the other person” can arise.
    • To resolve this: adjust other parts of reality to absorb the “misalignment” effectively.
      • If the object you are pointing at appears in a different position for the other person, display the pointing finger at a different position for them.
  • Demo of a 2D space: https://asym2d.vercel.app/

• Implementation:

  • Sharing a Set of Various Situations’ Necessities Instead of Sharing a Lump of Objective Facts in Asymmetric Reality
    • Traditional objective VR space
      • The server has a shared database of spatial positions (x, y, z) of various items.
        • Table: (0,1,0), apple: (0,2,0), User A’s avatar: (1,1,1), User B’s avatar: (-1,1,1), and so on.
        • Data represents the one correct reality.
        • Data is complete = contains all information about the world.
      • Each user’s client constructs the same reality based on this shared database.
    • Asymmetric Reality that shares a “set of necessities of various situations”
      • The server has a shared database of constraints on spatial positions (x, y, z) of various items.
        • Constraints vary based on the situation or context.
        • Data is incomplete = multiple possible worlds that satisfy the constraints exist.
      • Example: When shaking hands: Shared Constraints | Constraint | Importance | | — | — | | Relative Angle of User A and B’s Gaze | 90% | | Relative Position of Avatars of User A and B | 70% | | Relative Position of Avatars and Room Interior | 30% | | Size and Height of the Room | 10% | Individual Local Constraints | User | Constraint | | — | — | | A | Due to the small room, wants to display compactly | | B | Due to the spacious room, wants to utilize space to display information widely |

• Each user’s client constructs a different reality based on shared constraints and individual local constraints.

• Why does “misalignment” occur?

  • Balancing individual optimization and shared reality
    • Examples of common structures in previous discussions
    • Examples of individual optimization: restaurant meal preferences, playback position and speed of classes, VR content display
    • Examples of shared reality: eating together, synchronous communication in classes, collaborative VR work
  • Alternatively
    • When multiple people share reality, there are things they “truly want to share” and things they may not
    • By creating a certain “misalignment,” it can absorb another “misalignment”
    • Example: Real-time simultaneous interpretation between Japanese and English speakers. There may be a “misalignment” in the auditory information, but the essential shared content can still be communicated

• Research

  • Q: How can we differentiate what to share and what not to share?
    • Creating “misalignments” based on context, situation, and local constraints
  • Q: How to share what we want to share while maintaining beneficial “misalignments”?

• To generalize this discussion a bit more:

I thought we should aim for “versatility of constraints” rather than “freedom from constraints.” People might struggle if there are infinite choices without constraints, so having the ability to control constraints on choices from a meta perspective would be valuable (Seems applicable to sports rules, smartphone screen time, VR gravity, etc.) @(blu3mo) October 22, 2023

• Thank you!
  • Twitter: @blu3mo https://twitter.com/blu3mo
  • Facebook: Shutaro Aoyama https://www.facebook.com/aoyama.shutaro.77
• I am in Tokyo until 1/14 and would like to meet people. I would like to go for tea/meal, etc.
• Looking for an internship next summer (May-August)
  • Interested in research and experimental development