AR Screen Projection

Github

The immersion offered by VR headsets is difficult to replicate with a flat screen, but I had this idea to add a new level of interaction with my monitor. I have come back to this project a few times over the years, but I am very happy with how the newest version has turned out.

Using Unreal Engine 5’s VR integration and an HTC Vive tracker + base station set up, I was able to create a system for using the tracker to give a sense of perspective to what the player is seeing on the screen. I hope to create some immersive experiences soon, but for now, I am happy with bringing the concept to life.

But how does it work?

Picture 2 screens. One of them is your monitor in the real world, and the other is a virtual screen in a 3D environment. In order to simulate viewing the virtual world on the real screen, we must do two things.

  1. We must track the real-world location of the viewer, and mimic that movement in the virtual world

  2. We must capture what the virtual viewer sees through the virtual screen. This is what is then shown on the real-world screen to the real-world viewer

The first step is simple for us. Many solutions have already been developed. For this case, I used an HTC Vive Tracker to determine my real-world location. That data was sent to the game engine and used to transform a virtual camera to the corresponding location in the virtual world.

The second step is where it gets complicated. Creating a view like this requires the use of off-axis projection. In general, both real and virtual cameras view the world through a right pyramid whose apex is at the camera. Since it is a right pyramid, the apex is in line with the center of the base, meaning the camera sees directly in front of it. Off-axis projection breaks this alignment, allowing the camera to see through a non-right pyramid, whose base is shifted in some direction perpendicular to the camera’s forward vector. By creating a pyramid whose edges intersect the corners of the virtual screen, we can construct a view frustum which lines up exactly with that virtual screen. This means that the view through the screen from the camera is projected onto the real-world screen.