Ever since Facebook bought Oculus, it’s been investing in virtual reality in a big way. Beyond just hardware like the Rift and the Go, it’s also putting a lot of money and research into software. It delved into the realm of social last year with Oculus Rooms and Facebook Spaces, but that’s not where Facebook’s VR ambitions stop. At F8, the company unveiled a new way for users to experience VR: by recreating their memories.
At the F8 keynote this week, head of social VR Rachel Franklin showed off an experimental feature where Facebook’s machine learning algorithms could convert a 2D photo or video into a virtual three-dimensional environment through point cloud reconstruction. According to Facebook, the tech uses “photogrammetry” to create 3D space from flat photos or videos. This means that you could conceivably transform a photo of your 10th birthday party into a virtual recreation of your childhood living room.
I sat through a demo of this feature at F8, and it felt a lot like being in a dream. Thanks in part to the point cloud reconstruction mentioned earlier, the virtual room that I was in was not lifelike in the slightest. Instead, it was as if I stepped into a world of pointillistic art, which lent the environment an ethereal, magical quality. When I wave my cursor hand around, the dots that make up the room shift and moves, emphasizing the unreal nature of the world around me.
Scattered throughout the place — which seems to be a recreation of a real Oakland home — were cut-out squares floating in the air. When I hovered over them with a cursor, I saw thumbnails of photos and videos, all of which were supposedly taken in the room that I was in. When I clicked on the thumbnails, I teleported over to them so that I could see the photos and videos up close. One was a photo of a family, while another was a short video clip of a young couple getting ready for prom.
The next environment I was in was yet another living room, and this time all of the photos and videos were taken during a birthday party. The thumbnail near the couch revealed a posed photo of the family, while the one by the dinner table showed the cake. In the very last demo, I was transported over to a pointillist version of the Palace of Fine Arts in San Francisco. This time, the photos taken at the landmark were lined up in a path surrounding the lake.
Facebook told me that this pointillism effect is used to help fill in the gaps, and to “create a sense of presence in the space.” But more than that, the fuzzy nature of the environment also evokes a sense of memory. After all, most people don’t dream in HD. Instead, dreams and memories are misty and hazy, and a pointillistic world where objects are blurry and ill-defined contributes to that feeling immensely. Peppering that environment with real-life photos can feel pretty surreal as well, like you’re hopping in and out of reality. I sometimes felt like I was going through forgotten memories like in Eternal Sunshine of the Spotless Mind.
According to Facebook, the demo I experienced is just a prototype, and it has no plans to share specific implementations. In the future, the company hopes to help people share moments in “new and rich ways.” “We think in the future, people may be able to use technology like this to revisit moments and places that are important to them, with people they care about,” said a Facebook spokesperson in a statement.
In short, think of this as a Facebook album come to life. I can imagine a future where instead of forcing your relatives to sit through a boring vacation slideshow on the TV, you’ll be strapping them to headsets so that they can relive your trip to Italy with you. They still might not appreciate it, but at least it won’t be quite as boring.