Candidacy for Print Screen and Yafo Creative's Digital Art & VR Residency

Motivation

My candidacy for the Print Screen and Yafo Creative's Digital Art & VR Residency aims at carving the physical and economical time required for my thoughts and those of others. For that the traditional techniques up to now have been meditation, religious and non religious retreat and other means relying on technology e.g writing or drawing. I propose to use current technology, i.e. virtual reality (VR), as a mean for greater and deeper introspection. As shown further I have for the last 10 years explored tools for thoughts, from papers to software. The recent democratization of VR, augmented reality (AR) and mixed reality (MR) has pushed me to re-evaluate the proxy of technology to my thoughts. There is so much for to explore and I feel that all those new technologies are radically underused, if not misused, focusing on marketing or entertainment rather than helping us to understand ourselves better.

Consequently my candidature aims at providing a VR installation that will allow to use VR as a tool for thoughts and self exploration.

This very candidature is hosted on my Personal Information Manager (PIM) with its VR interface.

Using VR headset and WebVR enabled browser allows you to enter this experience: meditate then write your ideas and visualize them thanks to in-VR painting.


CV

Fabien Benetou

WebXR specialist

VR and AR software development on the web, VR and AR professional workshops

Professional projects

  • European Parliament, September 2018 - on-going
  • UNICEF Innovation Fund, WebXR Technical Adviser, September 2018 - November 2021
  • Aquinas Training, October 2018
  • CTO for LucidWeb from September 2017 to June 2018
  • VR NDA, June 2018
  • AR NDA, May 2018
  • European Parliament, April 2018
  • AR NDA, January 2018
  • NDA, December 2017
  • NDA, December 2017
  • Aquinas Training, December 2017
  • Aquinas Training, October 2017
  • NDA, July 2017
  • European Parliament, July 2017
  • NDA, March 2017
  • Aquinas Training, February 2017
  • UCB, September 2016
Name
Cimzia for UCB
Url
https://vatelier.benetou.fr/MyDemo/UCB/UCBPreview360s/
Logo
Description
Cimzia, an antibody molecule, invites you to travel through the human body. The installation ordered by UCB was featured for 3 days in a pharmaceutical conference in Belgium in September 2016 using 2 top of the line virtual reality headsets by HTC Vive. The experience was also showcased on A week of Aframe.
CallToAction
Get injected in the human body!
DateStarted
July 2016
DateEnded
September 2016

Process

To maximize value for money it’s important to answer those questions in that order:

  1. What’s my audience?
  2. How are they going to interact with my VR content?
  3. What are the specificity of the subset of audience who will be there?
  4. What information do they expect to gain from it?
  5. What Is currently preventing them from acquiring now ?

Then it’s possible to sketch a scenario on how an immersive experience will help that audience.

Once that’s done then the practical questions can be asked:

  1. Is it feasible?
  2. What’s the deadline?
  3. What asssts and skills are missing?

Finally in term of budget:

  • How much will this cost?

This process is iterative but it’s a good approximation.

Recent publications

Interviews

Guest speaker

Recurring

Upcoming

  • Metaverse, European Court of Auditors, October 2024

Past

Proof of concepts (more than ~150)

Future or on-going VR projects

Previous VR projects

WebXR (VR/AR) explorations in videos

Note that you can walk around using the WASD keys of your keyboard. It also works, obviously, in VR.

VR research

ContainsPersonalYoutubeContentToMigrate


PIMVR

VR interface to thoughts

Current exploration : open scaffolding to spacially and textualy explore interfaces https://git.benetou.fr/utopiah/text-code-xr-engine/

Exploring my triage and annotation process

To dive more in depth see the ~60min interview VR & Philosophy #04: A VR Wikipedia with Fabien Benetou.

I like to take notes. I like to take notes because I have ideas, a lot of them. Most of them are a bit stupid or not that original but, I hope, few of them are quite nice. In order to relax myself I write those ideas down. I also write down notes about events, books, recipes, etc. That helps me to organize my thoughts and, hopefully, have more and better ideas. In order to do that I used :

  • paper notebook, very flexible and mobile but can't search
  • random files, flexible but no synthesis
  • 1 text file with numbered lines to allow for recursively reference previous ideas, quickly gets messy
  • multiple text files, mess
  • paper mindmaps and electronic mindmaps, amazing for synthesis, doesn't scale so well

About 8 years ago I settled for an online wiki which now has about 1475 pages. In fact I have a network of wikis, a lot more than just 1 wiki and some of them are even offline or not publicly accessible. I find it amazing because it lets me create new pages, link back to future and older pages, search, share with others, embed multimedia content including slideshows and even the the latest craze like 360 photos or 3D objects and animations. Unfortunately it is quite intangible. More importantly the more it grows the more I feel the need to make it tangible.

I started to work on a 3D model of it to 3D print but it didn't give any useful result. I started to print pages of it to put on my walls but it doesn't work for 2 obvious reasons :

  1. 800 pages requires a LOT of walls,
  2. multiple pages are updated every day so reprinting isn't sustainable.

Thankfully nearly 2 years ago I tried the Oculus and was blown away. My second reaction after "Wow... that's amazing" was of course how I could apply this to my notes. I started to see what existed and didn't find anything relevant. I then started to make several prototypes with the technology available at the time (cardboard + threejs). Since then both the hardware and software evolved. My understanding also evolved thanks to failed and successful prototypes.

This brought us to, later 2016. The latest prototype then worked with 2 or more HTC Vive networked and used Aframe 0.3.1. It means 2 persons could manipulate a set of notes together, in a shared virtual space.

Here are some visuals :

Graph visualization using D3 (partial)

Relying on d3.force.layout() expected to converge then adapting positions.

Testing aframe-htmlembed-component allowing to embbed relatively simple HTML but also CSS images and SVG. Unfortunately most pages look too complex.

Thanks to its convenience though available on the entire wiki with a dedicated user action (appending ?action=xr to any page URL of this wiki)

Data visualisation concept applied to this wiki

Managing with zones (inspired by VRHackathonBXL External Mind's prototype but also few months earlier my Application.Valve)

Graph visualization using D3 (partial)

See also 3D graph demo + ngraph.forcelayout3d

Initial threejs prototype (gaze only)

Networked Vive prototype

Visual metaphor

Draw on any page of this very wiki with custom brushes (cf save()/loadFromUrl() for persistence )

Hubs explorations

  • sync seletecd PDF to reMarkable
  • organize Github issues
  • laod remote content

Your suggestions?

Your suggestions?

There is still a LOT to do. More precisely :

Status of latest prototype :

Working

  1. grab objects
  2. save state over sessions of elements and code
  3. graph based layout
  4. handle dead textures (with fallback querying to server for live generation)
  5. link traversal
  6. periodic dataset refresh (hourly for partial update and nightly)
  7. networking (using NetworkedAframe or Mozilla Hubs)
  8. loading dataset (e.g JSON file or via API like Github issues)
  9. displaying part of dataset
  10. displaying part of used codebase
  11. replay past session (replay)
  12. display past painting (e.g. homepage)
  13. navigate in the local filesystem https://github.com/Utopiah/vrify
  14. in VR page editing and saving with page re-rendering on request https://vatelier.net/MyDemo/WikiVREditor/
  15. D3 based Observable notebook
  16. apply level of details (using e.g. mflux/aframe-lod) as PoC (with texture issue)
  17. Cytoscape based visualization and graph analysis including headless mode that might be interesting in a worker (e.g. betweeness centrality)
  18. type text and edit text (pinch to move, keyboard to input, optionally virtual keyboard pinching letters)
  19. execute text as code (own JXR, based on AFrame shortcuts and JavaScript)
  20. federation (via ImmersSpace)
  21. desktop or container streaming
  22. executing in container (to expand beyond JavaScript to other languages)
  23. grouping text or code
  24. movable virtual keyboard
  25. WebDAV support

To do

  1. integration (perpetual challenge)
    1. see https://git.benetou.fr/utopiah/text-code-xr-engine/
  2. design a proper UI to improve efficiency (KPI to define)
  3. extending WebDAV support to handling MIME types

Overall see https://git.benetou.fr/utopiah/text-code-xr-engine/issues

What's the VR added value

  • full focus
  • bringing back physicality to cognitive tasks
  • unlimited display space while still being room-scale

References

What it is not

This is also not about managing a desktop like Virtual Desktop or envelopVR . Those are also nice tools to expand a classical computer desktop but here the focus is on personal information management. Not how to start a game, how to watch a video or install a program but rather how to sort your notes about the games, how you felt watching the video and how maybe thinking and writing about both can help you discover what you truly like and eventually become able to make it yourself.

Note that this is not about memorization. I tried memorization and it doesn't work for the kind of creative tasks I'm interested in. In my opinion memorization is great for repetitive tasks required for a known optimal solution already exists. For discovering new solutions I believe understanding the structure of a problem or a set of information is way more efficient., if not the only way. Consequently the goal of this project is not to provide a way to recollect information but rather to organize visually and thanks to that process to discover the underlying structure of information, or at least a structure that allows for efficiently accessing and using that set of information, thanks to manipulation and first person spatial navigation. For memorization see instead my MemoryRecipe that relied an RSS feed to go against the Forgetting curve.

Past work

The dataset

Why a crowd-funding campaign

  • order my thoughts and prototype
  • get feedback from others on what I have done
  • estimate if there is a need
  • estimate if that need can make me focusing on it financially sustainable
  • getting funding to pay for proper visual and interaction design

See also the proper Kickstarter preview. Consider that it might also be a Patreon funding or... maybe no crowd-funding at all. This page is first and foremost a way to organize my thoughts by explaining to others. Asking for funding begs for *perfect* clarify so it makes it a good exercise.

Related wiki pages

See also


Virtual reality references

Own motivation

Own projects

With users

With prototype

Note that I have more than 150 prototypes and PoCs on http://vatelier.net/MyDemo/

See also my VR explorations as short Youtube videos

Without prototype

From personal use

Timeline

  • cognitive science studies during SC02 around 2004
    • during which I read one of my favourite research paper in cognitive science namely "Where brain, body and world collide"
  • AR extension Overlay in late 2009
  • 1st note from at least WithoutNotesFebruary11
  • applied to Oculus in 2013 because... why not
    • "At this time, and given your lack of work status in the US, we have nothing to offer you. Please watch our website for potential openings as we grow."
  • tried Oculus in later 2014 via Hello Bank popup store in Brussels
  • 1st event attended BeVRJanuary2015 then 1st hackathon VRHackatonBrussels2016 part of Portfolio
  • tried to hire someone to make ThreeJS/PmWiki interface work for me via Job4Bitcoin in March 2015
  • tought several workshops in 2015 via OpenTechSchool Brussels and at least 3 in 2016 via VRLab Brussels
  • vAtelier.net started in 2015 motivated by the early 3DVisualization by myself due to the failed hiring for the job
  • GoldmanSachs early 2016 report considered for ManagingMyInvestments
  • vrlab-brussels.info with its numerous workshops since early 2016
    • actually complained 1 year before during BeVRJanuary2015 about the lack of "home-brew" projects!
  • 2nd hackathon, 2nd "win" at VRHackatonUtrecht2016 in June 2016
  • 1st hardware workshop resulting in 3 new designs in June 2016
  • 1st large (~20 persons without previous setup) multiplayer demonstration during CafeNumeriqueBXL in June 2016

Beyond that see VR?

Vive

(:vrview360:)/pub/360/ViveAtHome.jpg(:vrview360end:)

Design & UX

Experience reviews

  • VRTX
    • ✔: conveniently select (edges, faces, etc)
    • ✗: low poly load, no scaling of object
  • DesignSpace
    • ✔: preloaded maps, easily add objects
    • ✗: table outside of the chaperon
  • Fantastic Contraptions
    • ✔: physics to solve puzzles
    • ✗: can't recombine previous part of solutions
  • NVIDIA FunHouse
    • ✔: fluids on virtual HMD, hairs, broken objects
    • ✗: nothing particularly fun
  • SoundStage
    • ✔: node based interface
    • ✗: links between node staying too long
  • UnseenDiplomacy
    • ✔: non Euclidian space, forced starting point with countdown
    • ✗: key elements to solve puzzle outside the chaperone
  • getspace
    • ✔: ?
    • ✗: ?
  • Yours?
    • ✔: ?
    • ✗: ?

See also my Steam reviews.

To explore

See also

ContainsPersonalYoutubeContentToMigrate


Thank you

Thank you first for organizing Print Screen and Yafo Creative's Digital Art & VR Residency. I believe the event itself will help to explore the boundaries of what VR can allow and how we can interact with the world around us.

Secondly thank you for taking the time to read my imperfect and lenghty canditure. It forced me to a certain extent to clarify my thoughts, which is precisely what I would like to offer to others in a more comprehensible fashion.

A special thank you to Itay Mayerovits for inviting me to apply and believing in what I'm trying to make and share with others.

Feel free to contact me for any question you would have.