I’ve been on a bit of a hiatus lately, getting some things together since I got a job offer to work in AR.

Currently I’m working through a Udemy course on the Unreal engine.

I’m also waiting to see what my schedule looks like to fit in personal projects and possibly projects with other people.


Replacing Reality with Vuforia experiments cont’d

The last post spanned a few sessions; there’s a lot of reading and a bit of head-scratching to be done. I also had another brief session implementing swipe functionality [found forum post with code 1, 2] with rotation; it works on its own but not in Vuforia for some reason. Also in my searching, I found some code on the Vuforia forums that lets the user dynamically switch the 3D virtual model with another 3D virtual model at runtime. That will come in handy later.

This session I read about and played with User-Defined Targets. Great for adding targets at runtime, but the target is a camera image / shot. Might work if you manage to just take a photo of whatever you’re replacing and enable extended tracking. But what about walking around the real object and losing sight of the tracked image? Would the user first have to take multiple images around the object?

There’s a discussion about handling multiple User Defined Targets on the Vuforia forums here. However, the Vuforia Core Sample for User Defined Targets shows that you can use multiple images to display copies of the same augmentation. I took photos face on of the short side, long side, and top. Vuforia can’t recognize the short and long sides when the camera is pointed at the corner where they meet.

For my implementation, we might need multiple images to correspond to different rotations of the same model, depending on how the Persistent Extended Tracking works. Now I’m poking around the Vuforia Core Sample User Defined Target example to see how it’s implemented.


[You know what else is fun? Having to build everything onto my phone every time I need to test something because my webcam has terrible image quality. It’s long enough to read a few sentences of Ready Player One.]

Just tested extended tracking with the sample Vuforia app and a cube the size of the target. I took a photo of the chair parallel to the top. The tracking is a lot better than I thought it would be, although the model still shakes. I can walk about 3 meters away and down a step and the cube is still there in roughly the same spot.

I made a set of virtual walls in Blender and put that into the sample app. Anything “inside” those walls will be entirely virtual since it will be drawn on top of the real object.

[What a wonderful time for my data microUSB to start dying. I know I can build and run over WiFi, but my concern now is making sure my phone is charged before the cord dies, which means no more usage of the phone.]

The challenge juuuust within my means right now is to build an interface like the sample app interface to:

  • capture an image and make it an image target
    • I’m not sure how to do this with the prefab provided. The API examples are in the Vuforia docs here.
    • Will also need to somehow specify the dimensions of the mask/virtual wall box volume, either by typing (need access to virtual keyboard) or by dragging and anchoring the sides of the box.
  • put a model down to “replace” the object that is the image target, i.e. position relative to the mask (the virtual walls)
    • rotate and adjust the position of the object
    • swap out models from a catalogue
  • a menu to do the above things or replace the image
  • more than one object to be masked?

The challenge outside my means right now is to write a shader for the mask/virtual walls:

  • on the outside of the walls, display what’s on the inside of the opposite wall and draw the virtual model on top
  • on the inside of the walls, display an image (a photo or an image generated from the surroundings) of what’s behind the object to be displayed, so that it looks transparent
    • The inside walls matter for things smaller than the mask volume/the real world object
    • This could be a Render Texture displaying a camera view of a larger image of what’s “behind” the object, to account for perspective. This is done like the views through the AR “portals” that are popular right now.
  • on the bottom of the mask, if the object is smaller than the mask, display a photo of the floor or an image generated from the floor



Augmented reality experiments with Vuforia

So something I’ve been wanting to do in AR is try to replace real world objects with virtual ones, like a cup with a goblet or a notebook with a spellbook. I decided to start learning AR with Vuforia (and Unity) because it was the most mentioned after ARKit in my experience, but I plan to learn ARCore as well and play around in OpenCV maybe.

Yesterday I made my first AR project in Unity by taking the default image target and putting a cube on it in various places in respect to the image target. I found out that the app does not render the image target like I expected, so say a cube placed ‘behind’ the image will still be drawn on top of the image though perspective wise it appears ‘behind’ the image plane from some angles. Occlusion is an open problem in AR but I would have still expected that the image target could be rendered since that’s what the app is recognizing/matching.


The cube is placed “behind” the image target in Unity but is rendered in front.

This would still probably be what I want though if I want to ‘replace’ the real world object by drawing on top of it. (I am so grateful for high level tools like Unity right now, instead of having to figure out how to implement something 3D with OpenCV.)

First up I’ll do a very low budget hamfisted attempt at ‘masking’ an image target. I just took two not very precisely positioned photos; a better masking photo would be created from the surroundings by image processing/AI.

The image target is far from optimal but Vuforia still recognises the scene.

The ad hoc solution:

Mask the book with a photo of what’s underneath the book. Unfortunately it doesn’t match the lighting since it’s been 2 hours since I took the photo, and this is distractingly obvious. Better would be to mask with a photo that is copied from the surroundings at the start of the app runtime. With the right image processing tools that’s a pretty common thing to do…I think.

Also, I’ll need to experiment with the 3D/multi image targets to keep the mask going at steep angles like below. (Also, shadows.)

What to try next, masking a 3D object (one with significant depth like a soup can or table) or making an app to choose virtual objects to put on top of the mask? I haven’t done much app making outside of VR games, so let’s take a look at that next time.


So I was looking up other solutions to this, and Pottery Barn has an app (made in ARKit) where you can replace furniture in an existing room…by drawing virtual opaque walls in the room and adding in front of that. Wow. I was expecting something way more sophisticated like the app extrapolating what the actual walls look like, but no. Ok, if that’s the state of things this may be a harder problem than I thought.

Here is a video from 2010 that is pretty much doing what I want to do, albeit with very small objects and uniform surrounding areas. The authors call it “diminished reality”. It is their field of research. It’s the kind of thing I’d need to learn real time image processing with OpenCV for I think.

Capstone: The Path of Love drafting

My capstone project for the Udacity Virtual Reality Developer Nanodegree is basically a personal project, so I made the game about some personal things, like persevering through depression. I wanted to make it a “non-depressing” game while still capturing some of the frustration in the face of positivity that living with this mental illness causes.

But I didn’t want it to be a depressing game, so I made the depression manifest in the game mechanics (analogous to spoon theory) and made the story of the game an uplifting love story.

The Path of Love

The Path of Love is a mobile game aimed at VR beginners. It is primarily a love story between the main character Prydzylia, a sorceress with a curse, and the princess Celeste. The princess has invited the sorceress to visit her at the lake, and the sorceress has to deal with her curse as she makes her way there. The base game is a tile path game like Tsuro, where the player lays down a semi-random path toward two destinations.


You can read more about my planning here on my blog. The biggest cut I made from planning to execution was making immersive scenes for each path destination, such as an apothecary shop interior and a lakeside night scene. I would have had to do a lot of the 3D modelling myself to do so, adding at least another week. It is my biggest regret, since immersiveness is part of the point of VR.

User Testing


The first user test focused on the setting tiles part of the game. The scale and atmosphere came across right (a sunny garden). The tester figured out the controls with a little bit of confusion (they were new to VR). I had originally called the energy it takes to put down tiles ‘spoons’ without explanation, so I had to change that.

The second user test encompassed the start scene as well as the destinations of the game. There was some confusion over what the destinations were so I added text labels, lacking time to make appropriate visual models. Also, the destinations still initialized on top of each other sometimes despite my provisions against that, so I increased the minimum distance separating them. In the end, the start scene came across well as an interior “beginning” scene, and the emotions of frustration mixing with hopeful positivity were clear.

Final Game Walkthrough

The player starts in a magical study, where there is a scroll on a desk. Tinkly mysterious music plays. Clicking on the scroll plays a tinkly sound and reveals a message from Princess Celeste, inviting Prydzylia to meet her at the lake. She says to “conserve your energy” and “go to the doctor”, which are hints to the rest of the game. Clicking on the door transports the player to the main game.


The player starts on a crossroads tile. A vaguely medieval musical theme plays. Looking around, the player locates the two destinations, the doctor and the end (“To Lake Clarica”), which initialize in random locations every game. The player clicks on the blue tiles to generate the next tile in the path, and clicks on the ground of the tile to rotate it.Screenshot_20180409-154024.png

Also on the ground is a note displaying how much ‘energy’ the player has left, to a maximum of 6. A tile takes 2 energy to place. This is Prydzylia’s curse; the number of actions in a given time are severely limited. Energy regenerates a random amount every 5 seconds. If the player tries to place a tile without enough energy to do so, a scroll drops. Inside the scroll is a random encouraging message from Celeste.


Clicking on the Doctor destination plays a shop bell, and a door opening and closing. The maximum energy is increased to 18, but no energy is added. I chose to do this because taking antidepressants takes a while to see any effects, and going to the doctor takes a lot of effort.

Clicking on the purple ball in the end destination (“To Lake Clarica”) plays a victorious tinkly sound and reveals Celeste. Clicking on her reveals a message from her, which is the end of the story.Screenshot_20180409-154425.png

There are two possible endings. The obvious one is to visit the Doctor first and then go to the end. Then when the player sees Celeste, she proposes. The other ending is if the player goes directly to the end. Then, Celeste doesn’t propose but continues to support Prydzylia and dealing with her curse.


It’s bittersweet to realize that despite all the sweat I poured into my final project, I’ll have surpassed my best work even a month from now. This is in many ways My First Game; I can’t expect a prototype to be AAA quality in the (less than) two weeks I had to work on it. I’m still learning to let myself learn.

This project also leaves me with the primary problem of virtual reality content: what makes content effective in virtual reality, and more valuable than in the familiar 3D game on a 2D screen? I think this is a more difficult question for mobile/Cardboard which, while being the most accessible VR, only has one input. I don’t think this particular experiment was successful in answering this question, but not every exploration gives positive results, and not every important result is positive.

Here’s to more experiments in virtual and augmented reality.

Capstone: The Path of Love planning

I did all my planning via my notebook or this blog, which I didn’t want to post until I’d finished my capstone. Here is a consolidation of that information in a document like the one in the Udacity rubric.


The Path of Love is a game that is primarily a love story between the main character Prydzylia, a sorceress with a curse, and the princess Celeste. The story is that the princess invited the sorceress to visit her at the lake, and the sorceress has to deal with the curse as she makes her way there.

Base game: Tiles with maze pieces that you have to put together to build a path toward certain scene(s). 2 views: overall and maze. The overall view was omitted since it wasn’t really necessary; you only build one path in most circumstances.

Iterations: changes in italics

  • additional scenes/islands: Beginning, Middle (a date), Doctor/Apothecary, End (proposal by the sunset)
    • Only had time to make the doctor and end destinations and the start scene
    • didn’t make scenes, only tile islands, and had a stand-in model for Celeste instead of making a whole end scene with terrain, lights, fireflies etc.
  • “encounters” on random tiles
    • didn’t have time
  • love story where some of the encounters are love notes and hints about the scenes
    • I kept this to motivate the love story and the curse
  • one of the scenes ends in a proposal
    • I implemented this very roughly via text
  • Laying down tiles costs “spoons” (energy) and you will get stuck at one point, at which point she will give you more tiles and tell you to seek help. You can choose to ignore her and keep going toward the end, but there won’t be a proposal scene.
    • I only implemented the tiles costing energy to put down, and changed to energy regenerating slowly after a few seconds
    • I put in the branching storyline where if the player goes straight to the end destination, Celeste does not propose.
    • I changed “spoons” to energy since I cut the video that mentions it.
  • Visiting the doctor gives you more energy, enough to lay down tiles to get to 2 scenes, more if you keep going back. (But it takes energy to get there and you have to wait for her to help you.)
    • I made visiting the doctor increase the player’s maximum energy, and didn’t add to the current amount of energy the player had, to mimic how going to the doctor takes energy and antidepressants take time to work.
  • Clicking on the crystal ball in the study plays a video, a ‘note to self’ from Prydzylia, with hints about the game.
    • I originally added this because I misread the achievements and thought I needed to complete all the fundamental achievements including a video player. Since I didn’t need it, and my video editing and acting skills are basic at best, I cut it. I had my costume all planned out though…

Here are the planning pages from my notebook about what I wished to add and some of the text.

Capstone: user test results

I’ll post the two tests so far together. I had the same tester for both, so I should probably get someone else to test the final submission version…

User Test 1

User experience with VR: none

How big are you compared to the scene? Life sized. I felt like i was in a garden?
Can you identify the interactive objects in the scene and what they are for? Figured out when the dot becomes an empty circle, it’s something I can click on. Clicking on an empty blue tile gives me a new piece of ?land. clicking on the pieces can rotate them. Eventually figured out the blue balls floating above actually moves you to the space, totally thought for the first bit that it deletes pieces (since it moves you forward it looks like a piece got taken off). Couldn’t figure out what the Spoon number meant or what the goal of the game is..initially thought I needed to complete a path but then I could keep adding more land
Any glitches/bugs? Just the above moving vs. deleting thing. And not sure if you meant for some of the pieces to not be click/rotatable and some of the floating blue balls to not be clickable. Also there’s a white ball in the sky that I can’t do anything with?
Other comments or suggestions? I assume this is the beginning phase of the project? The goal for the player is not too clear.

Things to change:

Initially the energy it took to place tiles was called ‘spoons’, and I was going to put in a video to describe that. I changed ‘spoons’ to ‘energy’ and put in a scroll hinting at that instead.

I should also change the rotation so that clicking on a tile rotates the tile you clicked on, instead of rotating only the current tile. I had put up colliders so you couldn’t teleport across the walls/”bushes” of the path, but the bushes are not as tall as the waypoints. I could change that. Tester says that would make the walls feel oppressive since the bushes are already pretty big.

The white ball in the sky is the sun, I think, just a bit smaller than expected. I made it bigger.

User Test 2

1. What’s the mood, lighting, and scale of each scene? Mood of the indoor room is very much “start of game” like. I liked having some reading/instructions in a scroll form. Outside is garden like. No issues with scale and lighting.
2. What are the controls of the game? Same as before. I couldn’t figure out what the purple ball did near the house..and sometimes clicking a tile would turn it but other times it would turn the tile next to it.
3. If you get stuck, can you find the restart button? Yes.
4. What things are affected by physics/animation? Mm like the encouragement scrolls would fall to the ground. The sign by the door would flap when tapped.
5. What emotions is the game trying to evoke? I felt frustrated trying to figure out how it worked. But overall the theme was positive and hopeful through all the scrolls.
6. There are two different endings. How do you get those endings? Can’t comment..couldn’t figure it out. I did figure out this time that energy limits your expansion and time refills energy (I think?) Which I didn’t know last time what the spoons meant. But still couldn’t figure out the goal..like I thought maybe I’m supposed to build a path towards the door (but I also thought I must’ve just left from that house since I exited a door..but I come out of the house a few blocks away so maybe I’m supposed to go to the house) ..there were no other objects in sight to clarify the goal. But maybe that’s part of the game. I dunno..
7. Any other comments? The word on the mat starts off as spoon and changes to energy after the first click. Glitch?

Great job btw
It looks pretty cool
me: does it

Yeah I think so
Definitely starting to remind me of..runescape type thing 😛 I hope you don’t take that as an insult haha
But it’s more real game like than before

me: I gotta start somewhere lol


2. Rotation is buggy for some reason. I should probably check that the current tile rotation has been taken out.
5. That was exactly what I was going for, frustration in the face of positivity. It’s not a ‘fun’ game. But how to make that clearer that that is the point? There’s an encouraging scroll comment along those lines, but chance of getting it is 1/20.
6. So the end destination isn’t clear enough. Maybe some more text labels? A lake? ??? I also never thought about that the player might think that house was where they exited from. I’m not sure how to indicate that the house is a different house other than text. (The start tile is a four-way tile, like the door they exit through in the start is a portal and you don’t see it after.)
7. I caught and fixed that, naturally right after I sent the apk to the tester…

I also fixed the issue where sometimes the two destinations spawn on top of each other. Since the vectors of the positions are 3D, I just have to check that the x and z components are greater than half the combined lengths of the destinations. It’s a bit anticlimactic considering what I used to do with matrices.

ETA: Between my tester and my beta readers for my writeup, people had the most trouble with the concept of using waypoints to teleport around. I wonder what makes this more intuitive/clear. Maybe adding a ‘dash’ instead of teleport? I found this at any speed makes me dizzy though. Maybe a different visual? I’m not sure.