latlongs and arnold

From here:

http://pedrofe.com/oculus-camera/

Now that VR headsets are being made available to everyone at affordable prices there is an increasing demand for VR content. Developers are struggling since there is no standard to generate this VR content and it is not easy to reach all of the potential viewers.

In order to make VR a success, we need to provide the audience the content they eager to see before they lose interest and excitement in this new media.

Acknowledgements:

Thanks a lot to Borja Morales (@reality3d) for his great help and support.

Offline rendering for VR

There are already a lot of people generating amazing 2D and 3D content using offline renderers. Extending their work with a VR reality camera that outputs pre-rendered VR videos would have the following benefits:

• Leverage assets already created and immediately generate VR experiences.
• Use more complex and realistic scenarios that cannot be achieved with real-time 3D engines.
• Users would only need modest hardware to run the VR content at a high frame ratio and resolution, and all of them will experience the same quality results independently of their hardware.
• Porting it to WebVR and mobile platforms is easy. This increases the potential audience of users that could enjoy the content.

All of these advantages make videos generated with an offline VR camera to be the easier method to distribute VR content among our audience.

Unfortunately, by now, this also has some disadvantages:

• Head tracking is not considered.
• Due to the way the stereoscopic method is used, some artifacts could appear.
• Lack of interactivity.

Oculus VR camera approach

You can get the camera code and binaries from here:

https://github.com/pedrofe/oculus-camera

Implementation

The following is an Oculus VR camera implementation built for the Arnold renderer, a Monte-Carlo unidirectional path-tracer developed by Solid Angle. The implementation is simple, basically two cameras offset by a distance with a specific 360º projection. But there are some important steps to avoid artifacts and to expose to the Developer some convenient parameters that will provide him the flexibility to work in different scenarios.

Here is the full list of attributes for the Oculus Camera:

Mode:

There are four mode options available so that you can get the result that better adapts to your pipeline.

Side by Side

Over Under

Projection:

Depending on the selected projection, and options, each sample will correspond to a ray direction so that all of the space around the camera is completely covered.

 

LatLong Projection

NVIDIA Cube Map

3×2 Cube Map

Here you will find an explanation about the NVIDIA Cube Map format.

Thanks a lot to Mirada Studios CG Supervisor Julian Sarmiento, that helped a lot to define and test the 3×2 Cube Map format.

The 3×2 Cube Map has the advantage of better aspect ratio images and that it has four planes (positive X, Y and Z, and negative X) that are contiguous as you can see in this image:

Eye separation:

Defines the separation between the right and the left camera, needed to achieve a stereoscopic effect.

The camera origin position is updated for each sample and displaced from the center perpendicularly to the ray direction. Here is a picture explaining this.

Doing this per sample level and not per pixel creates a better result than using two physical cameras.

Pole merging:

While this method creates a very nice 3D effects for objects around the viewer, some nasty artifacts will appear at the top and bottom poles. This is because the camera offset makes it impossible for the rays to reach the top and bottom positions as you can see here:

And here you can see an example of how poles look like in this case:

 

We can fix these artifacts by smoothing the camera displacement when rays point upwards or downwards. This solution will remove the stereoscopic effect in the poles, but in practice, it is not very noticeable. In this case, the camera rays will look like this:

The pole merging will generate a result like this:

 

Different settings are provided to control this pole merging so it can be adjusted depending on the scene.

First, top and bottom poles parameters are independent as they could have different requisites in your scene. For example, the pole merging at the floor might need to be very smooth if it has lot of detail, but top merging can be more aggressive if it has a flat color, giving the upper hemisphere a more relevant stereoscopic effect. An aggressive merging at the top will look like this:

Merge Mode:

Additionally, the Oculus VR Camera exposes three parameters to define the merging function. Usually a Cosine function (Cos) will be smoother and less prone to artifacts.

Merge angle:

Defines the angle in degrees from where the merge starts to take effect. The nearer the angle to the pole (90º top or -90º bottom), the bigger stereoscopic effect you will see below it, but the most probable artifacts will appear at the poles.

Below you can see the difference between a start top angle from 0 to 80 using a cos merging function:

 

Merge Exponent:

When using the Cosine (Cos) function of the Merge mode, Merge Exponent will raise the Cosine function to the power defined in this parameter. In practice, it defines how fast the merge will take place in the merge poles interval. Here you can see some example values:

Pole merging Workflow

To maximize the stereoscopic effect in the scene while avoiding poles artifacts, the top and bottom poles can be adjusted independently to better suit the specifics of a scene.

The artifacts at the poles can be of two different kinds:

Distortion artifacts:

The single image you see from an eye has some deformations at the poles. Perpendicular lines do not meet at 90 degrees. These artifacts tend to appear as you increase the Merge Angle. Some examples:

  

Stereoscopic artifacts:

If you increase the Merge Exponent to avoid deformations as a result of a high Merge Angle value, you might find stereoscopic artifacts. Here if you take a look at a single eye image, you might see a different distortion, but when you see both right and left images you will notice a circular wave on the pole. Here is an example:

Right eye / Left eye:

  

The workflow that works for me is the following:

• Increase the Merge Angle until you get a correct amount of stereoscopic on all the area of interest.
• If artifacts have appeared on the poles, increase the Merge Exponent until they are not noticeable.
• If the artifacts are still noticeable, slowly decrease the Merge Angle.

Advantages / Disadvantages of the Oculus VR camera approach

Advantages:

• It works with any scene component  that can be rendered in an offline renderer. Meshes, Hair, Particles, Volumetrics and complex shading networks.
• Easy learning curve to begin to create VR content. Just add the Oculus VR camera to your existing project.
• Modest hardware requisites to reproduce the content. Any platform that can reproduce a video with the required projection is fine to experience the generated content.
• Content easy to distribute either as a video file or via video streaming. It can be reproduced using a dedicated software or app, or with web standards like WebGL and WebVR.

Disadvantages

• Poles: By default, poles will show very evident artifacts. This requires that you adjust the stereoscopic effect for each scene and smooth it near the poles, thus diminishing the stereoscopic effect.
• Tilt: Due to the way the stereoscopic effect is done, tilting your head will destroy the stereoscopic perception.
• Parallax: When you move your head along any axis there is a change in the viewpoint that the offline VR scene can't take into account. This can diminish the immersion of the experience, since we can only react to head rotations.

Example: Mery Video

Making this video was pretty easy. The following was done:

• Add an Oculus camera and adjust it with the correct parameters.
• Fix the scene so that you can look in any direction. Behind the windows some images were added to simulate the outdoor as well as a couple of columns to hide some lighting artifacts that didn't appear in the original scene camera.
• A single frame with the credits was rendered and added back to the final video inside Nuke.

The final video can be seen here:

http://pedrofe.com/rendering-for-oculus-rift-with-arnold/

Derived projects from this video:

• http://threejs.org/examples/#vr_video
• https://github.com/francoislaberge/mary-gearvr

Future developing

Parallax

Just now, translation movements of your head will not have a response in the stereo images you are watching. Creating a method that will simulate this parallax effect when you move your head will have the following advantages:

• More immersive experience.
• Less dizziness.
• Stereoscopic effect will work even if you tilt your head.
• No issues at the poles.
• Eventually, it could require less render time than current method.

Anyway, it will also have some disadvantages:

• Position tracking headset is needed (that is not possible just now in most cell phone devices). But even without it, features as better tilt movement and not artifacts at the poles would be useful.
• A more complex rendering workflow will be needed.
• A custom video player with more hardware power will be required.

Here is a video of my current investigation about this problem:

Giroptic

http://techcrunch.com/2015/12/15/giroptic-is-building-the-gopro-for-virtual-reality/#.6ex3fq:FFpB

A 360' cam to watch, perhaps?

Using Unreal 4 with HTC Vive for steamVR

https://www.unrealengine.com/blog/unreal-engine-4-releases-with-steamvr-support

---copypasta:

Epic Games is shipping official support for Valve's SteamVR with next week's first preview release of Unreal Engine 4.8. Anyone can download Unreal Engine 4 for free, access the source code, and create projects for SteamVR.

Unreal Engine 4 enables developers to consistently deploy high-fidelity, high frame rate experiences across all major VR platforms. SteamVR with Unreal Engine 4 is plug-and-play, and with the full advantage of features such as room-scale VR, input, and Lighthouse, Valve's laser-based tracking solution.

SteamVR is completely integrated into Unreal Engine 4 across Blueprint visual scripting and native code, so projects can be built without dependency on programmer support if desired. 

"Epic is once again taking a leadership position in the technology solution space by embracing the industry-wide momentum toward VR," said Gabe Newell of Valve. "With SteamVR integrated into its engine tech, Epic is making it that much easier for Unreal Engine developers to be successful."

"This technology is incredibly freeing," said Tim Sweeney, founder and CEO of Epic. "There is a magical sense of immersion in walking around a VR space and directly interacting with it."

"Epic's Unreal Engine has helped developers build incredibly realistic 3D games and applications for nearly two decades, and now with integrated SteamVR support for HTC Vive, devs can use the same familiar Unreal Engine to build mind-blowing new VR experiences," said Dan O'Brien, Executive Director for HTC Vive, HTC Corp. "When Unreal Engine 4 devs need to build immersive VR environments, they have the industry's best VR platform and experience at their disposal."

As one of the first studios to work with the SteamVR dev kit, Epic already has a proven track record of ensuring Unreal Engine 4's compatibility with Valve's SDK. Epic's Showdown can already be experienced on SteamVR using the HTC Vive headset. In addition, Dovetail Games' upcoming fishing simulation, Dovetail Games Fishing, is a UE4-powered experience recently demonstrated in Valve's booth during the Game Developer's Conference.

"We work closely with Valve to ensure Unreal Engine 4 is highly optimized for the latest SteamVR features, with particular focus on minimized latency," said Nick Whiting, lead engineer at Epic Games. "Epic is continuously in step with VR upgrades and innovations, so UE4 developers are set up for success in creating awesome experiences right out of the box."

Epic Games Senior Technical Artist and Designer Alan Willard and Epic Games Community Manager Chance Ivey will be on hand to support developers in the first Room-Scale Vive VR Jam on May 9-10 and Room-Scale Vive VR Demo Night on May 11 in Austin, Texas.

For more information on Unreal Engine for VR, visit unrealengine.com/vr.

Developed by Epic Games, the Unreal Engine brings high-fidelity experiences to PC, console, mobile, the Web and VR. Unreal Engine accelerates the creation of games, applications, visualizations and cinematic content. Download Unreal Engine for free at unrealengine.com, and follow @UnrealEngine for updates.

Copypasta: Photogrammetry in THE VANISHING OF ETHAN CARTER

This…

…is not a photograph. It is actual 3D model of a church from our game. Before I tell you all about how we created such photorealistic assets (and, maybe surprisingly, why we are not using them in fully photorealistic way in our game), let us think about what makes it look real.

Look around you – do you see any tiling textures?

You shouldn't, but if you do – look closer. That brick wall or those floor tiles are not, contrary to popular belief, a textbook definition of tiling. Look how some edges are more worn out than others, how some parts seem smoother than others, how dirt and dust settled in certain areas. Some parts may be chipped off, some areas stained, on some parts mold or rust started to settle… Ok, maybe those last ones are not in your fancy neighborhood, but you get the point.

And it's all not random either. If you really wanted it, you could probably make sense of it all. The floor might be more worn out around the front door, or where your chair wheels constantly scrub a patch of the floor, and the outer wall might be darker from the side that gets hit by the rain more often, etc.

You could make sense of it all, but who cares? Your brain usually doesn't – it's real, it's normal, nothing to get excited about. However, your brain does take notice when things are not normal. Like in video games. Even if on the unconscious level, your brain points out to you all those perfectly tiling textures, all those evenly worn-out surfaces, those stains placed in all the wrong places – and whispers in your ear: LOL!

I'm really proud of Bulletstorm — I directed all of environment artist's efforts — but well, sometimes art was just as crazy as the game itself. Take a look at this floor surface, does it seem right to you?

It's not that technical limitations prohibit developers from creating things that feel right, not on modern PCs and consoles anyway. The problem – if we put time and money constraints aside – is that more often than not the graphics artists' brains don't care about the conscious analysis of reality either. We can be as quick as you to point out unrealistic looking asset in other games, but we rarely stop to think about our own work. Need to make a brick wall texture? Basically a bunch of bricks stuck together? Old-looking? Sure thing boss! If we're diligent, we may even glance at a photo or two before we get cracking. But even then, quite often we look but do not see.

In The Vanishing of Ethan Carter, you'll see some of the most realistic environment pieces ever created for a video game. Assets are no longer simplistic approximations of reality – they are reality. But it's not that somehow we have magically rewired our brains to be the ideal reality replicators…

Enter photogrammetry.

With photogrammetry, we no longer create worlds while isolated from the world, surrounded by walls and screens. We get up, go out there and shot photos, lots of photos. And then some. Afterwards, a specialized software — we are usingPhotoscan from Agisoft — looks at these photos, and stares at them until it can finally match every discernible detail from one photo to same exact feature in other photos taken from different angles. This results in a cloud of points in 3d space, representing real world object. From there, the software connects the dots to create a 3d model, and projects pixels from photographs to create a texture.

I'll spare you the details about Lowe's Scale Invariant Feature Transform (SIFT) algorithm, solving for camera intrinsic and extrinsic orientation parameters and other details of what that software does – all that matters is that you feed it withgood photos taken around some object and you get the exact replica of that object, in 3D, in full color, with more detail than you could ever wish for.

Take a look at this slideshow to better understand the process of turning a bunch of photos into an amazing game asset:

You can find extra video in today's post on our Tumblr.

Photogrammetry is incredible. I have been making games for 20 years, I have worked with amazing talented artists on huge AAA blockbusters like Bulletstorm or Gears of War, and you could say I am not easily impressed in the art department. But each new photoscan gets me. So much detail, so many intricacies, but most importantly, all of them just make deep sense. Cracks, stains, erosion – Mother Nature has worked a billion years on some of these assets, it's almost unfair to expect comparable quality from artists who spend no more than few days on similar assets.

Enough convincing, see it for yourself. We have found a pretty new and unique way to let you experience photogrammetry of The Vanishing of Ethan Carter. What follows below is not just static screenshots of our game assets. After you click on the image, a 3D object is loaded, then the texture downloads and voila, you get to experience early glimpse into our game world.

You can orbit, pan and zoom camera around these objects, or just sit back and let the automatic rotation showcase them. I highly recommend to take charge of camera (hold down LMB, rotation direction depends on the position of the mouse pointer in the image), especially after going full screen, to fully experience the detail and quality of these assets.

Speaking of quality – these are raw unprocessed scans, and some of them are actually lower res than what we have in the game.

Twenty six photos covering same boulder from every possible angle, and you get this:

39 photos taken around a cemetery statue, and you get this:

Photogrammetry gets confused by moving subjects. If you want to scan human face, even tiny head/facial muscles movement between photos means errors in scan and lots of manual cleanup and resculpting. But you can get around that issue, with this:

Fake plastic hand is optional. You set up the gear like this:

You fire all cameras and studio flash lights at exact same millisecond and you get this:

You don't need to always cover entire object in a photo, you can shoot in chunks, as long as photos overlap enough. This lets you go bigger – 135 photos (over three gigapixels of data!) and you get this sizable rock formation:

With enough skill you can scan entire locations – over two hundred photos (around five gigapixels) and you get this waterfall location, complete with water frozen in time. Before you ask, no, water in our actual game is not still.

Want to go bigger? How about entire 100 feet of quarry wall? 242 photos (almost six gigapixels!) and you get this:

How about bigger than that? How about the entire valley? This one took probably around one zillion photos. Note that for gameplay reasons we actually discarded half of the valley scan and built the rest ourselves, so here you can only see the scanned half, duplicated.

Any bigger than that and we'd have to shoot from the orbit, so that's where we drew the line.

At this point you might be tempted to think "anyone can snap few pics, dump them into software and wait for finished game level to come out".

Sadly, things are not that simple. For starters, if you are not really skilled at scanning, you often get something like this:

Or, when you are trying to scan a person next to a wall, you get this:

All these dots – wall surface with scanned person's shadow. But actual scanned person is nowhere to be seen. Not really what we were after, unless we wanted to scan a ghost.

It is relatively easy to get into photogrammetry, but it is really difficult to masterit.

Most people know how to snap photos, but very few know this craft really well, and even for those few, a change of mindset is required. You see, most of things that photographers learn is the opposite of what photogrammetry requires.

Shallow depth of field (DOF) looks sweet and helps our minds process depth information on a 2D photo, but photogrammetry absolutely hates it. This ties directly to shutter speed and ISO settings. What is widely accepted by photographers, on pixel level may actually produce enough camera shake or scene/background movement and ISO noise to mess up your scan.

It helps if you can control lighting but again, lighting that works for normal photography might kill your scan instead of improving it. You absolutely need to avoid high contrast lighting – even if you think your camera can record enough dynamic range to capture detail in both highlights and shadows, photogrammetry software will strongly disagree with you. Also, anything that brings out highlights in your photos (usually a desired thing), will yield terrible results.

Everything in your photos should be static, including background and lighting. Imagine how hard it is to get the wind to stop shaking those tree leaves and grass blades, or how hard it is to convince the sun to conveniently hide behind clouds.

When we were taking trips to the mountains, we were met with more than a few raised eyebrows for scoffing at perfect sunny weather and for summoning the clouds. Well, consider this – you're doing Van Damme's epic split (well, almost) trying to balance your feet on some boulder rock formation, and then the sun shows up. You have to stop taking photos, and keep balancing your body in this position for next 30 seconds to 5 minutes waiting for another cloud. Ouch.

I could go on for hours about the challenges of acquiring imagery for photoscanning, but don't be fooled into thinking that it ends there, with "simply" getting perfect photos.

You often need to mask out and process photos in a number of ways. Sometimes you need to manually help align photos when the software gets confused, and then, fingers crossed – software will finally spit out an object. As you have seen in our raw samples above, there might still be errors and gaps that you need to fix manually. Then you either accept okay texture quality or begin long, tricky and tedious work of improving the texture projection and removing the excessive lighting and shadowing information to achieve perfect looking source object.

Job done? Still far from it.

Scanned object will usually weigh between 2 and 20 million triangles. That is the entire game's polygon budget, in a single asset. You need to be really skilled at geometry optimization and retopology to create relatively low poly mesh that will carry over most of original scan geometry fidelity. Same thing with the texture – you need amazingly tight texture coordinates (UVs) to maximize the percentage of used space within the texture.

But even then you will most likely still have something 4-16 times larger than what you'd like. At The Astronauts, we have come up with a compression trick that let's us achieve better compression ratio than industry standard DirectX DXT compression, with very minimal quality degradation. This helps a bit, but we still need to make miracles with texture/level streaming. Thank gods for heaps of memory on modern GPU hardware.

You are probably tired from just reading about how much work it is to get the most out of photogrammetry, and will probably scratch your head when I tell you we're actually not going for full photorealism. Take a look at this shot of a certain old house in our game:

All this work to make true-to-life assets and then we 'break it' with somewhat stylized lighting and postprocessing, and mixing scanned assets with traditionally crafted ones? What's up with that?

We didn't scan all these assets for you to inspect every tiny pixel and marvel at technical excellence of our art pipeline. We went through all the hard work on these assets so that you can stop seeing assets and start seeing the world. Feeling the world. It may look photorealistic to some, it may seem magical to others, but if we did our scanning right, it should above all, feel right.

And the end of the day, photogrammetry is a tool. Nothing less, but nothing more. It's still up to designers and artists to decide what kind of world they are creating, and on what journey they want to invite the players.

In our case, we're making a weird fiction game, a dark tale, and not a documentary — and this needs to be supported by a certain visual style that goes beyond photorealism (and that is sometimes a bit harder to achieve than pure photorealism).

And on that cue, here is a brand new screenshot from the journey that The Vanishing of Ethan Carter offers, an example of our artistic choice. Additionally, here are high quality wallpapers: 1920×1080 (16:9) and 2560×1600 (16:10)

More Photogrammetry info

https://www.garagegames.com/community/blogs/view/22897

----Pasta:

Photogrammetry and Torque 3D: Week 3 -First Results!

by Ron Kapaun · 11/25/2014 (4:34 pm) · 31 comments

Photogrammetry and Torque 3D: Week 3 -First Results!

As many of you already know, I have been experimenting with a different way of producing models for game engines. I started off working with laser scanning and have since moved over to Photogrammetry. What is photogrammetry you might ask. Well, according to wikipedia photogrammetry is the science of taking measurements from photos and extracting exact positions for surface points. Sounds complicated and well, you would be correct in that assumption.

I was first introduced to this method of modeling when I purchased and played 'The Vanishing of Ethan Carter' created by 'The Astronauts'. They are a group of Polish game developers who split from Epic Games Poland aka 'People Can Fly' whom you may know from titles like 'Bulletstorm' and 'Painkiller' as well as ports for Gears of War. I enjoyed many of those games but, I can not ever recall being 'floored' by the visuals.

After playing the game, I started doing what I always do. Researching methods for how I could 'incorporate' more of the real world into my virtual worlds. Like I said, I already own the David Laser scanning system and it does really well for certain things. However, giant boulders and crazy dead trees and large areas of 'nature' are just not suited for scanning (at my price range anyway.)

ENTER PHOTOGRAMMETRY

The system that The Astronauts used to create most of the game assets in TVoEC was Agisoft PhotoScan. This is a neat little piece of software from a developer in Russia. I initially checked it out but, I was a tad 'sticker shocked' by the 'Pro' edition price. So I left and went back to my laser scans. I got involved in a conversation on one of my blogs here and found out I was mistaken. I did not need the 'pro' edition for what I wanted to do. I am not planning on measuring archeology sites or documenting the flow of rivers or any of that other crazy stuff. So I returned and found the Standard Edition for around $180 US Dollars. I tried it before I purchased it using a 30 day trial. I did 2 sets of 'scans' and I was SOLD.

So, I am currently learning Agisoft PhotoScan Standard Edition. You can see more about this nifty piece of software by following this link; www.agisoft.com. The concept is pretty simple, you take a series of photos of an object. Put those photos in the software and magically it produces a 3D model. Sounds simple, alas there is much more to it.

I started out with just my trusty 14 megapixel GE400X camera, a crap load of batteries and a dream. Needless to say, the equipment has grown a tad. I am still using the same camera but, now I drag along a bag with 'markers', tape, a solid tripod, a crap load+ of batteries, my trusty laptop and a bunch of SD cards, tape measure and laser line level and pointer (don't ask).

I had to re-learn photography. The closest I can compare photo-scan photography to would be texture photography. No harsh shadows (I live for cloudy days now), low ISO, no distance blur, no digital zoom, cut back on the Monsters (gotta have steady hands) etc. Then you have to walk around in circles taking photos (I mean that literally.... circles.) Some of the images below took anywhere between 50 and 120 individual shots, and I should have done more on many of them. At 14 megapixels per picture that is a TON of visual data. All those images eat up 16 gigs of ram on my development machine FAST. It's no joke, the software chews up ram like crazy, also I HIGHLY recommend a Geforce video card. They incorporated CUDA into the point cloud analyzing segment of the process and the GPU helps speed up the processing time GREATLY.

So, my first couple of simple experiments went ok. You can see some of the results below. These are all rendered in Blender using the Cycles Renderer.

.
.
.
Not bad but, how the heck do I get these high poly models into Torque? (Some of these are over 1/2 million polys ALONE.) Not only that but, I uninstalled 3D Studio Max and now I am ONLY using Blender. Well, BACK to the research. I have been experimenting and working out a solid workflow in order to make this happen. Alas, I have some results I am happy to show. Keep in mind these are experimental images and I just tossed them into my current build of T3D. The only major things I did to my build was incorporate Verve, the DirectX 9 Refactor, and a bunch of other little 'fixes and tweeks'. For the most part, it's stock T3D.

.
.
.
.
So, using NOTHING but Blender, Meshlab, and a TON of photos I am now able to produce these simple shots. (basically just to test the lighting and performance.) Heck, I even used the default tree just for leaf shadows. The maximum poly count is 2K for one model (that would be that little model with the dead tree growing out of the rock.) The lowest poly count is 250 (the path pieces) All of the models are also HEAVILY LODed. I am pleased so far with the results. It sort of reminds me of Crysis (the original) on Max setting for PC. The textures are still a bit muddy but, that is my workflow, NOT the system. Heck the original textures on the high poly models are actually true 4K images. You can zoom in and count grains of sand on those. So the texture problem is in my workflow. You will note some spots where they look almost 'smeared'. I think that is a bad "Object Cage' when baking the diffuse in Blender.

SO, WHAT IS NEXT?

Like I said, I am not thrilled with the texture quality on the low poly versions of these models. They could be better and once I get my brain more wrapped around Blender's 'cage' system for baking textures I am sure I can improve them even more. I am able to make really nice normal maps from the high poly models and project them onto the low poly model properly so, it has to be something with sample sizes or the cage is not properly aligned.

I started small, little rocks, twigs, ect. I moved up to objects (most are in the screen shots) that are slightly bigger than cars. Therefore, my next step is buildings. I think I have a system planned out that will allow me photo-scan the exterior and the interior of a building and merge them together. I will start with exteriors and interiors alone first, then move on from there. I also intend on mocking up a small interior project for testing with the Rift.

People/Humans, yep I am going to try photo-scanning a person. I am in the process of building a neat little rig that will allow for overlapping photos that can take photos from a set point on a set of outer rings that encircle the subject. This way I can use one camera to take the photo set. (Many large studios use multiple cameras and a geo-dome looking rig but, I am not that wealthy so, gotta make do.)

Basically, I need to build a HUGE library of this stuff. Thankfully, between my laptops, and my desktops I can have machines grinding on this stuff pretty much 24/7 for awhile.


IN CONCLUSION

Overall, I think this system of working will become my new 'norm'. Since the higher quality meshes are faster to produce, look better in the engine (in my opinion) and add the level of realism I have been after, I can't see going back to my old methods. Once I get a solid flexible work flow in Blender outlined, I will post it for everyone to read and review. (I may do a series of video tutorials on this whole process as well...I don't know yet. I am still pretty 'clunky' in Blender but, I should get more efficient as I go).

The last stage will be building a demo level in T3D/UnReal and Unity and video to show off what can be done. More to follow on this grand adventure. Thanks for reading and if you have any questions feel free to ask. I am by no means an expert but, I will try and help.

Ron

Photogrammetry lift

There is a way to get a Vive dev-kit.

And here they're talking about CG made from photogrammetry data, so that you can walk around a room-scale rendering of the photos.

http://steamcommunity.com/steamvr

Here's a link to a Russian photogrammetry app:

http://www.agisoft.com/

-----------------------

I've pasted the article here, just in case it goes....

It's not magic, but...

There's a good chance that you already own a high-end 3D scanner, capable of capturing beautifully lit and textured real-world scenes for virtual reality.

This process known as photogrammetry certainly isn't magic, but it's getting close. It involves using images from a conventional digital camera to create three-dimensional scenes, exploiting the differences in photos taken from different positions and angles. There's plenty that will confuse it - and getting decent source material can be a bit of a dark art - but when everything works properly, the results in room-scale VR can be both compelling and unsettling. Being virtually transported to another, real location - and then walking around in it - still feels like something from the far future. Some of the most immersive VR experiences I've had so far have involved scenes captured in this manner.



This article is aimed as a general introduction to photogrammetry for VR purposes, and as a compilation of hints and tips I've accumulated. This isn't the definitive guide, but it should help get you started and will warn of possible pitfalls along the way. Go forth, and capture the world!

Software

The software I've gained the most experience with is Agisoft PhotoScan, from a small Russian company in Saint Petersburg. While it has more of a GIS background (with plenty of features geared towards surveyors and aerial photography) it has gained quite a following in the more artistic side of computer graphics.

PhotoScan has proved highly configurable and versatile, but it certainly isn't the only software of its type available - alternatives include Autodesk's cloud-based 123D-Catch and Memento. Performed locally, the process requires some pretty chunky hardware for particularly advanced scenes, so expect to throw as much memory, CPU and GPU at it as you can afford. If you're careful, you can limit scene complexity to what your hardware is capable of processing in a reasonable amount of time. Although the eight-core monster at work is very much appreciated...

You may have heard of photogrammetry being used on the recent indie game The Vanishing of Ethan Carter, for instance - where it was used to capture surfaces, props and structures for use in a relatively traditional 3D art workflow. They have posted some excellent general advice online, mainly geared towards capturing individual props - whole scene capture has some differences.

How it works

The general principle behind photogrammetry involves having at least two photographs from different angles of every point you want three-dimensional information for - it will identify visual similarities and, using maths, figure out where these similar points are located in space. This does mean it is limited to static scenes containing opaque, non-specular surfaces - as mentioned, it's not magic.

The software will take your photos and, if there are enough common features between different photos, will automatically calculate all the camera positions. The process is a little like automatically stitching together photos for a panorama, only here you absolutely don't want the camera to stay in one place. Think lots of stereoscopic pairs, although it can work with most potential photo alignments.

Once it's aligned all the cameras (or not) and generated a sparse point cloud, you then get it to produce a dense point cloud - potentially hundreds of millions of points if you've overdone the quality settings. From that you can generate a mesh - with potentially millions of triangles - then get it to create textures for that mesh. (We've been able to render some surprisingly heavy meshes in VR!)

Hardware

I've been using some relatively humble equipment - a Canon EOS 7D digital SLR for most things, usually with the 17-55mm f/2.8 EF-S lens. For low-light stuff a tripod is really helpful, but I've got some surprisingly good results from just hand-held shots. You want as deep a depth a field as possible (f/11 seems a decent compromise between depth of field and sharpness) with as low an ISO as you can achieve (to stop sensor noise from swamping details).

Any decent digital SLR with a sharp lens should work well - for full VR scenes, a wide-angle lens can be incredibly helpful, making it far easier to capture the scene with a reasonable number of photos in a limited amount of time, while reducing the chance you've forgotten to get coverage of some vital area. The wider the lens, the trickier it can be to get a decent calibration, however - a Sigma 10-20mm lens of mine suffers from severe inaccuracy in the corners relative to a typical lens model, while a Canon 10-22mm lens is a fair bit better. The 17-55mm lens mentioned earlier, while a fair bit narrower at the wide end, has all its distortion corrected away - giving great all-round results at the expense of more photos being required. (It's also stabilised, making hand-held shots much easier in lower light or with a lower ISO.) Images from GoPros can be used too, but expect to fiddle around with lens calibration a lot.

I've had some fantastic results from a borrowed 14mm cinema lens on an EOS 5D Mark III - quite implausible hardware for general use, but stupendously wide angle (14mm on full-frame!) and with its minimal distortion being fully corrected away.

The process should work with photos taken with different lenses and/or cameras - for a room-scale scan I can do an overall capture with a super-wide-angle lens to make sure I've got full coverage, before going close-up on individual objects. Be sure to minimise the lens differences to those you actually need - if you have a zoom lens, make sure you're at fixed focal lengths (taping down the zoom ring is extremely useful!) or you'll end up with many different 'lenses' with different, more error-prone calibrations. Switching to manual focus and staying at a particular focus distance can also eliminate focus 'breathing'. (Make sure you switch back again afterwards!)

(A note on the fixed focal length issue: over around an hour and a half I captured a fantastically detailed scene in Iceland with my super-wide-angle lens. Since I was shooting in HDR, the camera was on a tripod, and I was moving the tripod and camera for each set of shots. For whatever reason, the zoom ring on the lens slowly drifted from 10mm to 14mm - the tripod meant I wasn't doing my usual hold-zoom-at-widest-with-my-left-hand trick. I managed to rescue the captured scene by creating an unwholesomely large number of lens calibrations in PhotoScan, evenly distributed across the shots - luckily the scene had enough photogrammetric detail in it to make this acceptable - but it still isn't as high quality as it could have been. So yes, even if you know about the potential problem - tape the zoom ring in place anyway. It could save you from a lot of frustration later.)

Continued in part two...

Photogrammetry in VR - part 2 of 3

21 September - Cargo Cult

Process

Since photogrammetry works on identifying small details across photos, you want shots to be as sharp and as free from sensor noise as possible. While less important for really noisy organic environments (rocky cliffs scan fantastically well, for example), it can be possible to get decent scans of seemingly featureless surfaces like painted walls if the photos are sharp enough. If there's enough detail in the slight roughness of that painted surface, the system will use it. Otherwise, it'll break up and potentially not scan at all - do expect to have to perform varying amounts of cleanup on scans. Specular highlights and reflections will horribly confuse it. I've read of people scanning super-shiny objects like cars by sprinkling chalk dust over surfaces - you won't get good texture data that way, but really good geometry.

Think of topologically simple, chunky scenes to scan. Old buildings, ruins, tree trunks, rocky landscapes, ancient caves and broken concrete all scan brilliantly - while super-fine details, reflective, shiny or featureless surfaces tend to break down. Do experiment to see what works and what doesn't. Scenes that are interesting in VR can be quite small - lots of layers of parallax and close-up detail can look fantastic, while giant scenes (stadiums, huge canyons and the like) can be oddly underwhelming. Think room-scale!

Scanning the scene can best be done in a robotic, capture-everything manner. For a wall, it can involve positioning yourself so the camera direction is perpendicular to the wall, ensuring you have its whole height in the viewfinder. Take a picture, take a step to the side, then repeat until you run out of wall. You can then do the same from a different height, or from different angles - remember those stereoscopic pairs - remembering to get plenty of overlap between images moving to 'new' areas, with plenty of photos taken when going around corners. 'Orbiting' objects of high detail can be useful - make sure to get photos from many angles, with adjacent photos being quite similar in appearance. If anything, aim to capture too much stuff - it's better that than being clever then missing something. Expect a couple of hundred photos for a detailed scene. (It's fine to use 'Medium' or even 'Low' when generating your dense point cloud - super-fine geometry isn't necessarily what you need for a full scene scan. Dense point clouds with too many points also require phenomenal amounts of memory to build meshes from...)



Most of the work I've done so far has involved low dynamic range imagery (which makes hand-held photography possible) - it's still possible to capture full lighting from a scene with careful manual exposure settings. Shooting RAW and then dragging the shadows and highlights into view in Lightroom can help a lot here - it's not quite HDR, but it helps. Being able to fix chromatic aberration in software is also extremely useful, removing instances of those separated colours which look so awful in texture maps. Export full-resolution JPEGs for processing in the software, making sure they're all of the same alignment (portrait or landscape) - otherwise it may decide they're from different cameras. Don't crop, distort or do anything else too drastic. One trick to use is to pull the highlights and shadows in for a set of images to be used for photogrammetric purposes, then re-export with more natural lighting for the texture generation stage.



A full high dynamic range workflow is possible in AgiSoft PhotoScan. Capture bracketed shots on your camera, then merge them to OpenEXR - you can later have the software export textures to OpenEXR for tonemapping at runtime to give full, real-world HDR lighting. You do need a tripod, and there's a lot more data involved - but you'll potentially get useful-for-photogrammetry detail from both shadows and highlights. For one scene I captured, geometry was generated from HDR images and then textures from carefully manicured TIFFs from Lightroom. Look into Nurulize's work relating to HDR photogrammetry capture here.

If you have something which appears in photos when it shouldn't (for example cars, walking pedestrians, bits of Mars rover, astronauts) it is possible to use image masks to eliminate them from consideration. Unwanted features such as lens flares and raindrops on the lens can also be removed in this manner.


Source imagery: NASA / JPL-Caltech / MSSS

Cleanup

One almost-unknown feature in Agisoft PhotoSCan that's proved incredibly useful is its ability to project photos on to another mesh. I've cleaned up a number of rough architectural scans by essentially tracing around the noisy geometry in a modelling program, then reimporting the super-clean mesh into PhotoScan as an OBJ. Since I kept the scale and alignment the same, I could get it to reproject the photos on to my new UVs - resulting in low-poly, super-clean models preserving all the light and texture present in the photos. (Things which can scan very badly, such as featureless flat surfaces, can be the easiest to model!)

This also gives the opportunity to obtain geometry in different ways. Kinect-based scans, Lidar scans, geometry from CAD models - so long as the scale and shape is appropriate, you'll be able to overlay high-resolution photo-sourced texturing this way. You won't get specular maps without a lot of extra work, but the sensible UVs will make them much easier to produce...



A frustrating thing to occur when scanning a scene is to have a seemingly perfect dense point cloud dissolve away into thin, stringy blobs - or even nothing at all. Office furniture suffers badly from this, as do other thin or detailed surfaces. A workaround can involve generating sections of mesh separately and then performing cleanup - compositing them all together into one mesh for final cleanup.

Natural scenes can really benefit from a simple, quick trick I found - involving putting a sphere or cylinder around the scene, then using PhotoScan's project-to-existing-UVs feature to project imagery on to it. While not necessarily 'correct', the human brain seems to appreciate being fully immersed in colour and detail - an incredibly quick scan I did of some redwoods forest became quite compelling this way. Scanned geometry is limited to the ground and treetrunks in the foreground - everything else is flat and 'fake'. For large-scale terrains, a simple-geometry version in the background can look great, with effectively a photosphere for the sky and more distant landscapes.



Continued in part three...

Photogrammetry in VR - part 3 of 3

21 September - Cargo Cult

Rendering

You've got your enormous meshes and textures - so now what? I've done some experimentation with Unity and it's super-capable of rendering this stuff in a brute-force manner. Make sure your textures are limited to 8192x8192 maximum, your shaders are set to unlit textured and that your meshes are rendering without casting shadows and you should be good. You can generate mesh colliders for each segment of mesh produced, should you want virtual objects to collide with your scanned geometry - it's also possible to set up lighting which roughly matches the 'real' lighting from the scene, with baked cubemaps from the scanned geometry. I have a custom shader for scanned geometry which can receive shadows and render with distance fog and a detail texture - it can look quite effective with virtual objects 'embedded' in the world like this.



Adding ambient audio and a few simple particle effects, moving clouds and similar can make the scene appear dramatically more alive – much more than a static, dead model.

Performance-wise, the brute-force-and-ignorance technique of getting Unity's default systems to render giant meshes with lots of huge textures works surprisingly well on a decent GPU. Something ridiculous like a 1.5 million triangle mesh from PhotoScan with twelve 8k DXT1-compressed textures should run fine on a typical VR-oriented system.

There are much more advanced ways of rendering this stuff should you choose - Amplify Creations are doing some interesting work with virtual texturing in Unity - the performance of which is nicely VR-friendly, we've discovered.

Conclusions

As mentioned earlier, this article is not meant as an exhaustive, step-by-step guide to photogrammetric scene scanning - rather as an introduction and collected set of hints and tips. An important thing to realise is that it's a technology that's easily within reach - needing just a consumer digital camera, some software and a half-decent computer for processing. It's hard to state quite how surreal these scanned scenes can be, where even rough test scans can come perilously close to transporting your brain to another location entirely.

I'm fascinated to see what people will end up scanning. Inaccessible places, disappearing locations, the mundane, the fantastic - all can be visited from the comfort of your own VR headset!

If you have access to the VR demos, you can find this content under SteamVR Demos > Iceland Cinder Cone and SteamVR Demos > Valve Demo Room.

Ga