My First Experience with Photogrammetry

Now that we'd been set the brief, we began lectures about photogrammetry in the classroom.

Our tutor showed us example meshes generated by previous years and some personal ones he had produced himself. The process was described to me as fairly simplistic: take a wide range of photographs of your subject in a 360 degree view (ensuring you've covered all possible angles), feed them into the software and voila! A 3D mesh built from the photos! The important guide we were given, is that each image in a sequence needed to match 70% of the previous image as you work your way around the object.

The software we were using in class is called Meshroom, a free executable file from AliceVision (located here). This software has several positives straight off the bat: it's free, it simply runs an executable file so no installation is needed and, in terms of it's UI at least, it's very user friendly! You simply drag the photographs of your object into the designed space on the left, click the green 'Start' button at the top of the viewer and off you go! I've included a screenshot of the UI below so you can see what I mean:

Meshroom User Interface

Like I said, it all seemed very straight forward (provided your object photographs were suitable). But as we know in this industry, that's very rarely the case when it comes to using it yourself, so I'm curious to see how I'll get on with this new application.

---

First Attempts with Meshroom

Now I came to trying out photogrammetry for myself! We were set the task of finding suitable objects outside of college, photographing them as instructed and feeding them into Meshroom to produce high quality 3D models.

I have several shelves worth of videogame memorabilia; collectible statues, helmets etc. So decided to use a few of those. Using my prior knowledge of photography and lighting from my previous projects, I also decided to perform some additional setup for the objects rather than simply placing them on the floor and walking around them taking pictures.

My initial thoughts were that by using something like a greenscreen, I could eliminate unnecessary background geometry being generated and save myself some time on cleaning the meshes up. Additionally I also placed the object on a turntable so that I could keep the camera stationary and simply rotate the object rather than the other way around. Lastly, I used some white studio lamps I had to create clear flat lighting on both the sides and top of the object to avoid shadows being recorded. All in all, I was confident that my first attempt at photogrammetry would go off without a hitch because of this additional effort... Unfortunately I was very wrong.

Before discussing why this ultimately failed, I first want to compile the photos I took here so you can see what the Meshroom executable had to work with. I've therefore included front, back and side images of my Fallout Brotherhood of Steel helmet that I used, as well as a screenshot of the full collection of images below:

Helmet Front View

Helmet Left View

Helmet Back View

Helmet Right View

Helmet Full Image Gallery

I think you'll agree these images fit the initial brief; they're high quality (4k resolution) with clear lighting and covering a wide variety of angles. So like I said, when I took these into Meshroom I expected some good results! Unfortunately it somewhat fell at the first hurdle.

To elaborate a bit more on Meshroom itself, it runs through several stages as it processes your images; detecting the motion of the camera is a crucial one of these. This is where Meshroom uses the background of the object to detect the motion path and angle you, and therefore the camera, took around the object while photographing it. In this case, because the background was a flat shade of green and the camera was stationary, Meshroom could detect no motion whatsoever during this stage and the process ultimately failed.

Of course I know now this is the reason why, but at the time I made the uneducated assumption that it was because of the item itself. The helmet is rather one dimensional, having essentially the same colour and texture for 90% of it, so I figured this may have been the cause. I therefore tried the same approach with a different object: my Crash Bandicoot collectible statue. I've included a gallery of the images I took of this below in the same format as the helmet:

Statue Front View

Statue Left View

Statue Back View

Statue Right View

Statue Full Image Gallery

Surprisingly, this set of images did actually make it past the 'Structure from Motion' phase, but unfortunately did also ultimately fail at producing the results I wanted & expected.

Essentially, like I discussed above with the helmet, because the background was completely stationary, Meshroom was under the impression that these were all different objects being photographed from one angle. This resulted in a mesh that was a mash of 30+ statues from all different angles into one oval of geometry. I've included a screenshot of the mesh below so you can see what I mean:

Failed Statue Mesh

So yes, as you can see this is unfortunately not what I was hoping for. However, stubborn as I am, I was convinced that the quality of the images and framing of the object was not the issue. I therefore decided to do some researching into photogrammetry (probably what I should have done in the first place) to see if others had encountered similar issues.

I discovered that others had used a similar approach to myself when approaching photogrammetry (studio lights, a turntable and a greenscreen) and likewise had the same disappointing result. A few did mention however that by masking out the greenscreen in photo editing software, they had in fact managed to create some fairly decent results! I therefore took to photoshop and painstakingly masked out the greenscreen on the 41 images I'd taken of the statue. I've once again included a gallery below so you can see the result:

Masked Statue Front View

Masked Statue Left View

Masked Statue Back View

Masked Statue Right View

Masked Statue Full Gallery

While this absolutely did produce better results, with Meshroom this time detecting motion around the object, the final mesh was unfortunately still not up to the quality I expected.

Because the greenscreen naturally had some shadows on it, I was unable to fully mask it out without manually erasing around the statue for every single image, which resulted in hovering geometry. Additionally, for whatever reason, the geometry of the mesh ended up 'bubbly' in appearance, and large chunks of the statue were missing (most obviously a large chunk of it's arm). I've included an image of the final result below:

Failed Statue Mesh with Masking

I'll admit I was disappointed with this result, and was left wondering what exactly I'd done wrong. I decided to wait until our next lecture and discuss this with my tutor. He explained that while on first impressions of photogrammetry everything I did made sense, it's actually the wrong approach entirely. We observed the work of other students, who had far more successful models being generated simply by photographing an object outside on a cloudy day! This goes against what I thought I knew about photography, but my tutor helped me to understand why in points which I'll discuss in my initial evaluation below.

---

Initial Evaluation

The Captures

1. Despite what you would believe, the use of high intensity lighting is actually detrimental for photogrammetry. The ideal environment for an object is outdoors on a cloudy day so the shadows are completely flat.

2. Greenscreens cause more problems than they solve in a photogrammetry project. Having a one note background will result in issues with the software as this is what the chosen application will use to determine motion around the object.

3. Turntables are also a no for the same reason as the greenscreen. Ideally you need the background to stand out from the object you're capturing, and also feature enough variation in the shots so that the application can detect you moving around it. Rather than rotating the object and keeping the camera stationary, you'll have far better results keeping the object stationary and rotating around it yourself.

Meshroom

Meshroom is incredibly intuitive for what appears to be such a simple application on the surface. It has several pros: it's free, it's an executable file so requires no installation and it's very simple to use with your input only really needed for adding photos and clicking start.

At present, I have no cons to discuss, as I have no comparable to put alongside it. However I'm sure this will change in the future as I plan to experiment with other photogrammetry applications.

---

Scanning Strategy

As I will be capturing subjects for my final scans during my Winspit Quarry trip in the next few days (more on that in a future post), now seems like the optimum time to discuss my scanning strategies for the objects.

I know from the experience garnered here a few key details: the lighting of the object needs to be flat, a cloudy day with the subject outside is ideal conditions and that a full 360 degree view of the object needs to be captured from multiple elevations (with each image containing 70% of the view of the previous one). With these points in mind, I've produced a visual aide below demonstrating what my approach will be when it comes to capturing my subjects:

Visual Representation of Capture Strategy

Now I'd like to just breakdown what's shown in the image above:

• The cloud borders are obviously to emphasize that the ideal capture lighting conditions are; a grey cloudy day with even lighting.

• The rock drawing in the image is a placeholder for the subject (as given that our captures will be made at a quarry site, I'd image the majority will be rocks).

• The outer (green) ring of cameras are to indicate my approach to capturing my object at waist height/low to the ground (depending on the height of the object obviously). I'll need to capture it in a complete circle, ensuring each image contains 70% of the view captured on the previous one. The actual number of cameras in the image isn't accurate, as there's only 16. Ideally I'd like to capture 20-30 images from this angle.

• The inner (cyan) ring of cameras are there to indicate the additional angle/s of captures. In the case of the image, using the rock as a low-to-ground example, I've opted to indicate capturing the subject from above (but would also substitute this with lower to the ground shots or closeups depending on the subject). This, in the same way as the above point, will need to be completed in a full 360 degree view with each capture covering a minimum of 70% of the previous. Once again the amount of cameras in the image is only placeholder, as I would again prefer 20-30 images captured from additional angles.

I feel that if I stick to this capture strategy I've designed above, I should have some wonderful results when it comes to creating photogrammetry models of my chosen subjects! We'll have to wait and see until after the quarry trip.