Building a Photometric Stereo Photogrammetry Rig

• Preamble
• What is a Photometric Stereo Photogrammetry Rig
• Where things get complicated
• The Turntable
• The Lights
• Automating Cross Polarization
• The Brain(s) of the System
• The Software: processing a mountain of data
• The Rig: first attempt
• The Rig: second attempt and where I'm now
• Conclusion

Preamble

A few notes before we start. This is a hobby project, I have no formal education in any of the stuff I'm talking about here and most and foremost I'm not an electrical engineer (or any other). So what I'm going to show here is most certainly not safe - or particularly smart. It was never meant to be a professional product and it's not the most efficient way to do it.
When I started, I knew nothing about photometric stereo or even that rigs like these existed. All I wanted to do was combine substance designers "make a normal map from photos with different light directions" with photogrammetry. I had no idea what I was getting into.

Of course I didn't believe I was the first person to come up with that, far from it, it's a relatively straight forward idea if you put one and one together; but I only realized that there was a small niche group of people building these kinds of rigs (mostly for art and historical artefact preservation) when I was looking for software solutions that I could use months after I got started.

So, since I'm clearly not an expert, what's the point of this article? Simply put, I want to live in a world where open exchange of knowledge and ideas is saught after and valued. The only way I can make this happen is by doing it myself. I have always learned from others and I hope others can learn from me. While I can afford to be so ridiculously idealistic, I fully understand why people who do this commercially want to keep their methods to themselves.
For me, this project was just a fun challenge to myself if I could do it, no more, no less. My ego wants me to believe that I came up with most stuff here by myself but that would probably be self-deception. I have certainly picked up ideas and methods along the way from others, even if non-intentional.

To summarize, this is meant to be a collection of lessons learned and starting points for anyone like me who might want to do the same but has no idea where to even start. As I've learned, the ressources out there aren't exactly forthcoming with details, or if they are, it's not written for people like me who wouldn't understand a math paper if their life depended on it. So, hopefully, this article can help someone build or improve their project. It's what I would have loved to find when I started. Also, don't hesitate to contact me, I love talking about this stuff.

What is a Photometric Stereo Photogrammetry Rig

The idea behind a photometric stereo photogrammetry rig is that you can scan the geometry as well as the surface properties of an object in one go. That means you will end up with both the geometry and a (somewhat) complete PBR texture set of the surface - meaning Albedo, Normals, Roughness, and if possible Reflectivity and Metalness.
To do that, lots and lots of photos are needed additionally to the usual ones shot for photogrammetry. Each camera position requires a set of images where a light source is illuminating the object from different angles. With these light angles it's possible to calculate a screen space normal map. At an absolute minimum, you need 4 light angles: Top, left, bottom, right. If you think about it, that's essentially what a common tangent space normal map looks like if you only look at the red and green channels separately. That said, the more lights used, the better the results - at least to a point of diminishing returns.

To extract Albedo and Reflections we require cross-polarisation. We want to shoot every light angle with and without cross-polarisation, so we can substract one from the other and end up with the diffuse as well as the specular reflection response. With the 12 light angles I use, this means shooting 12 cross-polarized imges, and 12 parallel-polarised ones ( plus 1 photo without lights to be able to subtract any ambient stray light). 25 shots in total for each camera angle. From these sets of images we can build our textures and project them onto our protogrammetry model.

Where things get complicated

So far, so good - sounds quite reasonable, right? Well, the devil is in the details and that, unfortunately, has quite substantial consequences going down the complete pipeline.

If you have done any photogrammetry, you know that sharpness is paramount. You don't want motion blur, you don't want blurry images. With photometric stereo, this gets a lot worse: every single image for one camera angle has to be a pixel perfect match to the last one. If the camera moves even so much as a pixel, you will see it once you calculate your normals or subtract the diffuse and reflection passes.
In practice, this means…

• don't even think about touching the camera for changing settings or pressing the trigger
• even forgetting to disable vibration reduction will ruin a set (ask me how I know)
• the focus must stay exactly where it is for all images in one set
• if you use an old school DSLR, the mirror flipping up will ruin your day
• if you use a modern mirrorless … the shuttershock will
• you can't touch your lens or rig to rotate the circular polarizer from non to cross-polarised between sets
• you have wooden floor boards? Don't you dare to move around anywhere near the rig (ask me how I know!)

You get the idea. It's much more finicky than regular photogrammetry and essentially means you need to control everything electronically from afar and touching is a no-no. Generally, anything that has mechanical movement is to be avoided if possible, but we'll get back to that later. You may get away with some, but if you can avoid it you should.

Aside from physical restrictions and issues, shooting 25 images per camera angle is nothing to sneeze at either. It slows down shooting, it's taking up hundreds of gigabytes of storage, and you need to run software over that massive dataset to create all the textures which takes some time as well. All in all, it's a massive headache… or a fun set of engineering challenges... depending if you're doing it for fun or not.

The Turntable

Starting with the easy stuff: the turntable. Technically, you can do without one, but if you want to be even remotely efficient you need to use an automatic turntable. Not only will it make sure you don't accidentally bump into the objects while turning by hand, it also makes sure you save tons of time, as the rig and the turntable will rotate and shoot at the most efficient rate.
In my case, I already had one built for my regular photogrammetry stuff. The stepper motor is about as weak as they get, but it has no problem turning the weights I regularly have to deal with. A kg or two are not an issue at all. The reason for using this particular stepper is that I already had it when I made my first turntable from an ardunio beginners kit that I had lying around. Making that was a great start into electronics and I have used it ever since. It's also really small and lightweight and easy to integrate… which isn't necessarily the case with the usual NEMA ones.
The only adjustment I did for the new rig was switching from wired communication to wireless using nRF24L01 modules. I was really happy when I found an Arduino Nano with one built-in. Makes it so much easier to work with.

The Lights

The fun part! Before we get to it though, it's important to remember the issues we have to overcome. What features do we need? Why LEDs and not flashes? How much "oompf" (in scientific terms) should they have?

As lined out in the issues paragraph before, the major problem to overcome is making sure nothing moves and everything is pixel perfect. As stated, flipping mirrors are a no-go and even the opening and closing shutter can make the camera wobble a little bit. In my case, with the super small stuff I regularly shoot, the effect of the shutter through the macro lens is quite severe. But even with more medium sized stuff I would try to avoid it. A pixel of offset is enough to be noticeable in the output, resulting in bright and dark outlines on contrasty edges - and with more images it's more likely to happen. What about electronic front curtain shutter? The shutter is already open before release, so.. no shock, right? Correct, but the shutter closes and opens again at the end of the image capture, so while the first image might be good, the second, third, etc may be misaligned.
What that leaves us with is full electronic shutter. No moving parts, no risk of misalignments. Unfortunately, electronic shutter also means we can't use flashes. A flash (usually) is too fast to be caught by the sensor readout from an electronic shutter and if your camera even allows it, they usually don't, you will end up with a narrow white line and that's it (for an explanation: https://www.dpreview.com/articles/5816661591/electronic-shutter-rolling-shutter-and-flash-what-you-need-to-know). Well, unless you buy a Sony A9 III with global shutter... sadly, I don't have that kind of money.

So, building with flashes is out of the question, but were it possible I would be very interested to know if they could endure the stress of being set off repeatedly.

Being limited to permanent light sources, the obvious choice is LEDs. I had no idea how much power I should aim for, but browsing through the available options at my go-to electronics shop, I ended up with a 40-ish Watt high CRI led chip (CRI = color rendition index, basically how good the colors of objects come out). As a rule of thumb: the higher the CRI, the lower the light output per watt, which is why super bright LEDs usually have a rather crappy CRI rating and why power draw isn't telling the whole story.
The question is if we should aim for super high power in the first place. As we will discuss later, what's holding us back is the number of images the camera can shoot per second, not necessarily the shutter speed at which it shoots. In my case, with my camera, it's relatively low, so shutter speed would only be a factor if it was too long. Shooting sets doesn't get any faster with shorter shutter speeds; I'm limited by the frames per second, so a more powerful led would therefore not make much of a difference.

What is relevant in any case though is getting rid of the heat. I was worried I could not properly cool the 40 W over the time of the shoot, and I would only know if it was good enough if I tried it out with a finished light. So, being who I am, I looked for the biggest heat sink I could fit in the space I was willing to design for, completely ignoring that metal has weight... foreshadowing! Later, closer to finishing the design I also added a little fan add-on to make sure I could cool the lights inbetween sessions if necessary.
Was that necessary though? I'm not sure. I don't think so, but I also haven't tested everything to that extent. I made sure in the control software that the lights never run at full power in testing modes and only turn on for the shortest amount of time while shooting. Originally, I was planning on printing the case in heat-resistant Polycarbonate but due to bed adhesion and warping issues, I went back to PLA, with the mindset that I could at any point replace the housing with PC if I had to.


To make sure I use as much of the light output as possible, further limiting the energy that goes into heat, I chose a 90° collimating lens to put in front of the LED (they usually come in 60, 90, and 120°). Tighter might have worked, but I wanted to make sure I have an even light spread, as the outer rim will always have a bit of a vignette.
It's not strictly necessary to have a lens, it works perfectly fine without, and would save quite some weight, but the more light I can get onto the scanning object and thus the sensor, the sooner I can turn off the LED - again, limiting the heat output.

The last piece of the puzzle is the polarizer on the front. For my previous projects I have been using polarization sheets. B+W sells them specifically for photo and video use for a rather hefty price tag, a bit too hefty for my taste. I made some tests and realized that cheap TV polarizer replacement sheets from aliexpress are essentially just as good and comparatively cheap. Apart from being a tiny bit warmer, I could not see or measure and difference from cheap to expensive. The difference in price might be due to heat-resistance or some other non-obvious aspect, I'm not sure.
Later, however, I found that the cheapest shittiest glass polarizers for 4 $ on aliexpress are remarkably good for the purpose I was needing them for. They have no coatings, the threaded housing is absymal and I would never put them on any of my camera lenses... but they are perfect for the lights. I can't see a difference in cross polarisation performance, the transmission is about as good as my very expensive B+W filter, and in terms of color tint and neutrality they are more than good enough. So, highly recommended from me. Compared to the polarizer sheets, I get a bit more light out of them and they are much more scratch resitant - which often is a problem with the flimsy brittle plastic sheets.


Summing it up, the design of the lamps is dependent on the amount of light you need to reach the ideal shooting speed given by your camera. This is affected by the distance to the object, and ends up being a balance act between heat, size, weight, and of course cost.
Were I to redesign the lights, I would try to minimize the weight and complexity as much as possible. Adding a fan and generally, adding holes for air circulation increased the risk of dust settling on the LEDs and glass tremendously. I haven't yet used the lights enough to know if this is going to be a problem, but I would be surprised if not, certainly in the long term. Ideally, I would want to try to build the next iteration as a closed unit that has the heatsink as part of the housing itself.

Automating Cross Polarization

Right from the beginning, the make-or-break issue of this project was the hands-free switch between cross-polarized and regular photos.

I will not go into the details of cross-polarization here, I will assume that readers of an article like this one are already familiar with it, but as a quick reminder: cross polarization means you are polarizing the light at the source and add another polarizer in front of the lens to block everything that comes from that light. This will effectively neutralize all specular reflections off of an object while letting the diffuse "basecolor" part through. That's because diffuse reflection happens inside the object and the polarized light reaching it will get de-polarized in the process. When it is then re-emitted from the object and travels through the lens, the filter can't block it and the camera records it.

As stated before, touching the polarizer in front of the lens is a no-go. We can not afford any potential misalignments. The first obvious idea is to motorize the 90° rotation of the filter - and there are conveniently already products out there that do that - they are not cheap though, and it doesn't solve the problem of vibrations and mechanical movement. It's also not exactly a quick process to turn the filter. All in all, not a great solution - though some people apparently have had success with it.

From videography I knew of variable ND filters. These filters are just two polarizers that are rotated against each other - which then blocks more and more light. I also knew that some film cameras have electronic ND filters built-in. So, consequently there had to be something like an electronic polarizer out there. It didn't take long to find a company called LC-tec which is a manufacturer of these electronic ND filters. It turns out, they also manufacture electronic polarizers - which isn't too surprising as these are essentially the same thing, just doubled up. I contacted a reseller in Germany and bought one of these modules (they are expensive!). Unfortunately, the biggest size you can get without buying a few thousand is around 50 mm wide. Not great if you're using a full frame camera and lens but what choice did I have?
Browsing through their spec sheets it dawned on me that electronic polarizers (or electronic ND filters) are simply LCDs… just with only one segment or pixel instead of many. If you look at your old school digital alarm clock, what's darkening the segments of the numbers is cross polarization!

With that mind-blowing realization I went and looked around for single-segment LCDs, and eventually I found a cheap and abundant source for them. Auto-darkening welding helmets. It's the ideal use case for an electronic ND filter. They auto-darken when they sense bright light and save your eyes from the welding arc.

I tried a few different ones and ended up using these: https://de.aliexpress.com/item/1005006092863576.html (True Color!). They are relatively cheap, have somewhat neutral glass, and are generally of decent quality. The only down side is that they are not using optical quality glass and have no anti-reflective coatings whatsoever. Given the lack of alternatives though, I'm more than willing to make that compromise. Compared to the professional LC-tec modules, they are about double the size and more than 10 times cheaper.

We don't want to use the whole assembly though. A welding helmet LCD has a lot more parts than we need. When I dissassembled my first ones I didn't quite understand how it works and I ended up with a functioning but sub-optimal solution (which is what I used to make all the examples and scans you see in this article). While writing the article though I realized that you can go a lot further and improve the quality even more.

Before we start, a disclaimer: if you do this, be aware that this is potentially dangerous. You're on your own here. I have a vague idea what's in there but I'm neither a chemist nor can I say what a chinese manufacturer puts in their panels. I've already been surprised once and I can't say for sure that my assumptions are all correct. If you follow these instructions make sure you wear proper safety equipment and that you do some research yourself first. The glass in these panels can break, and the liquid crystal fluid as well as the electrodes are not exactly healty stuff. Be careful!

First, we open the plastic casing. The two halfs are welded together, so some force is required. The easiest way is to use the hole where the cables come through and start prying apart there. Once it's open, you will find the panel in the middle and two PCBs at the top and bottom. The LCD is soldered to the upper PCB using 8 metal pins. You need to desolder them first. Be very careful. The metal pins break and detach easily and that will ruin the whole panel.


Once desoldered, have a look at the LCD panel. There are several layers sandwiched together. Not all panels are made the same and the ones I use have had changes in their manufacturing over the past 2 years as well. The overall construction and parts used are generally about the same though.
What you see here, from top to bottom, is the same as the photo above from left to right:


Not quite so obvious from just looking at the real thing, isn't it? In fact, I missed most of it in my first few disassemblies. I had no idea there were several plastic polarizers in there. They are extremely thin and I had missed them over and over again until I scratched some accidentally and figured it out by mere coincidence. First I thought they were very thin coatings but it turned out to be actual plastic sheets.
You will also have noticed that there are two LCDs in there. Frankly, I don't know why there are two instead of one. Could be safety, could be darkening contrast... I'm not sure. I found conflicting information about that. What is certain though is that the two LCDs both work for our purpose - so we get two for the price of one.
We now want to get rid of all the filters and polarizers so we only have the glass panels left. However, we do NOT, under any circumstances, want to split the taller and shorter glass panels from each other (both at 4 and 7). Don't try to split those, that's where the magic liquid stuff lives.

So, let's start with 1) , the UV/IR Filter. That one can be either glass or plastic. The plastic ones are very easy to remove, those out of glass are a lot more difficult. We want to soften up the glue so we can remove the layers one by one. The thin black glue strips are very sticky, especially to glass, so be very careful AND patient. The best way I have found to deal with it is using hot air to soften the glue first. Don't heat up the LCD too much, make it pleasantly warm to the touch, no more, no less. Now get a scalpel and run it around the black strip to try to find an entry point. With the plastic filter it's almost coming apart by itself, with the glass, you may have to work a bit harder. Don't try to pry it apart at one single point, the glass WILL break - believe me, I've done it. Run it around repeatedly and work from several sides. If it still doesn't budge, get some Isopropyl Alcohol and work it in with the scalpel, the glue will come loose sooner or later, it's just a bit of a mess to clean up when using IPA.

Once the filter is off, get rid of the 2) black glue strip. Then flip the panel over and try to find the edge of the 8) plastic polarizer sheet. Use the scalpel to catch the edge. It's there, it might just be hard to make out. This sheet is usually thicker and relatively easy to remove, that's why we do it first. Be patient and strip it off slowly, use some hot air if needed.


With the back polarizer removed, we continue to split apart the remaining two glass LCD panels at the black 5) glue strip. Same as before, use hot air and alcohol and be patient. This stuff is sticky.

You should now be left with two glass LCDs where one side (usually the shorter glass panel) has still a polazier attached to it (3 and 6). That polarizer is very thin and extremely easy to miss. If you sprinkle some alcohol on it or try to wipe off some glue residue you will notice that the front and back side behave differently and one side is more hydrophobic than the other. That's your clue that there is still something on it. You will also hear a slight difference if you tap the front and back sides with the scalpel. Use hot air again to soften up the polarizer glue and try to find an entry point - it's difficult, if you don't know it's there it's extremely hard to spot. I missed it all the time. Do that for both the LCD panels.

Alright, we should now be left with just the two double-glass LCD panels (4 and 7), 4 metal pins each. What now? What's the LCD actually doing?
This is going to be the heart of our cross polarisation switching. Since we have removed all the polarizers from the LCD, we need to polarize the lights (as usual) and we need a polarizer in front of the lens. The only thing the LCD does is rotate the polarization of incoming light by 90° if powered. That's the only job the liquid crystals do. If it's off it will just pass-through the light as it is. Once you power it, it will twist the incoming polarized light by 90°. That means we can now switch between parallel and cross polarisation by just powering the LCD on and off.


To set it all up and calibrate everything, we need a polarized light source, the powered(!) LCD and the filter on the lens. Look through the camera and turn the lens filter until the light gets blocked completely.

Technically, we could have reached the same result without taking apart the panel completely. In my first tests I only removed 1) and 8). The remaining polarizer sheets will then replace the camera lens filter. Works the same way. This isn't ideal though, as you are running two LCDs, and the additional plastic filters which block more light than necessary. Removing all the layers down to the blank glass will enable us to use our own high quality polarizers and remove as much plastic and glass from the optical path as possible.


The next challenge was finding a way to drive the LCD via microcontroller. LCDs are driven by alternating current, not direct current. So, the 2 connections (with 2 pins each) are the two poles that expect alternating voltage.
The LC-tec module I got was sitting in a box for a while so I didn't realize I was sent the wrong one and it was too late to send it back when I finally got to it. What was supposed to have a 4V drive voltage amplitude, needed 24V, so driving that one was a bit of a headache. I managed to make it work using a motor driver with an H-bridge but I can't say it was the most pleasant thing to figure out. Very stressful, given the price of that module and me not knowing what I'm doing.
The welding helmet LCD, however, is running on just about 5V which makes driving it very easy. LCDs generally seem not to be too fussed about the exact voltage, so the safe operating voltage span is rather wide.
What I figured out was that I could drive the welding helmet LCD simply by connecting it to two digital pins of an arduino nano. By turning one of them on and one of them off in alternating pattern, I had made a simple AC square wave. Don't ask me how exactly the arduino handles it but it does support it and it works without damaging anything. Could not be easier, but it took me a while to get there.


The original welding helmet controller ran the LCD on 45Hz. That sounds really low, but it is a matter of the how fast the LCD reacts and how quick the relaxation time is. The documentation of LC-tec's FPM below has a nice image to explain that. Still, I'm running mine at 100Hz for good measure.

https://www.lc-tec.se/wp-content/uploads/2019/05/FPML_FPML-AR-specification-1602.pdf

In the end, LCDs are rather robust and have a wide range of voltages and frequencies they can operate in. There's no real current flowing (it's acting more like a capacitor), so there's not really much damage you can do in the first place.

The Brain(s) of the System

As stated right at the beginning, I don't have any formal (or informal) education in any of the fields I'm working in here - electronics included. My level of electronic engineering can best be described as "putting lego together". I can just about power up an LED without googling it first. So keep that in mind when I show off the design of the control unit; I know it's neither smart nor efficient... but on the plus side, if I manage to cobble together something that works, anybody can.

The issue I had to sort out first was how to control the 12 lights. Originally I had planned to use 8 spotlights and a much more powerful ring light around the lens. The spots are drawing around 40W each, the ringlight was designed to be 120W. While it would have been convenient for testing and fine-tuning, making it so that all lights could be on at the same time was not necessary for the functionality of the whole rig. So with that in mind, my plan was to use existing pwm-controlled opto-isolated mosfet modules. Basically a small module that makes it possible to dim and control much higher loads with just a signal from a small microcontroller. There are plenty of these around in all sorts of configurations, and I had no clue how much heat they produce (if any). To be safe, I went with some that were rather oversized and more than enough for the loads I was planning for.

The first issue I ran into was that the pwm signal of the microcontroller was way too slow to be useful for a light like that. I immediately got bands of light in my images. I would either have to massively increase the pwm frequency... somehow, or only use the lights in full on or off mode. In the end I went with the latter and only used sub 100% duty cycle pwm for maintenance modes.

Apart from that, the first tests with just one module used were promising, so I went ahead and made a mounting PCB that could hold 4 of them. With 3 of those I could power 12 individual lights or devices. Unfortunately, just slapping together multiple did not go as planned. I suddenly had huge ripple on the output and never quite understood where that came from (my best guess is the capacitors charging and discharging back and forth).


In the end I changed the mosfet modules to a smaller version which used less space and also fixed that problem.

With that out of the way, all I needed to do now was throwing together the solutions I had already made for previous rigs and projects. Instead of tracking time in code and managing the LCD update frequency this way, I opted for splitting up the "brain" into an arduino mega for all the main functions and an arduino nano just for the lcd. This way the code was easy to write and maintain and the cost difference is negligible.

While it certainly isn't the best solution, I went with reed relais to short the camera remote release pins. This way, the camera is fully isolated from the rest of the circuitry and the electronics are about as simple as it gets. The downside is that reed relais have a limited life span due to their mechanical nature. So far I have yet to break one, but they are cheap and easy to replace - so we'll see.

I also added two auxiliary pwm mosfet controlled outputs for the fans, and things I may need in the future. Additionally, an IR receiver for a handheld remote was added, an oled display, and the wireless control for the turntable - and that's it.
Could I have fit all of this on a much smaller PCB if I had gone without pre-made modules? Yes, but this works and I can swap parts easily if I ever need to. Good enough for this project.


A short note about the camera focus/release: having full control over the camera would of course be ideal. Unfortunately, and especially with Sony cameras, this is easier said than done. There are SDKs and other ways to go about it (over PC usb or Raspberry Pi and third party libraries) but what I wanted was a portable and simple(!) way to do it. Nothing I have to boot up and power down gracefully. That left me with what I've already been doing for years - shorting the pins used for cable remote trigger doodads. That obviously has the big disadvantage that you can only trigger the autofocus and the release... and that's it. There is no feedback when an image is done and there is no way to know if a photo had been taken or not. In my old rigs I had a light sensor attached to the "writing" status LED of the camera to track when the image had been written to the card... but this time I bought myself out of the problem and purchased a CFexpressA card. With that, the internal buffer would never fill up and I could just shoot as fast as the camera allows - which, at around 6 images per second, isn't that fast - but faster than tracking the led state. The CFexpress cards and readers are eye-wateringly expensive, but I haven't regret my decision to buy one.

Long story short: it's a very simple system... and it works. There's certainly much room for improvements but on the plus side, it shows that it's not too hard to get something going, even for an idiot like me.

The Software: processing a mountain of data

Making the software to process all that image data is probably the most difficult task in the whole Project. I'm not a mathematician, far from it. I knew from the start that I wouldn't be able to do this myself - at least not without help. Still, the first thing I did, even before building any hardware, was making a "proof of concept" normals generation method inside Blackmagic Fusion. It was naive and would probably fall apart rather sooner than later, but it was good enough to give me the confidence to start with the hardware.

Later, shortly before finishing the first version of the rig, I went and searched online for already existing solutions. I searched far and I searched wide... but nothing really fit or if it did, it was too complex for me to understand or use. I've never been in academia and most papers just are unreadable for me. Luckily, at some point, I stumbled upon this page: https://github.com/visiont3lab/photometric_stereo/tree/master . While my python skills are not exactly stellar, setting up a python environment with numpy and openCV was not much of a problem. It didn't specify any terms other than the standard MIT license - but that is more than good enough for me.
I downloaded it, installed everything and tried to figure out how to use it. The information given is a bit sparse but with some testing and quite a few modifications I managed to get something running.
Before I could generate normals though, I had to sort and prepare the photos. I had to generate the albedo, split off the reflections and sort it all into one folder per light angle. I first used Maxscript and Blackmagic Fusion for that, but later - after familiarising myself more with numpy and openCV, I switched over to do that in a python script as well.
I then went and refined the script I downloaded. First I removed a bunch of stuff I didn't need and then went and made everything multithreaded. While the script itself isn't slow, running several processes in parallel did speed things up quite a bit. I can't remember exact timeframes anymore but I brought it down to just a few minutes for 40 GB (or 1500 images) in my test set.

All of this is still very much WIP and I don't really feel comfortable sharing it in this state. The difficult part, however, is already available and solved with the script from the link above. That's all that's really needed. The rest is just minor improvements and some basic image manipulation that can be done pretty much anywhere and with anything.

Once the Albedo, Reflections, and Normals are generated we need to make a typical photogrammetry model and reproject these textures back onto it. How you generate the model is pretty much irrelevant - as long as you get access to all the generated cameras.

You may have noticed that the generated normals are calculated in camera space:

This may look deceptively like what we want later on in our baked output model, but it's not what we need right now for reprojecting. For that, we want our normals to be in world space (or object space, it's essentially the same thing in this case). We need to convert the camera space normals to world space normals and for that we need an fbx with all the generated camera positions in it.
This again, is something that can be done in lots of different ways. In my case, I decided to use houdini for processing. It's probably not the fastest way to do it, but it certainly is one of the easiest. All we really need to do is load the normals images in cops (this was before the new cops was released), load in all the exported cameras, and multiply the screenspace image normals with the inverse of the camera transformation matrix. That's it - or at least I think that's it. I have never really tested it out much more than to a proof of concept state.

-- EDIT 2025/02/25: I just realized that I didn't specify that you also should zero out the translation of the camera matrix first. We only need the rotation, not the translation for this - it's normal vectors after all, not position vectors.

-- EDIT OVER


And with that, we can reproject all the textures back onto our Photogrammetry model. All we need to do now is bake those world space normals into tangent space normal maps - and this can be done pretty much anywhere with any baker. The only real challenge there is to make sure the axis and color channels match.


This is a very low-res and low-image-count test, but you get the idea and it proves the point I think.

The Rig: first attempt

Generally, putting some lights around a camera doesn't sound too difficult. My goal was to make something that has a wide diameter but could also be stowed away easily. If you have done anything in terms of videography, you're probably familiar with 15mm rods and clamps. That's the most obvious way to approach this problem - but unfortunately also comes with downsides.
My idea was to build a rig that could be split in the middle and fold back like two wings. The vertical rods were the pivot points and the lights and some clamps in the middle were used to lock it all down when in use. Unfortunately, while it all sounds good, it's not really practical. The main issue with a rod based system is that it all twists rather easily. You can never lock down everything so tightly that accidentally bumping the rig could not twist the whole thing a bit (and thus misalign the lights). It also wasn't exactly lightweight - but that was more an issue of the lights than the rods.
My original plan was to put all of this on a small camera crane I already had. Unfortunately, the weight was too much even without the camera in it. The crane was barely able to hold it all up, and the video head just straight up failed when trying to tilt the whole thing up and down.


I then finished, tested and used the rig on a sturdy light stand (as seen above), but couldn't tilt it up or down. All in all it was a failed attempt. I didn't like the potential of twisting rods, I didn't like the weight, I didn't have a solution for positioning and tilting, and overall it was already way too big for the small room I have.

The Rig: second attempt and where I'm now

The first thing I wanted to change was the rods. Ideally, I should have designed the lights to be much less heavy... unfortunately that train had passed long ago and I had to work with what I got. With weight being something I could no longer fix, adding more weight didn't sound like a particulary problematic idea. The crane would never work out, so I had to come up with something from scratch. My main goal was to design something that I could roll around and position freely around the object I wanted to scan - and if weight was no longer an issue, why not go full aluminium extrusions:

Pretty straight forward. Linear rails, 2020 and 3030 Aluminium extrusions and a couple of flanges as rotation points on the sides.
I built the main middle bit, easy enough, ... and then had it sit around for a year. While weight was no longer a problem - space was - more than ever. While I have all the parts cut and ready for assembly, I don't have any space to build it... or to shoot with it if I did. This didn't come as a surprise of course, but I had hoped I would come up with a solution eventually. I didn't. So, until I find some room somewhere, this project is on ice and will just sit there unused.

Conclusion

So, what did I learn? What would I do differently? Would I even recommend making a rig yourself?
For me, the main reason to do all of this was the challenge itself. I frankly don't have much use for it. You will also have noticed that there is a distinct lack of example scans - that's partially due to the unfinished state and not being able to make more scans, but also because I never went far beyond the "proof of concept" stage. Once I had something that worked, I moved on to the next problem to solve. I never even bothered scanning the whole shoe.
This also then partially answers the question of: should you make one? I honestly don't think many people need this. It's a very specialzed tool. The whole process is so complex and time intensive that you have to think twice if it's worth it. For art and historic artefact preservation? Sure! That's a great use-case. The more detail the better. For a random game asset? Maybe not. But that's of course a question everyone has to answer for themselves.
For me, this was all about "can I do that myself" and the anwer is yes - but I don't intent to really scan much or make a business out of it. The problem solving part is what drives me, not the application of the solution. I will have a use for it occasionally, sure, but it probably won't even pay off the costs for the whole thing.

In terms of lessons, the most important one here definitely was "weight" - never underestimate the impact of weight in your design. The lights essentially dictated all the design decisions afterwards and handcuffed me more than I want to admit. If I were to do it again, my first priority would be making it as light and simple as possible.

If anything in this article was helpful to you and you end up building something yourself, please drop me a line or two. I always love to hear what people found useful and what they made out of it.