OSL in Production

2022/08 UPDATE:

After using OSL for a couple of years I unfortunately must update this article to warn about all the bullshit that has happened with and around OSL in that time. This Article is only about OSL in 3dsmax, either native or in Vray. It does not apply to any other DCCs or implementations.
We recently were forced to immediately ditch our whole OSL shading system since the rendertimes suddenly increased to 13-20 times of the original (it is one single function call that has blown up). This is in addition to being stuck at max 2020 since max 2021-2023 has a fatal osl caching bug that hasn't been fixed for 4 months after reporting (and probably won't be). This is just the tip of the iceberg though since there were and are plenty of other bugs, lots of headaches and generally inconsistencies with it - not just in max but vray as well. We weren't able to make our system work in vray standalone, neither was it possible to rebuild everything in vrayosl since there are several bugs and unsupported features that are necessary and weren't fixed or tackled either.
To be fair though, since I seem to be one of the few people even bothering with OSL, there is little incentive to put precious developer time into it. But since it isn't supported properly I have to declare OSL dead for our productions. We still use a very specific irreplacable one (the interior fake shader with VrayOSL) but that's just one single node and not a network of nodes - which is hopefully not going to break. We remove as many OSL dependencies as we can.

What's OSL?

OSL or “Open Shading Language” is, as the name suggests, a programming language to write shaders. The language itself is generally platform and renderer independent. In theory, any shader should be usable in any renderer (that supports osl) on any platform.

Project environment primer

For this article, I will be talking about a recent production in 3dsMax using VRay. However, any (CPU) renderer, be it Corona or Arnold or even Scanline could have been used with the same system described below. Since 3dsMax is managing the osl evaluation, any renderer able to use 3dsmax textures can use the OSL system. OSL as a language supports both shaders and textures, but, for the time being, 3dsMax only supports textures. Also, note that several GPU renderer developers are working on supporting OSL.

OSL in production

People tend to see OSL as a “magic shader dev kit” that only really makes sense if you need one map that solves a specific problem, like the famous window interior fake from Julius Ihle https://julius-ihle.de/?p=2451 or my OSL adaption of Parallax Occlusion Mapping: Parallax Occlusion Mapping Shader . This can be, without a doubt, very useful and being able to write custom textures is one of the major advantages of OSL. However, OSL can also be used to build a custom shading system that can overcome the limitations of your DCC or Renderer.

Some of you may have seen my previous shading system, discussed in this article: https://www.ronenbekerman.com/timeride-vr-experience/#TexturingShading_shading_system . The following is a vastly improved replacement of this system based on OSL. “Schumann VR” https://www.schumannvr.com required a redesign of the old system to make it more versatile and overall improve the quality.

Requirements on the system:

• Hundreds of assets (primarily buildings) need to be shaded for far and closeup ranges in a short amount of time
• Shading artists should be able to change parameters like color, dirt, detail amount, etc per shader interactively while running VrayIPR
• Shaders should be light on texture budget (saving RAM)
• Anyone should be able to use the system, without prior knowledge
• Shaders might change throughout the production and then need to be updated in already shaded scenes

Before we get into the details on how this can be achieved, let’s look at a demo of the finished system:

So how does this work?

The window on the left in the video above is just a maxscript management tool. The interesting part happens in the materials. Let’s have a look at one of them.

1) The heart of the system, the control object. Any parameters the user can change are defined in this one node. We’ll get deeper into this below. Any connections going off the node control one or several other parameters somewhere in the graph.

2) The logic part of the material. This is basically like any other material in 3dsmax, just made entirely out of OSL maps (In this case I used a VRayDirt map instead of my osl replacement due to time constraints and some initial trace set difficulties. However it’s possible to do this entirely in OSL).

3) The Vray Materials(s). In this case, it’s a standard VrayMtl plus a VRayBumpMtl for additional edge damage. These could be replaced by any other material for a different renderer.

With that out of the way, let’s have a closer look at the control object.

The control object is also an OSL map that is basically just an interface for the user. The default settings are directly defined on the node, as seen below. The interesting part, however, is happening under the hood. We don’t want the user to change any parameters directly on the material, that would break our requirement of keeping every material an instance. What we want to do do is specifying the settings per object and material ID. How are we going to do that? With user properties. Many renderers are already supporting user properties natively and have their own maps for reading them. With user properties, you can store data directly on the object. For example, you can store the diffuse colors for a bunch of teapots directly on each object and read it out in the shader. Each teapot will have a different diffuse color that is defined in that user property despite all sharing the identical shader.

Well, why do we need OSL then? The problem arises when you want to use the same shader several times on the same object. For example on a multi-material that uses the same concrete shader 3 times on different material IDs. These would then either use the same settings on all of them or you would need to copy and rename the user properties for the duplicates. That means either accepting an awful limitation or breaking the instancing of the materials.
OSL to the rescue! With OSL we can use both the material ID number and the user property name to create unique settings for each material ID. The name of the user property is a combination of the material ID and the name (as seen in the image below).

The OSL control object then queries the user property for the current material ID that is being shaded and sets the material settings accordingly.

Granted, you need a little management script for assigning the materials and comfortably reading and writing the object properties but the heavy lifting is done by OSL.

Let’s recap, what are the advantages of this system?

• No user will ever have to change a single setting on the materials. That means that all materials using this system can be instanced in the scene and thus compile very rapidly. You are effectively using just a handful of materials on thousands of objects, each with their own unique look.
• Any material ID slot on any multi-material can have its own set of settings which are preserved if the object is duplicated
• Since all materials are instances and never need to be changed by the user, updating them from a central repository is trivial
• Once shaders are assigned and compiled, they don’t have to be recompiled when colors or parameters are changed, that makes shading with interactive rendering very fast

The Good, the Bad, and the Ugly

That’s all fine but is it worth switching to OSL based shaders just because of that? What if you don’t need that functionality? What are the pros and cons of OSL in more moderate circumstances?

The Good:

• Any exposed parameter of an OSL map can be controlled via maps somewhere else in the network. At least in 3dsMax, lots of parameters on standard maps are not accessible (only via controllers). That means that in OSL any setting can be controlled on pixel/sample level, which is a huge advantage over standard maps
• OSL is versatile, you can code solutions to problems that can not be solved otherwise
• OSL is relatively easy to learn, you don’t need much to create useful stuff
• In theory, any OSL based shader should be working similarly in any renderer or DCC.

The Bad and the Ugly:

• OSL maps need to be compiled before rendering. That can take a bit if you have thousands of them in the scene. However, it’s surprisingly fast after all, rarely a problem and can be managed.
• Generally, you want to avoid mixing standard maps and osl maps. Keeping it all in OSL means the maps can be compiled together. If you have other maps in between you have to compile and evaluate each “osl chunk” separately which will have a performance penalty. If you need to mix in some standard maps you want to keep it as far up in the chain as possible.
• In practice, some renderers or DCCs do not use the same names for object properties or face data (like material IDs). This means not all shaders will work everywhere.
• Support for OSL has only just recently begun to expand. Few people currently use it. Expect nasty surprises.
• Most likely not as fast as native maps from your renderer

Basic fun stuff using OSL

Ok, so we got a fancy system using lots of OSL maps and custom maxscript tools. What about more day-to-day practical functionality? Most of the OSL maps I used in my shading system are extremely simple; Mixing, multiplying, clamping, remapping ranges, that sort of stuff – just sometimes combined into more high-level nodes that make the graphs smaller and easier to maintain.

Here are some examples of how to extend your shading capabilities with simple shaders.

1) Manipulating UVWs. Easy and extremely useful. You don’t have to know any OSL to make use of it. This is a quick and dirty example that goes a little beyond (requires writing one line of code) but you can do a lot with the OSL maps that already ship with max.

2) A simple way of creating gradients in object or world space. This can, of course, be done with native VRay nodes but it’s super easy to just modify the existing remap osl shader and make your own specialized one-node version with mappable parameters.

3) Want to make your own heightblend node like the one in Substance Designer? OSL will save you the ugly mess of doing it with standard VRay maps.

Conclusion

While OSL can not solve all the shortcomings of shading in 3dsmax, it can be a tremendous help and enable more powerful workflows without having to be a full-time programmer. Little snippets of code can already make a huge difference.