Alchemist: Parametric Control of Material Properties
with Diffusion Models

Prafull Sharma1,2, Varun Jampani2, Yuanzhen Li2, Xuhui Jia2, Dmitry Lagun2,
Fredo Durand1, William T. Freeman1, Mark Matthews2

1MIT CSAIL, 2Google Research

Paper
Supplementary

Abstract

We propose a method to control material attributes of objects like roughness, metallic, albedo, and transparency in real images. Our method capitalizes on the generative prior of text-to-image models known for photorealism, employing a scalar value and instructions to alter low-level material properties. Addressing the lack of datasets with controlled material attributes, we generated an object-centric synthetic dataset with physically-based materials. Fine-tuning a modified pre-trained text-to-image model on this synthetic dataset enables us to edit material properties in real-world images while preserving all other attributes. We show the potential application of our model to material edited NeRFs.


"Change the roughness of the teapot."
"Change the transparency of the cupid statue."

"Change the metallic of the pot."
"Change the albedo of the Buddha statue."

Parametric Control of Material Attributes

We present the results of our method for editing roughness, metallic, albedo, and transparency of objects in unseen real images. As the relative attribute strength is linearly changed, the results show smooth edits changing only the desired material property while maintaining the high-level semantics and other information in the image.
Note: Please consider going full screen to observe the subtle changes in the material properties, especially in the case of roughness and metallic.

Roughness


"Change the roughness of the eggs."
"Change the roughness of the vase."
"Change the roughness of the pottery."
"Change the roughness of the bottle."

Transparency


"Change the transparency of the yoda."
"Change the transparency of the pumpkin."
"Change the transparency of the polar bear."
"Change the transparency of the vase."

Metallic


"Change the metallic of the teapot."
"Change the metallic of the apple."
"Change the metallic of the wrench."
"Change the metallic of the toy pokemon."

Albedo


"Change the albedo of the apple."
"Change the albedo of the shoes."
"Change the albedo of the cherries."
"Change the albedo of the unicorn statue."

Spatial Localization

We can limit the effect of our material edit to the segmentation of a particular object. As shown below, in case of two cats or two cups, we can use the segmentation maps for a specific instance to mask the relative material change.

Comparison Video Results

We compare our method to InstructPix2Pix prompt-only trained on our data (Baseline) on unseen real images for linearly varying values for the relative attribute strength. Observe the smooth transitions produced by our method as opposed to flickering transitions in case of the baseline.
Note: Consider viewing on full screen to observe the subtle changes in the material properties, especially in the case of roughness and metallic.

Roughness


"Change the roughness of the toy batman."
"Change the roughness of the statue."
"Change the roughness of the headphones."

Transparency


"Change the transparency of the vase."
"Change the transparency of the dolphin."
"Change the transparency of the binocular."

Metallic


"Change the metallic of the toy."
"Change the metallic of the duck."
"Change the metallic of the mario."

Albedo


"Change the albedo of the crab."
"Change the albedo of the statue."
"Change the albedo of the fish."

Material Editing in NeRFs

Material editing of NeRF on a selection of scenes from the DTU MVS. We edit training images to have reduced albedo or higher specular reflections. We then train a vanilla NeRF configuration and observe highly plausible 3D structure with the intended albedo, roughness, and metallic changes.
Note: These orbit animation views are outside of the training distribution, and thus the large amount of floaters are not unexpected, nor the background artifacts as the near-far bounds are set for the foreground only.

Figure 1
Scan 30 with (albedo: 1, roughness: 0, metallic: 0)
Figure 2
Scan 30 with (albedo: 0, roughness: -0.5, metallic: 0.5)
Figure 1
Scan 69 with (albedo: 1, roughness: 1, metallic: 0)
Figure 2
Scan 118 with (albedo: 1, roughness: -1, metallic: 1)