Multi-View Inverse Rendering under Arbitrary Illumination and Albedo

We present the Multi-View Inverse Rendering (MVIR) method that jointly estimates detailed shape, illumination and albedo. Our MVIR uses the initial shape robustly recovered by multi-view stereo and jointly estimates them by using the geometric and photometric smoothness terms and the normalized spherical harmonics illumination model. Our MVIR allows spatially-varying albedo and per image illumination without any prerequisites such as training data or image segmentation.




MVIR software requires an initial 3D model (camera poses and 3D mesh). As a basic usage, we create the initial 3D model using VisualSFM and CMPMVS. Please make sure VisualSFM.exe and CMPMVS.exe are on the system path. The script generate_dataset.bat will then run VisualSFM followed by CMPMVS and re-format their outputs for MVIR.

> generate_dataset [images_folder] [mvir_folder]

The [images_folder] is the root of imagery. The [mvir_folder] will have images subfolder, cameras_v2.txt and mesh.ply as below.

├── images/
│   ├── 00001.jpg
│   ├── 00002.jpg
│   ├── 00003.jpg
│   ├── ...
│   └── XXXXX.jpg
├── cameras_v2.txt
└── mesh.ply

Now ready to run

> mvirc -i [mvir_folder]

Some control parameters are provided for fine-tuning the MVIR. Use mvirc --help to see the full list of control parameters.

To test the MVIR software, please first try the tiny dataset The Gajoen 005.

Reproduction the result in [1]

You can generate the same result by using the Joyful Yell dataset (section 4.2 in [1]). For this, you need to download our The Joyful Yell dataset and make sure Screened Poisson Surface Reconstruction (Win64, PoissonRecon.x64.exe) is in your system path, then simply run

> run_joyful [joyful_dataset] [output_folder]

This script automatically perform double MVIR steps with re-computing surface mesh (section 4.2 in [1]) using control parameters which is same to ours. This process requires ~14GB RAM.

Description of output

The MVIR results are saved in MVIR-Output_YYYY-MM-DD_HH-MM-SS under [mvir_folder]. The most important files are mesh_result.ply and illumination.txt.

├── mesh_result.ply
├── mesh_raw.ply
├── illumination.txt
├── visibility.txt
├── visibility.ply
├── LightCubeMap_Local/
├── LightCubeMap_World/
├── LightSphereMap_Local/
├── LightSphereMap_World/
└── log.txt
  • mesh_result.ply : The refined shape (as mesh) and the albedo (as vertex color). The albedo is the one re-computed for improving the visual quality (see section 4.1 in [1]).

  • mesh_raw.ply : The raw output from the optimizaton in equation 1 in [1] (no albedo re-computation).

  • illumination.txt : The estimated illumination (the spherical harmonics basis).

  • visibility.txt : The vertex visibility (see section 3.1 in [1], Visibility calculation). The i-th row indicates the camera IDs (0-based) visible from the i-th vertex.

  • LightCubeMap, LightSphereMap : The visualization of the estimated illuminations for every input image. _Local shows the illumination map in the camera coordinates (the same format in our paper [1]). _World shows the illumination maps w.r.t. the global coordinate system.

  • log.txt : Processing log of the MVIR software.

Third-Party Software

The MVIR binary depends on the following third-party software (note that all needed libraries are included in the distribution zip file):


Name Description
The Gajoen 005 10 images to test our programs .
The Joyful Yell 37 synthetic images, its ground-truth camera parameters, initial mesh and ground-truth mesh.

Terms and Conditions

MVIR software is provided for research purposes only. In case you use this software for a publication, cite reference [1], please. See the LICENSE file provided with the demo software.


[1] Kichang Kim, Akihiko Torii, Masatoshi Okutomi, "Multi-View Inverse Rendering under Arbitrary Illumination and Albedo", In European Conference on Computer Vision 2016. (PDF)