Multi-View Inverse Rendering under Arbitrary Illumination and Albedo

We present the Multi-View Inverse Rendering (MVIR) method that jointly estimates detailed shape, illumination and albedo. Our MVIR uses the initial shape robustly recovered by multi-view stereo and jointly estimates them by using the geometric and photometric smoothness terms and the normalized spherical harmonics illumination model. Our MVIR allows spatially-varying albedo and per image illumination without any prerequisites such as training data or image segmentation.

Software

Dependnecies

Usage

MVIR software requires an initial 3D model (camera poses and 3D mesh). As a basic usage, we create the initial 3D model using VisualSFM and CMPMVS. Please make sure VisualSFM.exe and CMPMVS.exe are on the system path. The script generate_dataset.bat will then run VisualSFM followed by CMPMVS and re-format their outputs for MVIR.

> generate_dataset [images_folder] [mvir_folder]

The [images_folder] is the root of imagery. The [mvir_folder] will have images subfolder, cameras_v2.txt and mesh.ply as below.

mvir_folder/
├── images/
│   ├── 00001.jpg
│   ├── 00002.jpg
│   ├── 00003.jpg
│   ├── ...
│   └── XXXXX.jpg
├── cameras_v2.txt
└── mesh.ply

Now ready to run

> mvirc -i [mvir_folder]

Some control parameters are provided for fine-tuning the MVIR. Use mvirc --help to see the full list of control parameters.

To test the MVIR software, please first try the tiny dataset The Gajoen 005.

Reproduction the result in [1]

You can generate the same result by using the Joyful Yell dataset (section 4.2 in [1]). For this, you need to download our The Joyful Yell dataset and make sure Screened Poisson Surface Reconstruction (Win64, PoissonRecon.x64.exe) is in your system path, then simply run

> run_joyful [joyful_dataset] [output_folder]

This script automatically perform double MVIR steps with re-computing surface mesh (section 4.2 in [1]) using control parameters which is same to ours. This process requires ~14GB RAM.

Description of output

The MVIR results are saved in MVIR-Output_YYYY-MM-DD_HH-MM-SS under [mvir_folder]. The most important files are mesh_result.ply and illumination.txt.

MVIR-Output_YYYY-MM-DD_HH-MM-SS/
├── mesh_result.ply
├── mesh_raw.ply
├── illumination.txt
│
├── visibility.txt
├── visibility.ply
├── LightCubeMap_Local/
├── LightCubeMap_World/
├── LightSphereMap_Local/
├── LightSphereMap_World/
└── log.txt
  • mesh_result.ply : The refined shape (as mesh) and the albedo (as vertex color). The albedo is the one re-computed for improving the visual quality (see section 4.1 in [1]).

  • mesh_raw.ply : The raw output from the optimizaton in equation 1 in [1] (no albedo re-computation).

  • illumination.txt : The estimated illumination (the spherical harmonics basis).

  • visibility.txt : The vertex visibility (see section 3.1 in [1], Visibility calculation). The i-th row indicates the camera IDs (0-based) visible from the i-th vertex.

  • LightCubeMap, LightSphereMap : The visualization of the estimated illuminations for every input image. _Local shows the illumination map in the camera coordinates (the same format in our paper [1]). _World shows the illumination maps w.r.t. the global coordinate system.

  • log.txt : Processing log of the MVIR software.

Third-Party Software

The MVIR binary depends on the following third-party software (note that all needed libraries are included in the distribution zip file):

Dataset

Name Description
The Gajoen 005 10 images to test our programs .
The Joyful Yell 37 synthetic images, its ground-truth camera parameters, initial mesh and ground-truth mesh.

Terms and Conditions

MVIR software is provided for research purposes only. In case you use this software for a publication, cite reference [1], please. See the LICENSE file provided with the demo software.

References

[1] Kichang Kim, Akihiko Torii, Masatoshi Okutomi, "Multi-View Inverse Rendering under Arbitrary Illumination and Albedo", In European Conference on Computer Vision 2016. (PDF)

Contacts