Multi-View Inverse Rendering under Arbitrary Illumination and Albedo
We present the Multi-View Inverse Rendering (MVIR) method that jointly estimates detailed shape, illumination and albedo. Our MVIR uses the initial shape robustly recovered by multi-view stereo and jointly estimates them by using the geometric and photometric smoothness terms and the normalized spherical harmonics illumination model. Our MVIR allows spatially-varying albedo and per image illumination without any prerequisites such as training data or image segmentation.
Software
- MVIR 2016/10/25 for 64-bit Windows 7 and above (a small update on the visibility calculation).
- MVIR 2016/06/01 for 64-bit Windows 7 and above (ECCV 2016 version).
Dependnecies
Usage
MVIR software requires an initial 3D model (camera poses and 3D mesh). As a basic usage, we create the initial 3D model using VisualSFM and CMPMVS. Please make sure VisualSFM.exe
and CMPMVS.exe
are on the system path. The script generate_dataset.bat
will then run VisualSFM followed by CMPMVS and re-format their outputs for MVIR.
> generate_dataset [images_folder] [mvir_folder]
The [images_folder]
is the root of imagery. The [mvir_folder]
will have images
subfolder, cameras_v2.txt
and mesh.ply
as below.
mvir_folder/
├── images/
│ ├── 00001.jpg
│ ├── 00002.jpg
│ ├── 00003.jpg
│ ├── ...
│ └── XXXXX.jpg
├── cameras_v2.txt
└── mesh.ply
Now ready to run
> mvirc -i [mvir_folder]
Some control parameters are provided for fine-tuning the MVIR. Use mvirc --help
to see the full list of control parameters.
To test the MVIR software, please first try the tiny dataset The Gajoen 005.
Reproduction the result in [1]
You can generate the same result by using the Joyful Yell dataset (section 4.2 in [1]). For this, you need to download our The Joyful Yell dataset and make sure Screened Poisson Surface Reconstruction (Win64, PoissonRecon.x64.exe
) is in your system path, then simply run
> run_joyful [joyful_dataset] [output_folder]
This script automatically perform double MVIR steps with re-computing surface mesh (section 4.2 in [1]) using control parameters which is same to ours. This process requires ~14GB RAM.
Description of output
The MVIR results are saved in MVIR-Output_YYYY-MM-DD_HH-MM-SS
under [mvir_folder]
. The most important files are mesh_result.ply
and illumination.txt
.
MVIR-Output_YYYY-MM-DD_HH-MM-SS/
├── mesh_result.ply
├── mesh_raw.ply
├── illumination.txt
│
├── visibility.txt
├── visibility.ply
├── LightCubeMap_Local/
├── LightCubeMap_World/
├── LightSphereMap_Local/
├── LightSphereMap_World/
└── log.txt
-
mesh_result.ply
: The refined shape (as mesh) and the albedo (as vertex color). The albedo is the one re-computed for improving the visual quality (see section 4.1 in [1]). -
mesh_raw.ply
: The raw output from the optimizaton in equation 1 in [1] (no albedo re-computation). -
illumination.txt
: The estimated illumination (the spherical harmonics basis). -
visibility.txt
: The vertex visibility (see section 3.1 in [1],Visibility calculation
). The i-th row indicates the camera IDs (0-based) visible from the i-th vertex. -
LightCubeMap
,LightSphereMap
: The visualization of the estimated illuminations for every input image._Local
shows the illumination map in the camera coordinates (the same format in our paper [1])._World
shows the illumination maps w.r.t. the global coordinate system. -
log.txt
: Processing log of the MVIR software.
Third-Party Software
The MVIR binary depends on the following third-party software (note that all needed libraries are included in the distribution zip file):
-
OpenCV
http://opencv.willowgarage.com/wiki/
BSD 3-Clause License -
OpenCvSharp
https://github.com/shimat/opencvsharp
BSD 3-Clause License -
Ceres Solver
http://ceres-solver.org/
BSD license -
Armadillo (C++ linear algebra library)
http://arma.sourceforge.net/
Mozilla Public License (MPL) 2.0 -
RPly (ANSI C Library for PLY file format input and output)
Used in external tool (rply-convert.exe).
http://w3.impa.br/~diego/software/rply/ MIT license
Dataset
Name | Description |
---|---|
The Gajoen 005 | 10 images to test our programs . |
The Joyful Yell | 37 synthetic images, its ground-truth camera parameters, initial mesh and ground-truth mesh. |
Terms and Conditions
MVIR software is provided for research purposes only. In case you use this software for a publication, cite reference [1], please. See the LICENSE file provided with the demo software.
References
[1] Kichang Kim, Akihiko Torii, Masatoshi Okutomi, "Multi-View Inverse Rendering under Arbitrary Illumination and Albedo", In European Conference on Computer Vision 2016. (PDF)