The modeling we have done so far with VRML is for purely synthetic worlds, i.e. environments built by hand or with a modeler. In Image-Based Modeling and Rendering (often abbreviated to IBMR), the environments are generated automatically from the real world using camera images.
The simplest image-based rendering method is the use of imposters. An imposter is an image which is used to texture-map some surface in the environment to simulate a 3D object. Imposters exist in the real world in the form of trompe-l'oeil facades on buildings. In the Tenochtitlan VRML world (select the BallCourt view), imposters are used to represent some trees and some human figures. They are implemented in VRML as billboards. A billboard is a surface that automatically turns to face the viewer. That way, you never see the picture of the tree side-on, or from the back. The imposters in Tenochtitlan are themselves computer generated, but photographs could have been used.
The Facade system is an advanced image-based modeling and rendering system. It uses images in 3 ways:
To compute the rough geometry of objects (buildings) in the scene.
To compute fine geometry automatically using a stereo algorithm
To texture-map the surfaces derived in the first two steps.
For step 1., a user assembles several photographs of the same building. Then they select a primitive (such as a block) and identify edges of the primitive with edges in the photographs. The system is then able to determine the 3D geometry accurately, along with camera positions.
For step 2. surface detail, such as the stone relief, window and door recesses etc. are computed automatically from two or more images using stereo (which is how humans compute depth from two eye images).
For step 3. the images from the cameras are mapped back onto the objects. Because steps 1 and 2 are not perfect, the appearance of the object from different views is not perfectly obtained from a single texture map. Instead Facade uses the texture maps from several views of the same surface, and interpolates between the nearest texture maps to get the actual texture intensities. This gives a very realistic surface appearance from a wide range of viewpoints.
A video will illustrate the working of Facade.
Light field methods are pure image-based methods because they do not attempt to compute the object geometry at all. They simply store the set of all possible light rays emitted by a candidate object under a fixed lighting condition. The set of rays in 3D is a 4 dimensional set. You can parametrize it with a point on the ray and a direction, but more often it is convenient to parametrize the subset of rays passing through two parallel squares.
Let (u,v) be the coordinates of the ray passing through the first square, and (s,t) be the coordinates of the ray passing through the second square. By fixing (u,v) you are considering all the light rays coming from the object and passing through the point p which is on the first plane with coordinates (u,v). This is just a perspective projection of the object to this viewpoint, where the projection plane is the (s,t) plane. So if we think of the entire 4D array as a 2D set of 2D (s,t) images, that corresponds to looking at the object from all viewpoints in the (u,v) plane.
Note that you can generate any view of the object with the light field data, not just views in the (u,v) plane. To do this, notice that rays for a viewpoint behind the (u,v) plane intersect the (u,v) plane at several points. For each of these points, we need a rectangular subset of the (s,t) image at that viewpoint. The light field renderer (video) makes these calculations automatically, allowing a viewer to smoothly move around the object.
Light field data can be generated automatically from a model, or by mechanically taking many views of the object. While the 4D images that result are huge when not compressed, they shrink to an acceptable size (a few megabytes) after compression.