Texture Synthesis

In the past decade computer graphics experienced a wave of activity in the area of image-based rendering as researchers explored the idea of capturing samples of the real world as images and using them to synthesize novel views rather than recreating the entire physical world from scratch. This, in turn, fueled interest in image-based texture synthesis algorithms. Such an algorithm should be able to take a sample of texture and generate an unlimited amount of image data which, while not exactly like the original, will be perceived by humans to be the same texture.

In computer graphics, early work on texture synthesis (e.g. the seminal paper by Heeger and Bergen) was inspired by research in texture analysis and very much influenced by statistics (the use of global histograms of image features) and psychophysics (the use of multi-scale, multi-orientation filter banks). While the results for stochastic textures were very good, this class of algorithms was unable to deal well with more structured textures. In our 1999 work ([Efros and Leung,'99]) we tried to address this shortcoming by proposing a very different and extremely simple way of synthesizing textures locally, one-pixel-at-a-time. In recent years this algorithm has been used for synthesiszing a large spectrum of textures as well as filling holes in textured regions. In our latest work ([Efros and Freeman,'01]) we show that similarly good results can be obtained by an even simpler (and much faster) procedure of quilting together patches of the input texutre. Moreover, we demonstrate a method of transfering texture from one object onto another (e.g. rendering a man's face with rice). Click below to see the results and read the articles.

Our Texture Synthesis papers:

  • A. A. Efros and T. K. Leung, "Texture Synthesis by Non-parametric Sampling", ICCV 99
  • A. A. Efros and W. T. Freeman, "Image Quilting for Texture Synthesis and Transfer", SIGGRAPH 01

    • Some further work based on Efros and Leung algorithm (partial list):

  • [Wei and Levoy,'00]: optimizations based on multi-scale and VQ
  • [Harrison,'01]: proposes ordering pixels to be synthesized based on Information Theory, includes examples of texture transfer, and a GIMP plug-in
  • [Ashikhmin,'01]: uses correlation between neighboring pixels to speed-up synthesis, includes examples of texture transfer
  • [Hertzmann et.al.,'01] Image Analogies: a general texture transfer framework (software avaliable)
  • [Efros and Freeman,'01] Image Quilting: stitch together patches of input image, texture transfer
  • [Hertzmann et.al.,'02] Curve Analogies
  • [Levina, '02 PhD Thesis, Ch.2] Proves consistancy of Efros and Leung.
  • [Criminisi et al.,'03] Object Removal: by simply changing the order of synthesis, it is able to fill in hole in arbitrary images.
  • [Wexler'04]: optimization based synthesis, instead of in-order. Inspired PatchMatch (used in Adobe products).
  • PatchMatch: clever speedup of nearest neighbor patch search. Basis for Adobe Photoshop "Context-aware Fill".
  • [Buades 2005]: Non-Local Means fliter for denoising.
  • Surface Texture Synthesis: [Efros,'98 (class project)], [Ying et.al,'01], [Wei and Levoy,'01], [Turk,'01], [Gorla et.al.,'01].

    • Other Work in Texture Synthesis:

  • Kris Popat's PhD thesis
  • David Heeger's Pyramid Based Texture Analysis/Synthesis
  • [Zhu et.al,'97,'98]: FRAME model
  • Jeremy De Bonet's texture synthesis page
  • Eero Simoncelli's texture synthesis page
  • [Zhu et.al.,'00]: Julesz Ensemble
  • [Xu et.al.,'00]: Chaos Mosaic
  • [Schödl el.al,'00]: Video Textures
  • [Bar-Joseph et.al,'01]
  • [Liang et.al.,'01]: Patch-based sampling
  • [Soatto et.al.,'01]: Dynamic Textures
  • [Liu et.al.,'02]: Near-regular Patterns

    • Turns out, these ideas have been around for decades:

  • [Garber,'81] PhD thesis: "Computational Models for Texture Analysis and Texture Synthesis"
  • [Draves,'93]: Fuse program
    • Back to Berkeley Texture page or Alyosha's web page