HyperNeRF Explanation

HyperNeRF

• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

• (Park et al., 2021)

• SIGGRAPH Asia Technical Papers

✨ What HyperNeRF Can Do

• https://www.youtube.com/watch?v=qzgdE_ghkaI
• https://hypernerf.github.io/static/images/teaser.mp4
  • Constructing a 3D model from a video with moving viewpoints

🎥 What is “NeRF” in the first place?

• Representing Scenes as Neural Radiance Fields for View Synthesis

• (Mildenhall et al., 2020)

• https://www.youtube.com/watch?time_continue=92&v=JuH79E8rdKc&feature=emb_logo
  • Goal: Generate images from new viewpoints using multiple images from different viewpoints
  • Previous approach before NeRF:
    • Train a neural network to output images
    • ❎ Lack consistency in generating images from different viewpoints
  • NeRF:
    • Train a neural network to output colors and densities (opacity) of arbitrary spatial coordinates instead of images
      • Training data: Images from different viewpoints
      • Inputs: Coordinates x, y, z, viewing angles θ, φ
      • Outputs: Colors R, G, B, density (opacity) σ
      • 
    • Radiance Field: Collection of “colors and densities”
      • Approximate the collection of “colors and densities” with a neural network
      • -> Neural Radiance Field
      • -> NeRF
  • Q. How are images generated from “colors and densities”?
    • A. Simulate the movement of light entering the viewpoint (volume rendering)
      • Accumulate the colors of each coordinate along the path of light
      • Attenuate the light when it passes through areas with high density (opacity)
    • 
      • https://blog.albert2005.co.jp/2020/05/08/nerf/
      • Since it is a differentiable function, it can be represented by a neural network

📕 After NeRF - -

• Research aiming to achieve NeRF for objects with changing shapes
  • https://www.youtube.com/watch?v=MrKrnHhk8IA
  • https://www.youtube.com/watch?v=lSgzmgi2JPw
  • ❎ Struggles with shape changes involving topological variations
    • i.e. Difficult to represent non-continuous changes
    • Examples: “Opening and closing of a mouth,” “Tearing a paper in half,” “Clapping hands”
      • Non-continuous changes occur at the moments when “the mouth closes,” “the paper splits,” “the hands touch”
    • https://hypernerf.github.io/static/images/teaser.mp4

⭐️ HyperNeRF

• Idea: Non-continuous shape changes can be perceived as continuous shape changes in higher dimensions
  • Non-continuous shape changes:
    • 
  • Continuous shape changes in higher dimensions:
    • https://hypernerf.github.io/static/figures/level_set/interpolate2.mp4
    • https://hypernerf.github.io/
• Difference from NeRF:
  • Inputs: Coordinates x, y, z, viewing angles θ, φ, higher-dimensional coordinates W1, W2
  • Outputs: Colors R, G, B, density (opacity) σ
  • 
    • https://hypernerf.github.io/
• 
  • https://hypernerf.github.io/

🔬 Evaluation

• Task 1: Generate images from new viewpoints (unknown coordinates x, y, z, viewing angles θ, φ)
• Task 2: Generate images of new shapes (unknown higher-dimensional coordinates W1, W2)
  • - https://www.youtube.com/watch?v=qzgdE_ghkaI

📖 Summary

• Generate new images with different perspectives and shapes based on videos with changing perspectives and shapes.

• What’s impressive?

  • It can render the images beautifully even when the shapes change in the video.
  • It can render the images without any issues even when the topology of the shape changes.

• References

  • Deep Learning JP. DL Study Group Data2vec: A General Framework for Self-Supervised Learning In…. 4 Feb. 2022, www.slideshare.net/DeepLearningJP2016/dldata2vec-a-general-framework-for-selfsupervised-learning-in-speech-vision-and-language-251106954?next_slideshow=251106954. Accessed 13 Feb. 2022.
  • Mildenhall, Ben, et al. “Representing Scenes as Neural Radiance Fields for View Synthesis.” Communications of the ACM, vol. 65, no. 1, Jan. 2022, pp. 99–106, 10.1145/3503250. Accessed 13 Feb. 2022.
  • Park, Keunhong, et al. “HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields.” ACM Transactions on Graphics, vol. 40, no. 6, Dec. 2021, pp. 1–12, 10.1145/3478513.3480487. Accessed 13 Feb. 2022.
  • Yamauchi. “Neural Representation of 3D Space and NeRF.” ALBERT Official Blog, 8 May 2020, blog.albert2005.co.jp/2020/05/08/nerf/. Accessed 13 Feb. 2022.