Course Contents

From a single picture, humans reconstruct a mental representation of the underlying 3D scene that is incredibly rich in information such as shape, appearance, physical properties, purpose, how things would feel, smell, sound, etc. These mental representations allow us to understand, navigate, and interact with our environment in our everyday lives. We learn this from little supervision, mainly by interacting with our world and observing the world around us.

Emerging neural scene representations aim to build models that replicate this behavior: Trained in a self-supervised manner, the goal is to reconstruct rich representations of 3D scenes that can then be used in downstream tasks such as computer vision, robotics, and graphics.

This course covers fundamental and advanced techniques in this field at the intersection of computer vision, computer graphics, and geometric deep learning. It will lay the foundations of how cameras see the world, how we can represent 3D scenes for artificial intelligence, how we can learn to reconstruct these representations from only a single image, how we can guarantee certain kinds of generalization, and how we can train these models in a self-supervised way.

What you will learn

• Computer vision & computer graphics fundamentals (pinhole camera model, camera pose, projective geometry, light fields, multi-view geometry).
• Volumetric scene representations for deep learning: Neural fields & voxel grids.
• Differentiable rendering in 3D representations and light fields.
• Inference algorithms for deep-learning based 3D reconstruction: convolutional neural networks, auto-decoding.
• Basics of geometric deep learning: Representation theory, groups, group actions, equivariance, equivariant neural network architectures.
• Self-supervised learning of scene representations via 3D-aware auto-encoding.
• Applications of neural scene representations in graphics, robotics, vision, and scientific discovery.

For details see the Syllabus.

Prerequisites

No computer vision or graphics specific background is required. We will however generally expect you to:

• have taken a machine learning class with a focus on deep learning
• be comfortable with picking up new mathematics as needed ("mathematical maturity")

We expect you to have a solid knowledge of these specific topics:

• linear algebra,
• multivariate calculus,
• probability theory, and
• programming with vectors and matrices

  (such as in Numpy, PyTorch or Jax)

Grading Policy

Grading will be split between four module-specific problem sets and a final project:

70%	Homework Assignments 4 programming assignments × 17.5% each
30%	Final Project Proposal (5%) + Mid-term Report (5%) + Final Report & Video/Presentation (20%)

We encourage you to discuss the ideas in your problem sets with other students and AI tools, but we expect you to code up solutions individually. For paper presentations and final projects students may group up in teams of two to three. You may use up to 5 late days total for the problem sets (for exceptional situations, contact the course staff).

Schedule

6.S980 will be held as 1.5 hour long lectures in room 4-270:

Click the button above to ask your calendar application to subscribe using a webcal URL. Alternatively you can try this plain HTTPS URL.

Just in case, here are vendor-specific instructions for Google Calendar (under "Use a link to add a public calendar"), for Apple Calendar on macOS, and for Mozilla Thunderbird (under "Subscribe to it on the internet").

Office Hours

Most questions can be answered asynchronously on our piazza discussion forum. For anything else, we hold office hours:

Monday	4:00 – 5:00pm Prof. Sitzmann 32-340 (office hour not for homework questions)
Thursday	4:00 – 5:00pm David Charatan Embodied Intelligence Common Area (outside 32-385) Note that alternate locations may be announced via Piazza. (ask homework questions here)

Course Level

6.S980 is aimed at graduate students and advanced undergraduate students. It's a second-time offering/pilot course and thus scoped as a graduate-level seminar. This class will not count for qualification exams.

Related Courses and Credits

This course is resting on the shoulders of giants, namely, a wealth of courses crafted by amazing lecturers around the world, who all shared their slides with me. Please find a list below, and find the credits for every slide adapted from another course on the slides themselves.

• Learning for 3D Vision
  CMU 16-889, Prof. Shubham Tulsiani
• Advances in Computer Vision
  MIT 6.819/6.869, Profs. Bill Freeman, Phillip Isola, Antonio Torralba
• Deep Learning II, Part "Geometric Deep Learning"
  University of Amsterdam 52042DEL6Y, Prof. Erik Bekkers
• Computer Graphics in the Era of AI
  Stanford CS348I, Profs. C. Karen Liu and Jiajun Wu
• Computer Vision
  University of Tübingen ML-4360, Prof. Andreas Geiger

Image Attribution

Our primary background image is a crop from a work titled "Encounter" by Los Angeles–based photographer Josh S. Rose, shared on unsplash.

It depicts the Department and Water and Power building in Downtown Los Angeles (John Ferraro Building)—we chose the image because it reminded us of diagrams of layered deep learning architectures, 3D view frustums, and its overall abstract aesthetic.