**Realistic (& Advanced) Image Synthesis** *Fall 2024* -- Instructor: [Derek Nowrouzezahrai](https://www.cim.mcgill.ca/~derek) This course presents modern mathematical models of lighting and shading, and the numerical methods and algorithms needed to solve them and generate beautiful realistic images. Technical content will be delivered using a combination of in-class lectures, tutorial/lab sessions, and (optional) supplemental reading. Learning objectives will be solidified and evaluated through `Python` coding assignments (relying heavily on parallelization, user interfacing, and windowing with [`TaiChi`](https://www.taichi-lang.org).) Administrative Details ======================== The weekly schedule includes two 80-minute lectures on *Wednesdays* and *Fridays*, from 11:35 am to 12:55 pm in *Trottier Building 1100* (TR 1100), and a tutorial/lab session. Tutorial/lab sessions (intertwined) depend on the section you registered in -- please refer to Minerva to ensure you attend the right session. ECSE Course Code | Credit Value ---------------------------|--------------------
ECSE 446
|
3 Credits
ECSE 546
|
4 Credits
Course Overview ================ ![The effects of different reflectance models](images/ift6095/3statues.png width="40%" border="2") This course dives deep into the math and algorithms behind visual effects in modern *feature films* and *games*. After an overview of basic radiometry, we present the mathematical models of appearance and light's propagatation in physical environments. We then develop algorithms to help simulate these processes, generating realistic images. You will code state-of-the-art numerical methods used in visual effects, video games and machine learning. During the tutorial/lab sessions, we will direct you through common implementation stumbling points; however, students are those primarily responsible for overcoming the modeling and coding challenges behind the learning objectives. Please refer to the [topics](#CourseSchedule) and [programming assignments](#Assignments) we'll cover in the class, below. __For many students, this course will provide a first exposure to `Python` and -- arguably more importantly -- vectorization using `numpy` and parallelization using `taichi`. While the teaching staff will provide support during the tutorial/lab sessions, students are expected to commit to (potentially significant) independent at-home coding and debugging efforts during the semester.__ Physics-based Shading | Light Transport Algorithms | ----------------------------------------|---------------------------------------| What is light? What are the physical units and intuitive interpretations behind those quantities most commonly used to measure and represent light?

We'll very quickly establish a common language to use when discussing the physics of light. The emphasis here (as with the entire course) is to build an intuition, rather than unnecessarily complex mathematical constructs.| We'll derive a [rendering equation](http://en.wikipedia.org/wiki/Rendering_equation) that governs the steady-state equilibrium of light energy in a scene. We will generate realistic images of 3D worlds by numerically solving this equation.

Specifically, you'll code numerical solvers based on Monte Carlo integration, the leading family of strategies employed in the graphics industry. The methods we explore also apply to other problems in engineering, physics and statistics: as such, you'll develop a broadly applicable "_numerical toolbox_". |

_The course syllabus and schedule may be revised during the semester. The latest version is always online here, at_ http://www.cim.mcgill.ca/~derek/ecse446.html Requirements ================================================================================ We outline some high-level requirements and expectations, below. Prerequisites (only enforced for ECSE students) --------------------------------------------------- * [*ECSE 202*: Introduction to Software Development](https://www.mcgill.ca/study/2018-2019/courses/ecse-202) * [*ECSE 205*: Probability and Statistics for Engineers](https://www.mcgill.ca/study/2018-2019/courses/ecse-205) and * [*COMP 250*: Introduction to Computer Science](https://www.mcgill.ca/study/2018-2019/courses/comp-250) Generally speaking, we expect students to be comfortable with basic probability concepts (discrete and continuous), differential calculus and linear algebra. You'll be coding quite a bit in this course, so comfort with the build-implement-debug workflow, data structures and algorithms, and basic software engineering will come in handy. The assignments will be implemented in `Python`, relying _heavily_ on the [`TaiChi`](https://www.taichi-lang.org) visualization and parallel programming library. Textbook (available online: [HTML](http://www.pbr-book.org) and [PDF](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering).) --------------------------------------------------- ![Textbook](https://www.pbrt.org/images/bookcover-4ed.jpg width="25%") *Physically Based Rendering: From Theory to Implementation* (_4th ed._)
by Pharr, Humphreys and Jakob - This text is *the* comprehensive reference on realistic image synthesis. The book is packaged with its own open-source renderer (PBRT), and this implementation is exposed in a literate programming style throughout the book.

At over 1500 pages, we only expect students to refer to this book as a supplemental resource for topics covered in class. You can access an e-publication version online for free in a [web-friendly format](http://www.pbr-book.org) and through [the McGill library network](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering). !!! Tip Leveraging *lecture* and *tutorial/lab* time (i.e., consistent attendance, taking notes and asking questions) correlates much more with success than relying solely on the text. As alluded to earlier, sticking to a diligent coding routine that does not leave assignment tasks to the last minute is essential to your success in the course. Equipment --------------------------------------------------- You do not need to buy special computing equipment for this class (i.e., any computer that's less than six years old should suffice). All of our assignments will rely on a baseline `Python` and `TaiChi` environment. Time --------------------------------------------------- Attending and participating in lectures is strongly correlated with the successful retention of the learning objectives: we encourage attending every lecture. It is not unreasonable to plan to invest *nine (9) hours per week* for the 3-credit course or *12 hours per week* for the 4-credit version (this includes lecture, tutorial/lab and at-home coding time) on average. For assignments, it's reasonable to spend roughly *70% of your time coding* -- the remainder will be split between planning, debugging and testing. Start early and schedule appropriately. Attending tutorials/labs with your prepared questions is a good way to skirt any avoidable difficulties. Evaluation ===================================================================== Students will be evaluated based on four (4) programming [assignments](#assignments), to be completed _individually_. ECSE 546 students will have additional deliverables assigned to them in the assignments. Your score relies on the successful completion of the programming tasks, each described below. There is *no midterm or final exam*. A detailed breakdown of each of these graded components follows. Coding Assignments ------------------------------------------------------------ Assignments will use the [`Python`](https://www.python.org/) programming language. Assignments are to be completed individually.

Assignment
|
Weight
|
Due Date
-------------------------------|-------------------------|----------------- *Assignment 1* -- Primary visibility & per-pixel interpolation |
20%
| End Week #3
[September 15] *Assignment 2* -- Basic direct illumination |
25%
| End Week #7
[October 13] *Assignment 3* -- Advanced direct illumination |
25%
| End Week #12
[November 17] *Assignment 4* -- Global illumination |
30%
| End Week #15
[December 8] The assignment handouts -- which will be posted on myCourses during the semester -- detail the required tasks and submission procedures. Assignments must be handed in *no later than 11:59:59pm EST* on their due date. !!! ERROR: Late Policy: Students have __six__ (6) late days that they can use throughout the semester. Assignments handed in after the deadline will deduct from the late day allotment, rounded up to the nearest day. We will not grant extensions beyond the grace period. Once the grace days are exhausted, any subsequent late assignments will receive a grade of 0. Exceptional circumstances will be treated as specified in [McGill's Policies on Student Rights and Responsibilities](https://www.mcgill.ca/students/srr/). Extensions, Solutions and Absences ------------------------------------------------------------ Due to the cumulative nature of the coding assignments, it is essential that everyone begins assignments early, incorporates their submitted assignment feedback and -- when necessary -- track down and correct bugs in old assignments. We will **not provide source code solutions** for assignments -- it's your responsibility to make sure your code is correct, consulting with the TAs **during the tutorial/lab sessions**, as necessary. As [outlined earlier](#LatePolicy) -- once the six grace days are exhausted -- late work will result in a score of 0. Please read the [course schedule](#CourseSchedule) and cross-reference with the [assignment deadlines](#AssignmentDeadlines) to avoid missing deadlines -- especially if you have **planned absences** -- and start work early each week to minimize the impact of **unplanned absences**. !!! ERROR: Online Solutions Do not post solutions online. Refer to the [collaboration & plagiarism policies](#CollaborationPolicy) below. Course Topics & Schedule ===================================================================== The course combines slide-based lectures, hands-on tutorials, and take-home programming assignments. Attendance and in-class participation correlate with success on assignments and in the course. The course will roughly follow the schedule below, modulo adjustments during the semester:
: Welcome - Course administration, outline and schedule - Linear algebra crash review : Ray tracing 1 - Rays, intersections, pinhole camera, perspective - Object- vs. world-space intersection - Interpolating normals : Meshes - Scaling from triangles to meshes - Mesh data: storage and formats - Interpolating vertex data - Acceleration structures : Texture Mapping - Texture mapping parametric surfaces - UV mapping and barycentric coordinates : Texture Mapping - Texture sampling and filtering - Texturing geometric details - Environment and reflection mapping : Radiometry & The Reflection Equation - Definitions and relations - Bi-directional reflectance distribution functions (BRDFs) - The surface reflection equation : Assignment 1 Due Date : Monte Carlo Integration 1 - Probability review - Monte Carlo Integration - Basic estimator design : Monte Carlo Integration 1 - Sampling simple distributions - Ambient Occlusion thought experiments : Monte Carlo Integration 2 - Variance reduction - Importance sampling - Monte Carlo Estimators for Direct Illumination - cosine importance sampling - BRDF importance sampling : Microfacet Reflectance Models - Conductors vs. dielectrics - Macro-, meso-, and micro-scale - Microfacet theory - General microfacet model - instantiations - Layered materials - Oren-Nayar model - Data-driven BRDFs : Monte Carlo Integration 2 - Light Importance Sampling - environment emitters : Monte Carlo Integration 2 - Light Importance Sampling : Monte Carlo Integration 2 - Light Importance Sampling - spherical lights - mesh lights : Monte Carlo Integration 2 - Light Importance Sampling - spherical lights - mesh lights : Assignment 2 Due Date (): Fall Reading Week -- No Lecture (): Fall Reading Week -- No Lecture : Multiple Importance Sampling - Multiple Importance Sampling : Path Tracing & The Rendering Equation - The Rendering Equation - Path tracing - Next-event estimation - Russian Roulette : Bidirectional Light Transport - Light paths - Heckbert notation - Caustics - Duality of light transport : Bidirectional Light Transport - The Measurement Equation - Light tracing - Path integral formulations - Path construction as PDFs - Monte Carlo Estimators - Multiple Importance Sampling - Bidirectional Path Tracing : Photon Mapping - Complications -- SDS Paths - Motivation -- Spatial Regularization - "Backward" Ray Tracing - Description & Limitations - Photon Mapping - Basic Algorithm - Importance Sampling - Relationship to Bidirectional Techniques - Pros/Cons : Photon Mapping - Improving Basic Photon Mapping - Segmenting Maps and Techniques - Progressive Photon Mapping - Convergence Guarantees - Parallelization : Participating Media 1 - Visual effects - Classical transport theory - Defining participating media : Participating Media 1 - Radiative Transport Equation - The Volume Rendering Equation - Volumetric scattering and phase functions : Assignment 3 Due Date : Participating Media 1 - Solving the volume rendering equation - simple special cases - Single scattering - simple transmittance estimation : Participating Media 1 - Volumetric Path Tracing : Participating Media 2 -- Transmittance estimation - Heterogeneous transmittance estimation - Regular tracking - Ray marching - Free-flight transmittance sampling - Null scattering : Volume Photon Mapping - Basic volumetric density estimation - The Beam Radiance Estimate - Photon beams - Progressive photon beams - Higher-order photon primitives (): No Lecture -- classes end for the semester : Assignment 4 Due Date
Collaboration & Plagiarism ===================================================================== *Plagiarism* is an academic offense of misrepresenting authorship. This can result in penalties up to expulsion. It is also possible to plagiarise _your own work_, e.g., by submitting work from another course without proper attribution. **When in doubt, attribute!** We expect you to submit your own work. Assignments are individual tasks. That said, we want to promote an environment where you are comfortable discussing ideas together. **A good rule to follow**: * fully understand every solution you submit, * only submit code that was written by you, and * don't share your code (whether by file or "by eye"). McGill values academic integrity and students should take the time to fully understand the meaning and consequences of cheating, plagiarism and other academic offenses (as defined in the Code of Student Conduct and Disciplinary Procedures -- see [these](www.mcgill.ca/integrity) [two](www.mcgill.ca/students/srr/honest) links). !!! warning: Computational plagiarism detection tools are employed as part of the evaluation procedure of this course. In accordance with article 15 of the Charter of Students' Rights, students may submit any written or programming components in either French or English. If you have a disability, please advise the [Office for Students with Disabilities](www.mcgill.ca/osd) (514-398-6009) as early in the semester as possible. In the event of circumstances beyond our control, the evaluation scheme as set out above may require modification. Additional policies governing academic issues which affect students can be found in the Handbook on Student Rights and Responsibilities.