COMP 598 · Topics in AI for Robotics and Intelligent Systems

01Overview

This new course explores selected topics at the intersection of AI and robotics — especially intelligent exploration and information acquisition, and how large language models (also known as foundation models) can be used to make robotics more flexible and effective.

The course does not presume substantial prior exposure to robotics, but comfort with basic AI-relevant algorithms (such as A* search) is expected as well as knowledge of algorithms, linear algebra and basic statistics. About half the course is devoted to traditional lecture-style presentations, and about half to the discussion of recent papers from the research literature.

Because of the seminar-style format, student involvement matters a great deal. For students who are uncomfortable speaking up, some coaching will be included.

A note on course structure. This course is open to both undergraduate and graduate students. Some key aspects of the structure — including the evaluation scheme and the format of certain classes — will only be finalized during the first week, once the composition of the class can be observed.

02Teaching staff

Instructor

NameProfessor Gregory Dudek

Emailgregory.dudek@mcgill.ca

OfficeMC 103

HoursMonday & Wednesday, 11:30 – 12:30
Not held on days with a guest lecturer.

Zoommcgill.zoom.us/my/gdudek — by arrangement

Teaching assistant

NameTo be determined

HoursTo be determined

LocationTo be determined

03Textbook & readings

Primary textbook Computational Principles of Mobile Robotics, 3rd edition

Gregory Dudek and Michael Jenkin. Cambridge University Press, 2024. ISBN 978-1-108-73638-1.

We will be using the third edition, which is longer and more timely than the second or first. Available from Amazon.ca and through the McGill bookstore and library.

Beyond the textbook, the course draws on selected readings from the recent research literature; these are listed lecture by lecture in the schedule below.

Supplementary material

These are excellent books and resources, but they may not all be used directly in this course.

Planning Algorithms — Steven M. LaValle. Cambridge University Press, 2006. Available free online.
Robotics, Vision, and Control — Peter Corke. Springer.
Robotic Manipulation — a very nice online book from the most excellent Russ Tedrake at MIT.
Introduction to Robotics: Mechanics and Control, 4th ed. — John J. Craig. A good book covering the mechanical side, such as dynamics.
Springer Handbook of Robotics — the comprehensive reference for the field.
Refresher notes on probability and statistics — a compact review, from the Stanford CS229 teaching assistants.
Awesome LLM Books — a curated reading list on building with LLMs; pragmatic, and only tangentially related to this course.

04Detailed lecture schedule

Provisional. The schedule will be finalized after the first lecture and before the end of the first week. Timing and sequence may drift as the term progresses, as a function of student interests, emerging issues, and paper-discussion pacing.

#	Date	Lecture	Topics	Supplements	References	Slides
1	Sep 1	Introduction	Motivation, logistics, scope of the field and of the course, and the sense–plan–act paradigm.	Dudek & Jenkin, Ch. 1 Aggressive UAV flight and perching Examples of manipulators (Tedrake)	Dudek & Jenkin, Ch. 1	Intro
2	Sep 3	The diversity of robotics, and how LLMs change the game	Hard and easy problems in robotics. Analytic versus machine-learning alternatives.	Dudek & Jenkin, Ch. 1 A tour of robot hardware (Tedrake)	But what is a neural network? (3Blue1Brown)	middle of deck
3	Sep 8	Introduction to planning	Properties, definitions, and deterministic methods.	Dudek & Jenkin, Ch. 6		last third of deck
4	Sep 10	Planning II	Planning algorithms and procedures; probabilistic planning and RRTs.	Dudek & Jenkin, Ch. 6	LaValle, Ch. 4	TBA
5	Sep 15	Classical control and state estimation	State-space models, PID control, Kalman-filter intuition, and the sense–estimate–act loop.	MIT notes on state estimation	What is a Kalman filter?	TBA
6	Sep 17	POMDPs and belief states	Partial observability, hidden state, observations, beliefs, and information-aware planning.	POMDPs for Dummies	Stanford CS229, Lecture 20 — POMDPs	TBA
7	Sep 22	Deep learning and transformers	Representations, backpropagation, attention, sequence models, and transformers.	Hugging Face: how transformers work	Attention Is All You Need Attention, step by step (3Blue1Brown)	TBA
8	Sep 24	Large language models and foundation models	Pretraining, next-token prediction, prompting, in-context learning, capabilities, and failure modes.	Hugging Face LLM course	Let's build GPT, from scratch (Karpathy)	TBA
9	Sep 29	Multimodal foundation models	Vision–language models, embeddings, contrastive alignment, cross-attention, and multimodal prompting.	CLIP overview	Learning Transferable Visual Models from Natural Language Supervision (CLIP)	TBA
10	Oct 1	3D perception for robotics, in one lecture	Cameras, depth, point clouds, object pose, semantic maps, and scene graphs. Deliberately minimal, since vision is treated elsewhere.	Open3D point-cloud tutorial	Dudek & Jenkin, Ch. 4	TBA
11	Oct 6	Language grounding, affordances, and visual-language navigation	Connecting instructions and semantic descriptions to objects, maps, actions, and navigation goals.	Habitat embodied-AI platform	Vision-and-Language Navigation On Evaluation of Embodied Navigation Agents	TBA
12	Oct 8	Imitation learning, diffusion policies, and VLAs	Learning from demonstration, behaviour cloning, action chunking, diffusion policies, and vision-language-action models.	Diffusion Policy	RT-2: Vision-Language-Action Models	TBA
	Oct 9 – 14	Fall reading break — no lectures
13	Oct 15	World models and sim-to-real transfer	Predictive models, imagined rollouts, physical simulation, domain randomization, and policy transfer.	World Models — interactive article	World Models (Ha & Schmidhuber)	TBA
14	Oct 20	LLMs for hierarchical task planning	Language-to-goal decomposition, symbolic plans, task-and-motion planning, and high-level versus low-level control.	LaValle, Planning Algorithms	Empowering Robot Path Planning with Large Language Models	TBA
15	Oct 22	Planning under uncertainty and execution monitoring	Belief updates, verification, replanning, formal constraints, behaviour trees, and infeasible model outputs.	Underactuated Robotics	Robots That Ask For Help: Uncertainty Alignment for LLM Planners	TBA
16	Oct 27	Failure recovery, safety, and robotic jailbreaking	Failure detection and recovery, adversarial instructions, safety constraints, uncertainty, and provocative current examples.	NIST AI Risk Management Framework	On the Vulnerability of LLM/VLM-Controlled Robotics Generative AI Agents in Autonomous Machines: A Safety Perspective	TBA
17	Oct 29	Multi-robot planning and communication	Task allocation, coordination, communication constraints, swarms, and decentralized planning.	Multi-Robot Task Allocation: Complexity and Approximation	Multi-Agent Pathfinding: Definitions, Variants, and Benchmarks	TBA
18	Nov 3	Student paper discussions I	Grounded planning, navigation, and scene representations.
19	Nov 5	Student paper discussions II	VLAs, imitation learning, and long-horizon manipulation.
20	Nov 10	Student paper discussions III	World models, navigation, and predictive planning.
21	Nov 12	Student paper discussions IV	Safety, jailbreaks, trust, and failure recovery.
22	Nov 17	Student paper discussions V	Multi-agent coordination and field deployment.
23	Nov 19	Student paper discussions VI	Student-selected papers connected to the course themes and to current robotics.
24	Nov 24	Physical deployment, ethics, and project work	Deploying learned and planned behaviours, reproducibility, human oversight, ethics, and project troubleshooting.	NIST AI Risk Management Framework		TBA
25	Nov 26	Final project presentations	Student project presentations and critique.
26	Dec 1	Course recap and conclusion	Connecting perception, planning, learning, foundation models, uncertainty, and safe execution.
27	Dec 3	Reserve slot	Held in reserve for overflow presentations or a topic chosen by the class.

05Assignments and evaluation

Not yet final. The details of the evaluation scheme, and the format of some classes, depend on enrolment and will not be fixed until after the first lecture — based on attendance and the mix of students present. The tentative plan is below.

Two assignments.
A course project.
A possible final exam, potentially oral, pending discussion during the first lecture.

Evaluation will be based on three kinds of activity: class participation, independent work (homework and project), and a possible in-class formal presentation. With substantial enrolment, in-class presentations for every student may not be possible.

06Sharing your presentations

Students presenting papers should upload their slides so the rest of the class can refer to them. Use the upload page below; it goes live once the class is rolling.

www.cim.mcgill.ca/~dudek/598/upload.html

07Policies and technicalities

Academic integrity

Senate, on January 29, 2003, approved a resolution on academic integrity which requires that a reminder to students be printed on every course outline:

Whereas, McGill University values academic integrity; whereas, every term there are new students who register for the first time at McGill and who need to be informed about academic integrity; whereas, it is beneficial to remind returning students about academic integrity; be it resolved that instructors include the following statement on all course outlines:

McGill University values academic integrity. Therefore all students must understand the meaning and consequences of cheating, plagiarism and other academic offences under the Code of Student Conduct and Disciplinary Procedures (see www.mcgill.ca/students/srr/honest for more information).

Be it further resolved that failure by an instructor to include a statement about academic integrity on a course outline shall not constitute an excuse by a student for violating the Code of Student Conduct and Disciplinary Procedures.

Language of submission

In accord with McGill University's Charter of Students' Rights, students in this course have the right to submit in English or in French any written work that is to be graded.