Automatic video surveillance

E6998 -007 in Spring 2008, 3 credits
Mondays 6:10pm-8pm
Note: Room change: Schapiro CESPR 620 again from 18 Feb onward.

Drs Andrew Senior, Rogerio Feris, Ying-Li Tian

Research Staff Members at IBM T.J.Watson Research Center, {rsferis,yltian} _AT_

Links to: Grading Homework Resources Classes: 1 2 3 4 5 6 7 8 9 10 11 12 13


This course is an investigative survey of the state of the art in automatic video surveillance techniques, taught by three of the researchers who have developed the IBM Smart Surveillance Solution.

Video surveillance is one of the fastest growing areas of commercial application of computer vision and machine learning techniques and a very lively area of academic research.

The course will provide an overview of techniques for visual monitoring, including video surveillance, meeting mining, and human activity understanding. The course examines the basic techniques of processing video from static cameras, starting with object detection and tracking. We progress to examine further video analytic modules, including face detection, trajectory analysis and object classification. We will examine system design and specific problems in visual surveillance, such as the use of multiple cameras and moving cameras. Privacy issues will be examined, focusing on approaches where automatic processing can help protect privacy. Finally the course will look at some specific application domains and commercial systems.

The course will be illustrated throughout with research papers and data from public data bases, with some examples drawn from our own work at IBM.


Data Structures (COMS W3139) required. Any of the following are helpful but not required: Computational Techniques In Pixel Processing (CS 4165), Artificial Intelligence (COMS 4701), Computer Vision (COMS W4731), Visual Interfaces to Computers (CS W4735), or Machine Learning (COMS W4771). Contact instructor if uncertain.

Required text:

None; reprints of journal and conference articles will be used instead. However, a $40 document reproduction fee required.


Assignments (30%) Class participation (10%) and a final project (40%), which may be done in teams of two, following a formal written proposal due week 6 (10%) and part-way project evaluation (10%).

Late Policy

Assignments must be submitted by email to all three of us (email addresses above) by midnight on the date specified.

Late assignments will be penalized according to the following schedule:
0-24 hours 0.9
24-72 hours 0.7
72-168 hours 0.5
>1 week 0.0

Approximate schedule and topics:

Letters after Week numbers are Professor: Senior (S) Feris (F) or Tian (T).
See Columbia Calendar. Subject to change.
28 Jan 1SFT Course overview & grading. Project suggestions. Surveillance Overview and examples. Structure of surveillance systems. Demonstration of the IBM Smart Surveillance Solution.
Slides: Surveillance overview and architectures (Senior)
Slides: Class overview and project ideas (Feris)
4 Feb 2T Object detection: Image differencing, Background subtraction
Slides: Moving object detection part 1
18 Feb 3T Object detection: Advanced background subtraction and alerts
Slides: Advanced Moving object detection
25 Feb 4S Tracking: assignment problem and dealing with splits and merges
Slides: Tracking 1
3 Mar 5S Tracking: other techniques (Mean shift, Condensation, Snakes...) and alerts
Slides: Tracking 2
10 Mar 6F Face detection and tracking --> project proposal due
Slides: Face
Further materials
17 Mar Spring Recess
24 Mar 7F Object classification
31 Mar 8F Behaviour analysis
7 Apr 9T Architecture, Database and user interface. Search/Retrieval. --> project report / mid-evaluation
14 Apr 10S Moving cameras : Active control & processing. Multiple cameras camera hand-off
Slides: Moving/Multiple Cameras
21 Apr 11S Tracking in multiple cameras. Privacy protection & social issues in video surveillance
Slides: Privacy
28 Apr 12T Application domains: public sector, retail, meeting mining. Commercial Systems
5 May 13FT Catch-up. Emerging topics and future directions


Homework 0

Homework 0 is to just tell us about yourself. Please download this questionnaire (.txt) (.doc format), fill it in and email it to us (To all three of us: {aws,rsferis,yltian} @ before class 2.

Homework 1

Write a one page review of Bugeau and Perez Detection and segmentation of moving objects in highly dynamic scenes CVPR2007.

Homework 2

Write a one page review of Comaniciu, Ramesh and Meer Real-time Tracking of Non-Rigid Objects using Mean Shift .

Homework 3

Write a one page review of Vector Boosting .


Potential data sources for research projects:
Andrew Senior home page,