1、次讲座的题目/时间计算机视觉的背景及几何基础 (2/13,第1周)摄像机的几何标定 (3/6,第4周)刚体运动姿态估计问题 (3/27,第7周)姿态估计问题 (II)(或对应问题) (4/17,第10周)应用 (5/8,第13周)要求v听5 次讲座并积极提问,共同讨论(每次有约15-20分钟的提问及讨论时间)v至少完成3个实验中的一个(程序+报告)v(上机地点头两周内定，到时候我通知)v完成一篇(与实验相关的) “学术”论文v最终成绩计算:v本科生: 60%(实验) + 40%(文章)v研究生: 40%(实验) + 60%(文章)纲要v什么是什么是CV? 什么是CV? 它是从什么时候发展起来的
2、?它有哪些研究内容?它与哪些学科/领域相关?CV的若干问题及应用展望 v几何基础概率基础v一些相关资源Definitions of CV (1)v“Today, the study of extracting 3-D information from video images and building a 3-D model of the scene, called computer vision or image understanding, is one of the research areas that attract the most attention all over the w
3、orld.” from K. Kanatani, “Statistical Optimization for Geometric Computation: Theory and Practics”, 1996.CV的定义 (2)v“视觉，不仅指对光信号的感受，它还包括了对视觉信息的获取、传输、处理、存储与理解的全过程信号处理理论与计算机出现以后，人们试图用摄像机获取环境图像并将其转换成数字信号，用计算机实现对视觉信息处理的全过程，这样，就形成了一门新兴的学科计算机视觉计算机视觉”“计算机视觉的研究目标是使计算机具有通过二维图像认知三维环境信息的能力” “计算机视觉计算理论与算法基础”, 马颂德
4、, 张正友, 1998.v“计算机视觉是当前计算机科学研究的一个非常活跃的领域，该学科旨在为计算机和机器人开发出具有与人类水平相当的视觉能力。各国学者对于计算机视觉的研究始于20世纪60年代初，但相关基础研究的大部分重要进展则是在80年代以后取得的。” “http:/ image transformation, image restoration, image enhancement, thresholding, region labelling, and shape characterization.v“Tried to identify and classify objects in im
5、ages by techniques of Pattern Recognition (模式识别模式识别), which had been developed for the purpose of recognizing 2-D characters and symbols by feature extraction and statistical decision making by learning”.v“Many pattern recognition researchers believed that the paradigm of pattern recognition would a
6、lso lead to intelligent vision systems that could understand 3-D scenes”.v“However, they soon realized the crucial fact that 3-D objects look very different from viewpoint to viewpoint beyond the capability of 2-D feature-based learning; 3-D meanings of 2-D images cannot be understood unless some a
7、prior knowledge about the scene is given. Thus, Knowledge came to play an essential role”.v“This type of knowledge-based high-level reasoning is called the top-down (自上而下自上而下) (or goal-driven (目标驱动) approach.” v“In a sense, this approach corresponds to the psychological view toward human perception(
8、感知) that humans understand the environment by unconsciously matching the vast amount of knowledge accumulated from experience in the process of growth.”v“This view can be compared to what is known as the Gestalt psychology, which regards human perception as integration of the environment and experie
9、nce. ”vThus, the problem of how to represent and organize such knowledge became a major concern, and many symbolic schemes were derived. Establishing such symbolic representations is one of the central themes of artificial intelligence (人工智能人工智能), and machine vision was regarded as problem solving b
10、y artificial intelligence.v“However, the inherent difficulty of this approach was soon realized: the amount of necessary knowledge, most of which has the form of “if then else ”, is limitless, heavily depending on the domain of each application (“office scene”, “outdoor scene”, etc) and constantly c
11、hanging (e.g., today, many telephones are no longer black and do not have dials). However large the amount of knowledge is, exceptions are bound to appear, and computation time blows up exponentially as the amount of knowledge increases.”vMany combinatorial techniques were proposed so as to find pla
12、usible interpretation efficiently without doing exhaustive search. Such techniques include various types of heuristic (启发启发式的式的) search as well as special techniques such as constraint propagation (约束繁殖约束繁殖) and probabilistic relaxation (概率松弛概率松弛).v“Realizing that such computational problems are ine
13、vitable as long as knowledge is directly matched with features extracted from raw images, researchers began to pay attention to “physical/optical laws” governing 3-D scenes. In analyzing 2-D images, such laws can provide clues to the 3-D shapes and positions of objects. ”v“For example, the surface g
14、radients of objects can be estimated by analyzing shading intensities (shape from shading). The orientation of a surface in the scene can also be estimated by analyzing the perspective distortion of a texture on it (shape from texture). If objects are moving in the scene (or the camera is moving rel
15、ative to the objects), the 3-D shapes of the objects and their 3-D motions (or the camera motion) can be computed (shape from motion or structure from motion).” v“Although such analyses require appropriate assumptions about surface reflectance, illumination, perspective distortion, and rigid motion,
16、 they do not depend on specific application domains; they are called constraints in contrast to knowledge for the top-down approach. vThis approach is in line with the psychological view toward human vision that human perception occurs automatically when visual signals trigger computation in the bra
17、in and that this computational functionality is innate, acquired in the process of evolution. ”vThis view was asserted by J. J. Gibson, who had a great influence on not only psychologists but also machine vision researchers. vThus, a new paradigm (范例) was established. First, primitive features are e
18、xtracted from raw images by edge detection and image segmentation, resulting in primal sketches; next, approximate shapes and surface orientations are estimated by applying available constraints (shading, texture, motion, stereo, etc.), resulting in 2.5-D sketches; vthen, appropriate 3-D models (e.g
19、., generalized cylinders) are fitted to such data, resulting in a numerical and symbolic representation of the scene; finally, high-level inference is made from such representations. This is called the bottom-up (自下向上自下向上) (or data-driven (数据驱动数据驱动) approach, which is also known as the Marr paradigm
20、 after David Marr, who strongly endorsed this approach. Marr的计算视觉理论框架vMarr从信息处理系统的角度出发，认为视觉系统的研究应分为三个层次，即计算理论层次、表达(representation)与算法层次、硬件实现层次v计算理论层次要回答系统各部分的计算目的与计算策略，亦即各部分的输入输出是什么，之间的关系是什么变换或什么约束v表达与算法层次应给出各部分的输入输出和内部的信息表达，以及实现计算理论所规定的目标的算法.v硬件实现层次要回答“如何用硬件实现以上算法”vA major drawback of this approach
21、 is its susceptibility to noise. Computation solely based on physical/optical constraints is likely to produce meaningless interpretations in the presence of noise. This is because 3-D reconstruction from 2-D data is a typical inverse problem (逆问题), for which solutions are known to be generally unst
22、able with respect to noise. v“In order to cope with this inherent ill-posedness, many optimization techniques were devised so as to force the solution to have required properties. Such techniques are generally called regularization. Other types of optimization include a stochastic relaxation techniq
23、ue called simulated annealing (模拟退火), which was constructed by analogy with statistical mechanics, and the use of neural networks, which gave rise to a new view toward human cognition called connectionism. ”v“Today, many attempts are being made to enhance the reliability of image data. One approach
24、is to actively control the motion of the camera so that the resulting 3-D interpretation becomes stable (active vision). Another approach is using multiple sensors (stereo, range sensing, etc.) and fusing the data (sensor fusion). ”v“In order to fuse data, the reliability of individual data must be
25、evaluated in quantitative terms so that reliable data contribute more than unreliable data.” v“Some researchers are attempting to use only minimum information that is enough to achieve a specific goal such as object avoidance (qualitative vision, purposive vision, etc.). ”v- for detailed information
26、, read “intro_KKanatani.doc”相关领域v数学，物理学v脑科学(或神经生理学)v心理学，认知科学, AI, “计算机视觉发展得益于神经生理学、心理学与认知科学对动物视觉系统的研究，但计算机视觉已发展起一套独立的计算理论与算法独立的计算理论与算法，它并不刻意去仿真生物视觉系统”相关学科与相关课程的联系相关学科与相关课程的联系数字图象处理计算机视觉模式识别机器视觉计算机图形学线性代数集合论高级语言程序设计数据结构先后顺序重叠量反应相关程度基础知识计算机视觉专题(如图象与视觉计算)高等代数最优化方法。信号与系统计算几何Overview (1)v计算机视觉的几何学基础摄像机模型
27、v单摄像机(pinhole model/perspective transformation)v双摄像机 (epipolar geometry: fundamental/essential matrix)v三摄像机及更多(multi-view geometry)运动估计v对应点问题（correspondence problem）v光流计算方法v刚体运动参数估计（minimal projective reconstruction）2-view, 7 points in correspondence; (Faugeras)3-view, 6 points in correspondence; (Q
28、uan Long)3-view, 8 points with one missing in one of the three view. (Quan Long)几何重构（Geometry reconstruction）v立体视觉(stereo vision)vShape from X (shading/motion/texture/contour/focus/de-focus/.)Overview (2)v计算机视觉的物理学基础摄像机及其成像过程v视点、光源、空间中光线、表面处的光线.v明暗 (shading)、阴影 (shadow)光学/色彩 (light/color)v辐射学(radiom
29、etry)，辐照率, , 物体表面特性v漫反射表面（各向同性）Lambertian surfacevBDRF (bi-directional reflectance distribution function)Overview (3)v计算机视觉的图像模型基础摄像机模型及其校准v内参数、外参数图像特征v边缘、角点、轮廓、纹理、形状图像序列特征 (运动)v对应点、光流Overview (4)计算机视觉的信号处理层次v低层视觉处理单图像：滤波/边缘检测/纹理多图像：几何/立体/从运动恢复仿射或透视结构 (affine/perspective structure from motion)v中层视觉处
30、理聚类分割/拟合线条、曲线、轮廓 clustering for segmentation, fitting line基于概率方法的聚类分割/拟合跟踪 trackingv高层视觉处理匹配模式分类/关联模型识别 pattern classification/aspect graph recognitionv应用距离数据（range data）/图像数据检索/基于图像的绘制Overview (5)计算机视觉的数学基础v射影仿射几何、微分几何v概率统计与随机过程v数值计算与优化方法v机器学习计算机视觉的基本的分析工具和数学模型vSignal processing approach: FFT, filt
31、ering, wavelets, vSubspace approach: PCA, LDA, ICA, vBayesian inference approach: EM, Condensation/sequential importance sampling (SIS) , Markov chain Monte Carlo (MCMC) , .vMachine learning approach: SVM/Kernel machine, Boosting/Adaboost, k-NN/Regression, vHMM, BN/DBN (Dynamic Bayesian Network), vG
32、ibbs, MRF, vOverview (6)计算机视觉问题的特点v高维数据的本质维数很低，使得模型化成为可能。High dimensional image/video data lie in a very low dimensional manifold.v解的不唯一性 缺少约束的逆问题v优化问题CV的若干问题及应用展望v基本视觉系统如下：特征检测Shape from X识别图像低层特征位置与形状物体描述 涉及模块与系统的研究存在的问题与出现的一些新 思路，如“视觉信息处理系统的任务”， “关于模块化 问题” ， “局部特征与全局特征” ， “物体建模” ，等等 三维计算机视觉将会有极广泛
33、的应用前景, 如: 计算机人机交互；多媒体技术，数据库与图像通信； 生产自动化；医学；自动导航；三维场景建模与可视化纲要v什么是CV? 什么是CV? 它是从什么时候发展起来的?它有哪些研究内容?它与哪些学科/领域相关?CV的若干问题及应用展望 v几何基础概率基础几何基础概率基础v一些相关资源射影几何知识简介v欧氏几何：旋转和平移都是欧氏变换研究在欧氏变换下保持不变的性质（欧氏性质）的几何是欧氏几何如平行性，长度，角度等都是欧氏性质v射影几何：照相机的成像过程是一个射影（透视或中心射影）的过程它不保持欧氏性质，如平行线不再平行研究射影空间射影空间中在射影变换下保持不变的性质（射影性质）的几何学是
34、射影几何无穷远元素v平行线交于一个无穷远点；v平行平面交于一条无穷远直线；v在一条直线上只有唯一一个无穷远点；v所有的一组平行线共有一个无穷远点v在一个平面上，所有的无穷远点组成一条直线，称为这个平面的无穷远直线维空间中所有的无穷远点组成一个平面， 称为这个空间的无穷远平面射影空间v对n维欧氏空间加入无穷远元素，并对有限元素和无穷远元素不加区分不加区分，则它们共同构成了n维射影空间射影空间.v1维射影空间是一条射影直线，它由欧氏直线和它的无穷远点组成；v2维射影空间是一个射影平面，它由欧氏平面和它的无穷远直线组成；v3维射影空间是由3维欧氏空间加上无穷远平面组成齐次坐标v在欧氏空间中建立坐标系
35、以后，点与坐标有了一一对应，但当引入无穷远点以后，无穷远点没有坐标，为了刻划无穷远点的坐标，可以引入齐次坐标v在n维欧氏空间中，建立直角坐标以后，每个点的坐标为(m1, , mn)，对任意n+1个数x1, , xn, x0，如果满足x00, xi/x0 = mi, (i = 1n)则称(x1, , xn, x0)为该点的齐次坐标齐次坐标而(m1, , mn)被称为非齐次坐标齐次坐标v不全为0的数x1, , xn组成的坐标 (x1, , xn, 0)被称为无穷远点的齐次坐标v例 设在欧氏直线上的普通点的坐标为x，则适合x1/ x0 = x的任意两个数组成的坐标(x1, x0)为该点的齐次坐标，而
36、x为该点的非齐次坐标对任意x1 0，则(x1, 0)是无穷远点的齐次坐标射影参数交比射影变换射影平面中的对偶v“点”与“直线”叫做射影平面上的对偶元素v“过一点作一直线”与“在直线上取一点”叫做对偶作图v在射影平面设有点，直线及其相互结合和顺序关系所组成的一个命题，将此命题中的各元素改为它的对偶元，各作图改为它的对偶作图，其结果形成另一个命题，这两个命题称为平面对偶命题v对偶原则：在射影平面中，若一个命题成立，则其对偶命题也成立调和关系v若点对(P1, P2)和(P3, P4)的交比是-1，即 (P1, P2;P3, P4) = -1,则称(P1, P2)与(P3, P4) 是调和调和的v点对
37、(P1, P2)与(P3, P4) 是调和的当且仅当(1+2)(3+4) = 2(12 +34)其中i分别是Pi (i = 1, , 4)的射影参数完全四点(线)形中的调和关系二次曲线绝对二次曲线(Absolute Conic)极点与极线v对于一个二次曲线C和某个点A(向量)，由L=CA确定的直线(线坐标)称为点A关于二次曲线C的极线极线v当A在二次曲线C上时，点A的极线是过它的切线v对于一个二次曲线C和某条直线L (向量)，由A=C*L确定的点称为线L关于二次曲线C的极点极点v当L是二次曲线C的切线时，线L的极点是它上的切点v对极关系是射影不变的关系，利用这个关系可以对照像机进行标定三维射影
38、几何v点、直线、平面v二次曲面v扭三次曲线：与三维重建中的退化情况紧密联系几何基础的参考书另外,马颂德等的书中有章节简略介绍相关几何知识.“HZintroduction.pdf”有些介绍概率论简介v有关概率论的知识，自学一些 internet 资源vhttp:/www.cs.cmu.edu/cil/v-source.htmlvGoogle search computer visionComputer vision homepageComputer vision Research GroupsCVonlineComputer vision test ImagevPaper searchhttp:
39、/文章来源v主要资源: IEEE TPAMI; ICCV; CVPR; ECCV; ACCV; IEEE Trans. on Robotics and Automation; IEEE TIP; Computer Vision, Graphics and Image Processing (CVGIP); Visual Image Computing; International J. of PRAI; PRv其它资源: SIGGRAPH, ICIP, .v“每年的研究论文不下数千篇,发表的不下数百篇.”Tipsv“Computer Visions great trick is extract
40、ing descriptions of the world from pictures or sequences of pictures.”v“但是，这绝不意味着这些方法就是最优的方法了，也不意味着这些问题已完全解决了相反，目前的方法一般都没有完美地解决视觉信息处理中的问题，它们都或多或少地有些问题，需要进一步的研究，因此，读者只能将这些方法看作是解决一问题的思路或目前已有的较好的方法”v“It is a great time to be studying this subject.”v JUST DO IT!Homeworkv安装并熟悉 Intel OpenCV (可先看一下其FAQ http:/ 或其它开发包/工具箱(如Matlab).v认真阅读课本的前言及handout;v预习第三章及与几何标定相关内容: sturm99.pdf; TR98-71.pdf; calib_eccv.pdf; ZhangPAMI05.pdf; HammarstedtSturmHeyden-iccv05.pdf.vFirst project is hw1.pdf.
