Sparse Representation


General Abstract:  In practice, signals and data are not unstructured. Their samples usually lie around low-dimensional manifolds and have high correlation among them. Such characteristics can be effectively described by low rankness. As an extension to the sparsity of first order signals, such as voices, low rankness is also an effective measure for the sparsity of second order signals, such as images. In this tutorial, based on our own work, we review the theories, algorithms and applications of the low rank subspace recovery models in signal and data processing.

Part I (by John Wright): Low-Rank Subspace Recovery – Theories

Abstract of Part I:  We present the popular low rank models, including Robust PCA, Robust PCA with Outlier Pursuit, Low-Rank Representation, Latent Low-Rank Representation, and their exact recovery guarantees and closed-form solutions.

Part II (by Zhouchen Lin): Low-Rank Subspace Recovery – Algorithms

Abstract of Part II:  We present efficient algorithms to solve the low rank models. The convex ones include: Accelerated Proximal Gradient, Alternating Direction Method, and Linearized Alternating Direction Method. Nonconvex ones include: Iterative Reweighted Least Squares, Generalized Singular Value Thresholding, and Factorization Method.

Part III (by Zhouchen Lin): Low-Rank Subspace Recovery – Applications

Abstract of Part III:  We present representative applications of low rank models in signal/image/text processing and computer vision.


Zhouchen Lin

School of Electronics Engineering and Computer Science
Peking University, China

Bio:  Zhouchen Lin received the Ph.D. degree in applied mathematics from Peking University in 2000. He is currently a Professor at Key Laboratory of Machine Perception (MOE), School of Electronics Engineering and Computer Science, Peking University. He is also a Chair Professor at Northeast Normal University and a guest professor at Beijing Jiaotong University. Before March 2012, he was a Lead Researcher at Visual Computing Group, Microsoft Research Asia. He was a guest professor at Shanghai Jiaotong University and Southeast University, and a guest researcher at Institute of Computing Technology, Chinese Academy of Sciences. His research interests include computer vision, image processing, computer graphics, machine learning, pattern recognition, and numerical computation and optimization. He is an associate editor of IEEE Trans. Pattern Analysis and Machine Intelligence and International J. Computer Vision, a Senior member of the IEEE, and an area chair of NIPS 2015, ICCV2015 and CVPR2014.

John Wright

Assistant Professor
Department of Electrical Engineering
Columbia University, USA
Slides Video

Bio:  John Wright is an assistant professor of Department of Electrical Engineering, Columbia University, since 2011. He received his Ph.D. from UIUC in 2009. Between 2009 and 2011, he was a researcher at Microsoft Research Asia. He won the best paper award in COLT 2012. He is an area chair of ICCV 2015.

Cloud Computing


Part I (by Zongpeng Li): Virtual Machine Auctions in Cloud Computing

Abstract of Part I:  Cloud computing has emerged as a cost-effective solution to elastic computing. Computing, storage and communication resources in data centers are virtualized and packed into various types of virtual machines (VMs) to serve cloud users. The cloud market is a large and complex one, with VM transactions happen mainly in two fashions: long-term contracts and short term auctions. We will review a series of recent work on the design of short-term VM auctions between the cloud provider and the cloud users, including both one-round auctions and online auctions that span multiple rounds. The desired properties pursued in these VM auction design include truthful bidding, computational efficiency and economic efficiency. The main techniques applied include approximation algorithm design, primal-dual optimization, online algorithm design, and randomized auction design.

Part II (by Hong Xu): Load balancing in data center networks

Abstract of Part II:  Modern data center networks often use Clos topologies such as fat tree or leaf-spine that provide many equal-cost paths between hosts. Load balancing among these paths is thus crucial to the latency performance of millions of mice flows in a large-scale network. This tutorial covers two parts. In the first, I will introduce necessary background about Clos topologies, the de factor load balancing protocol used in commodity switches—ECMP, and the latency problem they cause. In the second I will talk about two new distributed data plane load balancing protocols that improve upon ECMP significantly. The first, RepNet, is an application layer mechanism that replicates each mice flow to distinct paths, and exploits the path diversity of congestion to improve latency. It can be deployed today without modifying hosts or switches. The second, Expeditus, is a distributed congestion-aware load balancing protocol. It monitors congestion information from ordinary data packets, and dynamically chooses the least congested path for each mice flow to improve latency. We present our design and preliminary evaluation of Expeditus that overcomes the fundamental challenge of scalability in a large-scale data center network.


Zongpeng Li

Associate Professor
Department of Computer Science
University of Calgary, Canada

Bio:  Zongpeng Li received his B.E. degree in Computer Science and Technology from Tsinghua University (Beijing) in 1999, his M.S. degree in Computer Science from University of Toronto in 2001, and his Ph.D. degree in Electrical and Computer Engineering from University of Toronto in 2005. Since August 2005, he has been with the Department of Computer Science in the University of Calgary. In 2011-2012, Zongpeng was a visitor at the Institute of Network Coding, Chinese University of Hong Kong. His research interests are in computer networks, including network algorithms, network coding, network function virtualization. Zongpeng was named an Edward S. Rogers Sr. Scholar in 2004, won the Alberta Ingenuity New Faculty Award in 2007, was nominated for the Alfred P. Sloan Research Fellow in 2007, and received the Best Paper Award at PAM 2008 and at HotPOST 2012. In 2014, Zongpeng received the Excellence Award from the Department of Computer Science, University of Calgary.

Hong Xu

Assistant Professor
Department of Computer Science
City University of Hong Kong, China

Bio:  Hong Xu received the B.Eng. degree from the Department of Information Engineering, The Chinese University of Hong Kong, in 2007, and the M.A.Sc. and Ph.D. degrees from the Department of Electrical and Computer Engineering, University of Toronto. He joined the Department of Computer Science, City University of Hong Kong in August 2013, where he is currently an assistant professor. His research interests include data center networking, cloud computing, network economics, and wireless networking. He was the recipient of an Early Career Scheme Grant from the Research Grants Council of the Hong Kong SAR, 2014. He also received the best paper award from ACM CoNEXT Student Workshop 2014. He is a member of ACM and IEEE.

Large-Scale Parallel and Distributed Optimization


Part I (by Wotao Yin): Operator Splitting for Parallel and Distributed Optimization

Abstract of Part I:  Operator splitting breaks a complicated and possible nonsmooth optimization problem into simple steps, which can be easily parallelized or distributed. The resulting algorithms are often very short and easy to implement and exhibit (nearly) state-of-the-art performance for large-scale optimization problems. Operator splitting has led to a large number of recent algorithms in machine learning, compressed sensing, medical imaging, geophysics, and bioengineering. The importance of operator splitting, a technique that dates back to the 1950, has significantly increased in the past decade.

This tutorial will overview the pipeline operator splitting, from identifying simple parts in the original problem, to applying operator-splitting schemes, and to developing parallel and distributed algorithms. We will cover a number of existing algorithms such as von Neumann’s alternating projection, iterative soft-thresholding algorithm, ADMM, various primal-dual algorithms, as well as new algorithms for more complicated problems. The convergence results are presented. Through examples, we also demonstrate that they lead to high-performance low-cost methods for large-scale optimization problems.

This talk includes joint work with Damek Davis, Zhimin Peng, Yangyang Xu, and Ming Yan.

Part II (by Tong Zhang): Stochastic Optimization Techniques for Big Data Machine Learning

Abstract of Part II:  Many modern big-data machine learning problems encountered in the internet industry involve optimization problems so large that traditional methods are difficult to handle. The complex issues in these large scale applications have stimulated fast development of novel optimization techniques in recent years.

I will present an overview of progresses made by the machine learning community to handle these large scale optimization problems, as well as challenges and directions.


Wotao Yin

Department of Mathematics
University of California, Los Angeles, USA

Bio:  Wotao Yin is a professor in the Department of Mathematics of UCLA. His research interests lie in computational optimization and its applications in image processing, machine learning, and other inverse problems. He received his B.S. in mathematics from Nanjing University in 2001, and then M.S. and Ph.D. in operations research from Columbia University in 2003 and 2006, respectively. During 2006 - 2013, he was with Rice University. He won NSF CAREER award in 2008 and Alfred P. Sloan Research Fellowship in 2009.

Tong Zhang

Department of Statistics
Baidu, China; Rutgers University, USA

Bio:  Dr. Tong Zhang is currently directing Baidu's Big Data Lab, and is a professor at Rutgers University. Previously he has worked at IBM T.J. Watson Research Center in Yorktown Heights, New York, and Yahoo Research in New York city. Tong Zhang received a B.A. in mathematics and computer science from Cornell University and a Ph.D. in Computer Science from Stanford University.

Deep Learning and Its Applications in Face Recognition


Part I: Introduction to Deep Learning

Deep learning has become a major breakthrough in artificial intelligence and achieved amazing success on solving grand challenges in many fields including computer vision. Its success benefits from big training data and super parallel computational power emerging in recent years, as well as advanced model design and training strategies. In this part of tutorial, Xiaogang will try to introduce deep learning and explain the magic behind it with layman terms. Through concrete examples of computer vision applications, Xiaogang will focus on four key points about deep learning. (1) Different than traditional pattern recognition systems, which heavily rely on manually designed features, deep learning automatically learns hierarchical feature representations from data and disentangles hidden factors of input data through multi-level nonlinear mappings. (2) Different than existing pattern recognition systems which sequentially design or training their key components, deep learning is able to jointly optimize all the components and crate synergy through close interactions among them. (3) While most machine learning tools can be approximated with neural networks with shallow structures, for some tasks, the expressive power of deep models increases exponentially as their architectures go deep. (4) Benefitting the large learning capacity of deep models, we also recast some classical computer vision challenges as high-dimensional data transform problems and solve them from new perspectives.

Part II: Deep Learning for Face Recognition

In this part of tutorial, Xiaogang will introduce their recent works on deep learning for face recognition. With a novel deep model and a moderate training set with 400,000 face images, 99.47% accuracy has been achieved on LFW, the most challenging and extensively studied face recognition dataset. Deep learning provides a powerful tool to separate intra-personal and inter-personal variations, whose distributions are complex and highly nonlinear, through hierarchical feature transforms. It is essential to learn effective face representations by using two supervisory signals simultaneously, i.e. the face identification and verification signals. Some people understand the success of deep learning as using a complex model with many parameters to fit a dataset. Instead of treating it as a black box, Xiaogang and his team further investigate face recognition process in deep nets, what information is encoded in neurons, and how robust they are to data corruptions. We discovered several interesting properties of deep nets, including sparseness, selectiveness and robustness.

In Multi-View Perception, a hybrid deep model is proposed to simultaneously accomplish the tasks of face recognition, pose estimation, and face reconstruction. It employs deterministic and random neurons to encode identity and pose information respectively. Given a face image taken in an arbitrary view, it can untangle the identity and view features, and in the meanwhile the full spectrum of multi-view images of the same identity can be reconstructed. It is also capable to interpolate and predict images under viewpoints that are unobserved in the training data.

Next, Shiguang Shan will introduce their recent practices on deep learning for face analysis and recognition. One of his team’s works is CNN-based feature learning for face recognition and expression recognition, which helps them win both the PaSC video-based face recognition challenge organized by the IEEE International Conference on Face and Gesture Recognition 2015 (FG’15) and the EmotioW2014 challenge organized by ACM ICMI 2014. Shiguang will introduce how their systems achieved the best results in both challenges.

Most DL models work well only with large-scale training data. Shiguang will then discuss how to learn deep models, even in case only “small” data are available. The basic idea is to approximate the high degree non-linearity by “dividing and conquering” or “piece-wise non-linearity”. Two example practices will be introduced. One is the Coarse-to-Fine Auto-encoder Networks (CFAN) for face alignment, which cascades several deep models. In the other work, titled Stacked Progressive Auto-Encoder (SPAE), the targets of the middle layers are imposed to satisfy the progressive prior of the pose changes. Both works imply that deep models elaborately designed can also work well in case of “small” data.


Shiguang Shan

Key Lab of Intelligent Information Processing(IIP)
Chinese Academy of Sciences, China

Bio:  Shiguang Shan received Ph.D. degree in computer science from the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China, in 2004. He joined ICT, CAS in 2002 and became a Professor in 2010. He is now the Deputy Director of the Key Lab of Intelligent Information Processing of CAS. His research interests cover computer vision, pattern recognition, and machine learning, especially focusing on face recognition related research topics. His work on face recognition has been applied to many practical systems in China. He has published more than 200 papers in refereed journals and proceedings in the related areas, which has been cited more than 6,500 times in Google scholar. He has served as Area Chair for many international conferences including ICCV’11, ICPR’12, ACCV’12, FG’13, ICPR’14, and ICASSP’14. He is workshop co-chair of ACCV14, and website co-chair of ICCV15. He is Associate Editor of IEEE Trans. on Image Processing, Neurocomputing, and EURASIP Journal of Image and Video Processing.

Xiaogang Wang

Assistant Professor
Department of Electronic Engineering
The Chinese University of Hong Kong, China
Slides 1 Slides 2

Bio:  Xiaogang Wang received his Bachelor degree in Electrical Engineering and Information Science from the Special Class of Gifted Young at the University of Science and Technology of China in 2001, M. Phil. degree in Information Engineering from the Chinese University of Hong Kong in 2004, and PhD degree in Computer Science from Massachusetts Institute of Technology in 2009. He is an assistant professor in the Department of Electronic Engineering at the Chinese University of Hong Kong since August 2009. He received the Outstanding Young Researcher in Automatic Human Behaviour Analysis Award in 2011, Hong Kong RGC Early Career Award in 2012, and Young Researcher Award of the Chinese University of Hong Kong. He is the associate editor of the Image and Visual Computing Journal. He was the area chair of ICCV 2011, ECCV 2014, ACCV 2014, and ICCV 2015. His research interests include computer vision, deep learning, crowd video surveillance, object detection, and face recognition.