Separating a combined appearance model into a part that deals with ID and a part that deals with residual variation allows classification of ID independently of confounding factors. It also has potential for applications in model-based tracking of faces. Intuitively, we can imagine different dynamic models for each separate source of variability. In particular, given a sequence of images of the same person we expect the identity to remain constant, whilst lighting, pose and expression vary each with its own dynamics.
In practise, the separation between the different types of variation which can be achieved using LDA is not perfect. The method provides a good first-order approximation, but, in reality, the within-class spread takes a different shape for each. When viewed for each individual at a time, there is typically correlation between the identity parameters and the residual parameters, even though for the data as a whole, the correlation is minimised.
For example, we can reason that the correlation between pose and identity must be class specific because of the 3D structure of the head; the way in which the appearance of the nose changes with pose, depends partly on its length - a person-specific quantity, not derivable from a frontal view. Ezzat and Poggio [5] describe class-specific normalisation of pose using multiple views of the same person, demonstrating the feasibility of a linear approach. They assume that different views of each individual are available in advance - here, we make no such assumption. We show that the estimation of class-specific variation can be integrated with tracking to make optimal use of both prior and new information in estimating ID and achieving robust tracking.
We describe in a class-specific linear correction to the result of the global LDA, given new examples of a face.
To illustrate the problem, we consider a simplified synthetic situation in which appearance is described in some 2-dimensional space as shown in figure 4. We imagine a large number of representative training examples for two individuals, person X and person Y projected into this space. The optimum direction of group separation, 
, and the direction of residual variation 
, are shown.
Figure 4: Limitation of Linear Discriminant Analysis: Best identification possible for single example, Z, is the projection, A. But if Z is an individual who behaves like X or Y, the optimum projections should be C or B respectively.
A perfect discriminant analysis of identity would allow two faces of different pose, lighting and expression to be normalised to a reference view, and thus the identity compared. It is clear from the diagram that an orthogonal projection onto the identity subspace is not ideal for either person X or person Y. Given a fully representative set of training images for X and Y, we could work out in advance the ideal projection. We do not however, wish (or need) to restrict ourselves to acquiring training data in advance. If we wish to identify an example of person Z, for whom we have only one example image, the best estimate possible is the orthogonal projection, A, since we cannot know from a single example whether Z behaves like X (in which case C would be the correct identity) or like Y (when B would be correct) or indeed, neither. The discriminant analysis produces only a first order approximation of class-specific variation.
In our approach we seek to calculate class-specific corrections from image sequences. The framework used is the Combined Appearance Model, in which faces are represented by a parameter vector 
, as in Equation 1.
LDA is applied to obtain a first order global approximation of the linear variation describing identity, given by an identity vector, , and the residual linear variation, given by a vector . A vector of appearance parameters, can thus be described by
where 
and 
are matrices of orthogonal eigenvectors describing identity and residual variation respectively. and are orthogonal with respect to each other and the dimensions of and sum to the dimension of . The projection from a vector, onto and is given by
and
Equation 6 gives the orthogonal projection onto the identity subspace, , the best classification available given a single example. We assume that this projection is not ideal, since it is not class-specific. Given further examples, in particular, from a sequence, we seek to apply a class-specific correction to this projection. It is assumed that the correction of identity required has a linear relationship with the residual parameters, but that this relationship is different for each individual.
Formally, if 
is the true projection onto the identity subspace, is the orthogonal projection, is the projection onto the residual subspace, and 
is the mean of the residual subspace (average lighting,pose,expression) then,
where 
is a matrix giving the correction of the identity, given the residual parameters. If is an p by 1 column vector, and an q by 1 column vector, then the matrix is p by q.
During a sequence, many examples of the same face are seen. We can use these examples to solve Equation 8 in a least-squares sense for the matrix , thus giving the class-specific correction required for the particular individual. The vector 
is unknown, but if we assume that the residual correction is linear, then can be found by normalising and about the local means of the sequence, 
, and 
, writing
and
Let 
represent the elements of
The elements of 
and 
are independent and the value of the ith element of is given by
Thus, each row of relates the residual variation, , to one of the identity parameters, 
. If we have N > q examples of the individual face, we can solve for each row, i, of the correction matrix separately. Let 
be a vector of the examples of seen and 
a matrix of the examples of 
seen. Let 
be row i of the correction matrix, then we can write,
This is simply an overdetermined system of linear equations and can be solved for the elements of by standard methods. Having found , we can, given a new example, with measured identity, , and residual variation, , solve Equation 8 to find , the corrected identity.
Each column of describes the effect of each residual parameter on the correction of identity. The magnitude of the column is a measure of how much new information has been learnt about the corresponding residual parameter. For example, if there is very little lighting change in the sequence, those residual parameters corresponding to lighting will have little effect on the correction, and the estimate will revert to the orthogonal projection in that direction.
In each frame of an image sequence, an Active Shape Model can be used to locate the face. The iterative search procedure returns a set of shape parameters describing the best match found of the model to the data. We can also extract the shape-free grey-level parameters from the extracted shape, and thence calculate the combined appearance model parameters.
Baumberg [1] has described a Kalman filter framework used as a optimal recursive estimator of shape from sequences using an Active Shape Model. In order to improve tracking robustness, we propose a similar scheme, based on the decoupling of identity variation from residual variation.
The combined model parameters are projected into the the identity and residual subspaces by Equations 6 and 7. At each frame, t, the identity vector, 
, and residual vector are recorded. Until enough frames have been recorded to allow Equation 13 to be solved, the correction matrix, is set to contain all zeros, so that the corrected estimate of identity, is the same as the orthogonally projected estimate, . Once Equation 13 can be solved, the identity estimate starts to be corrected.
Two sets of Kalman filters are used, one for the corrected identity parameters, in which the underlying model of motion is treated as a zeroth order, or constant position model, and another for the residual parameters, where the motion model is assumed to be first order, or constant velocity. This models the sequence realistically during tracking since the system model treats identity as fixed - something which is certainly true for sequences - and thus the tracking is robust to any noise in the tracking corresponding to apparent change of identity.
We present an example of this system applied to a face sequence. Figure 5 shows frames selected from a sequence, together with the result of the Kalman filter-based Active Shape Model search overlayed on the image. The filter tracks identity as a zeroth order process and residual variation as a first order process. The subject talks and moves while varying expression. The amount of movement increases towards the end of the sequence.
Figure 5: Tracking and identifying a face.
Figure 6 shows the values of the first 3 elements of the corrected identity vector, 
. Also shown are similar results without the class specific correction applied.
It can be seen that the corrected, filtered identity parameters are much more stable than the raw parameters.
Figure 6: First 3 parameters of corrected and uncorrected identity vectors. Parameters are scaled by their respective variance over the training set.
