This paper presents a method for fitting constrained models in multiple aerial images for building reconstruction applications. Compared to previous works, geometrical constraints are automatically inferred and enforced on the initial approximate model of building and the final constrained model is inherently compliant with the detected constraints through an implicit parameterization. Fitting models in images is performed through automatic matching between model edges and segments detected in the images. An iterative minimization enables to search for the constrained model that minimizes the distance with this image information. Results show that both the introduction of constraints on the model and the use of image information allow a significant gain in the precision of the reconstruction.