Patch models have proven successful to solve a variety of inverse problems in image restoration. Recent methods, combining patch models with a Bayesian approach, achieve state-of-the-art results in several restoration problems. Different strategies are followed to determine the patch models, such as a fixed number of models to describe all image patches or a locally determined model for each patch. Local model estimation has proven very powerful for image denoising, but it becomes seriously ill-posed for other inverse problems such as interpolation of random missing pixels or zooming. In this work, we present a new framework for image restoration that makes it possible to use local priors for these more general inverse problems. To this aim, we make use of a hyperprior on the model parameters which overcomes the ill-posedness of the local estimation and yields state-of-the-art results in problems such as interpolation, denoising and zooming. Moreover, taking advantage of the generality of the framework, we present an application to the generation of high dynamic range (HDR) images from a single snapshot. Experiments conducted on synthetic and real data show the effectiveness of the proposed approach.