Neural Networks (NN) are today increasingly used in Machine Learning where they have become deeper and deeper to accurately model or classify high-level abstractions of data. Their development however also gives rise to important data privacy risks. This observation motives Microsoft researchers to propose a framework, called Cryptonets. The core idea is to combine simplifications of the NN with Fully Homomorphic Encryptions (FHE) techniques to get both confidentiality of the manipulated data and efficiency of the processing. While efficiency and accuracy are demonstrated when the number of non-linear layers is small (e.g. 2), Cryptonets unfortunately becomes ineffective for deeper NNs which let the privacy preserving problem open in these contexts. This work successfully addresses this problem by combining several new ideas including the use of the batch normalization principle and the splitting of the learning phase in several iterations. We experimentally validate the soundness of our approach with a neural network with 6 non-linear layers. When applied to the MNIST database, it competes with the accuracy of the best non-secure versions, thus significantly improving Cryptonets. Additionally, we applied our approach to secure a neural network used for face recognition. This problem is usually considered much harder than the MNIST hand-written digits recognition and can definitely not be addressed with a simple network like Cryptonets. By combining our new ideas with an iterative (learning) approach we experimentally show that we can build an FHE-friendly network achieving good accuracy for face recognition