This paper describes a method for estimating the room impulse response (RIR) for a microphone and a sound source located at arbitrary positions from the 3D mesh data of the room. Simulat- ing realistic RIRs with pure physics-driven methods often fails the balance between physical consistency and computational ef- ficiency, hindering application to real-time speech processing. Alternatively, one can use MESH2IR, a fast black-box estima- tor that consists of an encoder extracting latent code from mesh data with a graph convolutional network (GCN) and a decoder generating the RIR from the latent code. Combining these two approaches, we propose a fast yet physically coherent estimator with interpretable latent code based on differentiable digital sig- nal processing (DDSP). Specifically, the encoder estimates a vir- tual shoebox room scene that acoustically approximates the real scene, accelerating physical simulation with the differentiable image-source model in the decoder. Our experiments showed that our method outperformed MESH2IR for real mesh data ob- tained with the depth scanner of Microsoft HoloLens 2, and can provide correct spatial consistency for binaural RIRs.