Explosions impact structures by causing deformation due to the overpressure, eventually resulting in cracks opening, fragmentation and ballistic projection of ejecta. In turn, the collapse of the structure can cause leaks of contained fluids. In such extreme deformation events, body-fitted methods are ill-adapted and numerical simulations usually resort to immersed boundary methods. In the present work, we choose to use a conservative cut-cell method in combination with a discrete element method which simplifies the handling of fragmentation. In the present work, we improve the numerical scheme by taking into account contact between solid elements, both from the point of view of the solid and in the handling of fluid-structure interaction. On the solid side, we show that the use of displacement degrees of freedom on the outer boundary faces enables to accurately enforce the contact boundary conditions, while conserving exactly a modified energy over the rebound thanks to the modification of the time integration scheme. Special care is taken of the non-penetration condition with frictionless contact. On the fluid side, we are able to preserve the conservation properties of the coupling scheme by modifying the fluid-solid interface reconstruction in the case of contact and adequately redistributing the fluid quantity swept by the interface over the time-step. Additionally, we address the issue of handling different time-steps for the fluid and the solid, and show how to keep the main properties of the scheme in that case. We will present numerical results demonstrating the robustness and conservation of the method in the case of solid/solid contact and with fluid-structure interaction.