This paper shows that assessing the economic costs of climate change requires to take into account (i) the specificities of infrastructures, namely a large exposure to climate and a long lifetime that makes it difficult to adapt them to a climate which is changing and uncertain, and (ii) the large reliance of the whole economy on key infrastructures. To do so, this paper proposes a two-sector model in which the first sector represents the production of infrastructure and housing services, and analyses its response to climate change. Then, it proposes a methodology to remove ripple effects in economic models and uses it to assess which fraction of the economic cost of climate change would arise from indirect impacts. Finally, the paper emphasizes (i) that changes in the investment structure can very efficiently reduce the impacts of cross-sectoral ripple effects ; (ii) that, surprisingly, damages are lower when considering infrastructure and housing inertia explicitly, provided optimal changes in the investment structure. Considering the fact that infrastructure and housing are strongly regulated sectors, and most of time heavily supported by governments, these results stress the need for and the efficiency of an adequate public response to reduce climate change impacts.