When we say we miss them, what we really mean is that it is a set of good qualities and experiences that we are missing. It is tenderness, conviviality and open-mindedness we love, first and foremost, rather than, inherently, the bodily home in which these qualities came to rest. We encountered these things in and with them and so assume that to lose the person is to lose everything linked to them.
