perhaps it's because the empty reminds us of our insignificance.

we came essentially from nothing, and one day will return to it.

Death, the void, they are unknown. They are other. We fear what we do not understand.

But that said, shouldn't we then fear ourselves? Who can rightly say they understand themself completely?

Themselves? That word always bugs me linguistically speaking.