We know that hell is not empty from the following passage (among others):
Revelation 20:13-15 (RSV) And the sea gave up the dead in it, Death and Hades gave up the dead in them, and all were judged by what they had done.  Then Death and Hades were thrown into the lake of fire. This is the second death, the lake of fire;  and if any one’s name was not found written in the book of life, he was thrown into the lake of fire.
The lake of fire is hell. Hades (Hebrew, Sheol) is the netherworld where the dead await final judgment (see Luke 16 for a description). Those who are damned will go to hell. And those not written in the Book of Life will go there. This is not speculation. This is biblical prophecy and a description of what occurs in the end times.
“Death and Hades” thrown into the Lake of Fire means in context that unsaved, unrepentant dead people were sent there.
The account in Revelation 20 is one of actual happenings in the future. It’s not a theoretical; it’s factual. Thus, “if any one’s name was not found written in the book of life, he was thrown into the lake of fire.” The “dead were judged” (Rev 20:12), etc.
But there are other cross-references concerning the book of life in Revelation:
Revelation 3:5 He who conquers shall be clad thus in white garments, and I will not blot his name out of the book of life; I will confess his name before my Father and before his angels. [implies a very real possibility of not being included]
Revelation 13:8 and all who dwell on earth will worship it, every one whose name has not been written before the foundation of the world in the book of life of the Lamb that was slain.
Revelation 17:8 The beast that you saw was, and is not, and is to ascend from the bottomless pit and go to perdition; and the dwellers on earth whose names have not been written in the book of life from the foundation of the world, will marvel to behold the beast, because it was and is not and is to come. [two definite assertions that there are people whom God has known would be damned from all eternity, not in the book of life. But God didn’t predestine the damnation; He simply knows about it, in His omniscience]
Revelation 21:27 But nothing unclean shall enter it, nor any one who practices abomination or falsehood, but only those who are written in the Lamb’s book of life. [clear, plain insinuation that the damned (not in the book of life) exist.]
We also have the following passage from St. Paul:
Philippians 4:3 . . . they have labored side by side with me in the gospel together with Clement and the rest of my fellow workers, whose names are in the book of life. [implies that there are those who are not in the book; i.e., the damned or reprobate or lost souls, sentenced to hell]
Old Testament passages also teach the same notion:
Daniel 12:1-3 “At that time shall arise Michael, the great prince who has charge of your people. And there shall be a time of trouble, such as never has been since there was a nation till that time; but at that time your people shall be delivered, every one whose name shall be found written in the book.  And many of those who sleep in the dust of the earth shall awake, some to everlasting life, and some to shame and everlasting contempt.  And those who are wise shall shine like the brightness of the firmament; and those who turn many to righteousness, like the stars for ever and ever. [the saved must be in the book, thus implying that others are not in the book. In the same context, it says flat-out that some of the dead will end up in “shame and everlasting contempt.”]
Exodus 32:32 “But now, if thou wilt forgive their sin — and if not, blot me, I pray thee, out of thy book which thou hast written.” (cf. Dt 29:20) [not as explicit, but appears to presuppose the notions more clearly expressed elsewhere, of a dual possibility of destination after death]
One of the bedrock principles of Bible interpretation — on which all Christians agree — is to interpret the less clear passages from related clear ones.
If I recall correctly, I don’t think that the Bible itself ever uses the phraseology of “hope all men will be saved” or similar. Perhaps I’m mistaken. I just did a search on the online Bible that I use (RSV) for “hope/saved” and “hope/salvation” appearing together, and came up with nothing along these lines.
I think it would be helpful to, alternately, use the terminology of “desiring that all men be saved” (as the Bible says about God) or “praying for all men to be saved” or “Jesus made it possible on the cross for any men to be saved.”
In other words, this is the language of universal atonement (God desires all to be saved, and provides enough grace for that to theoretically happen), rather than the heresy of universalism (all persons are in fact saved in the end). Those statements shouldn’t be controversial at all for anyone who knows their theology.
Explicit passages about judgment, such as Matthew 25, do not imply in the slightest (or so it seems to me, prima facie, anyway) that hell will be a lonely outpost with a few souls; let alone wiped out (as annihilationists, such as Seventh-Day Adventists and Jehovah’s Witnesses believe). I think folks can reasonably debate relative numbers of the saved/elect and the lost, or reprobate, or damned, but I don’t see how it’s biblically possible to posit that only a few or none are lost in fact.