[...]Well, an obvious problem is that recommendation letters are even more biased than publications. The tone of any letter would strongly depend on personal connections, friendliness, field preferences, etc, of the author. You should just accept that there is no way to objectively linearly order researchers, no matter whether you use journals, letters, or any other approach.
What you can demand is diversity in letter writers. If the candidate has friendly connections with several communities (i.e. in different countries and/or in closely related fields) then some of the bias goes away.
And if some has an Annals paper but cannot get a strong letter writer outside of their country to vouch for them, then it negates the Annals..