More on World-Building (or Starting an Internet Fight)

In the comments of my world-building post yesterday, Kelly thought I was being unfair with my generalized statement that ‘no one reads for world-building.’  I’ve been thinking about this, and, of course, if Kelly says he does this then, obviously, I’m wrong. 
But, since he pointed it out, I've been trying to imagine a story where this might be true for me, and I keep failing.  I can’t picture a novel I’ve read where I thought, “Wow, these character suck, the plot is inane, but, damn!  I’m going to keep reading because this house they live in is BRILLIANT.”
Yet, even as that came out of my mouth, I thought, “Well, maybe I did that when I read the Thieves’ World books, way back when.”  The idea behind the Thieves’ World was that it was, in point of fact, a cool underground to a high fantasy world and that a bunch of authors agreed to write in it.  But, even as I considered that example, I wondered if the world-building was the real appeal or what the people did in it (ie, the plot.)  I actually can’t say definitively.  I can say that the only things I remember about the anthologies after all these years is that the place they all lived was “The Sanctuary” and that I loved the characters of Shadowspawn and Tempest.
That bit of information makes me think that, ultimately, the world-building failed to impress.  If I’m left with characters after all this time, I must have been reading it on some level for the characters—or, if I was reading for world, it didn’t stick.  Decades later I remember almost no other details about the world besides the obvious (its name and the fact that the main population were thieves and underworld figures.)
I suspect that some people will tell me that the Harry Potter universe fits into the “world-building trumps character and plot,” but I suspect people who might say that bounced off the book and are angry because they’re successful when they don’t seem to be, according to many, terribly well written.  I actually agree that there are moments when JK Rowling’s words repetitive and cliché, but I think that what drew me in was perhaps EQUAL parts character, plot and world-building.  I mean, I do spend a lot of time thinking about where I’d be sorted, which is clearly a product of world-building success. But, would the books have been successful if the rest failed?  I’m not sure.  I mean, some people clearly think it did. 
But would I read a book that only hit world-building? 
Well, maybe Kelly’s point is that maybe I wouldn’t, but hewould.  I guess I’m curious, then, if youwould read a book that only worked on the world-building front?  If the characters and the world were meh-to-irritating, would you keep going if you felt the world was sufficiently interesting? And, can you name a title of a book where you did that?