Reading your post something glaringly obvious, but I didn't consider, came to mind....
I don't talk about novel/books often... here, rarely away from here... but I'm old so even though it's 'rare' it's been many times.
What I didn't consider is that IF someone does bring up a book most likely it's a novel that they are quite familiar with. So there's that.
But that doesn't explain how... and I don't know how often this is but it happens... I will bring up a novel in a conversation and the person or person's that I'm conversing with will have great knowledge of the book.
But perhaps that even has an obvious answer: I don't, often, discuss books that weren't best sellers, enormously popular.
I remember once bringing up one of my favorite authors on this board... Umberto Eco... here lies an example of not many people knowing him, remembering him, and even if they do remember one of his novels not understanding him well enough to discuss it. (read ME in that last statement).
BUT... Zooey, remember him? DID. I was amazed.
So anyway... yeah, you've convinced me, I think... some doubts but overall I think you're probably closer to the truth than I was. - JamesJM