If Matt Yglesias can make a list of influential books without the benefit of any readily-apparent aesthetic sense or human soul, surely I can knock this one out of the fucking ballpark. In no particular order, several books (largely fiction) responsible for making me, uh, me (as I understand it):
Tolkien, Lord of the Rings. I first read LotR over several weeks at age 15. That experience of lasting terror and rapturous melancholy is the closest I've come to an authentic sense of the full size and age of the world, which isn't a bad definition of 'religious experience.' Faulkner's line about the past being 'not even past' captures one of the key moral and aesthetic dimensions of Tolkien's work, which aims (among other things) to induce full simultaneous awareness of mythic past and the fallen, fatal present. I learned a lot about the mechanics and purpose of storytelling from this mighty book.
Arthur Clarke, Rendezvous with Rama (and sequels). It said to me: We are not alone - on earth, never mind across the universe - but our neighbours owe us nothing, and all creatures great and small are finally alone in death. The meaning of 'God' is the refusal to turn away from the immensity and darkness of space - not to understand, merely to be without fear. Rendezvous with Rama acknowledges the inscrutability of the universe, the cosmic disinterest that egocentric humans interpret as cruelty, while celebrating curiosity, invention, and the joy of knowledge. Whatever the original novel's weaknesses, however they were compounded by the overwritten and shoddy sequels, the story of Rama was for me a religious text, and instilled in me a powerful love of science - or let's say 'orderly knowledge of disorder' - which is still growing, nearly twenty years later.
Joyce, Ulysses. 'What is love? / One name for it is knowledge'; Ulysses is the most knowledgeable book ever written. Joyce taught me that the greatest complexity can be a means for apprehending the 'simplest' truth, that all is all. There's also a good bit about mommy'n'daddy issues, which seemed like a big deal when I read it in college. Ulysses taught me a new language of joy - and that joy is all language's purpose. This goes with me to the desert island.
Mark Danielewski, House of Leaves. I read this wild book as a junior in college, somewhere between Tropic of Cancer and Ulysses; since I was a child, no story has frightened me as this one did. If you haven't read it, don't read anything about it - just dive in. Incidentally, it's a sly, ironic kiss-off to critical theory too as well as an effective horror story - pity I didn't realize it at the time.
Terry Eagleton, Literary Theory. I went to grad school because of this book and the next one. Back then I didn't know enough to know it was a polemic, but knowing hasn't lessened my affection for Eagleton or this book. Quite the contrary, actually.
Erich Auerbach, Mimesis. I read the opening chapter ('Odysseus's Scar') at age 18 or 19 and it remained my template for What Literary Criticism Is For until...well, recently. Maybe still. Only a decade later did I begin to realize that Mimesis is less a literary history than a social-cognitive one, a map of love (of which 'criticism' is one shy form), a guide to the many revolutions the 'self' has undergone in the West.
Abelson and Sussman, Structure and Interpretation of Computer Programs. Engineering is poetry. Philosophy is poetry. This canonical computer science text has little enough to do with 'computers' as such, and 'science' it really ain't. SICP was the textbook for MIT's introductory CS class, 6.001; few experiences have shaken me as profoundly as my time with Abelson and Sussman's digital arcana.
Shel Silverstein, Where the Sidewalk Ends. If you've read it then you know.
Roald Dahl, Charlie and the Chocolate Factory. This bizarre little moral irony play stands, in my list, as proxy for other creepy 'children's' books like Sideways Stories, Scary Stories to Tell in the Dark, and the illustrated edition of The Night Before Christmas my mom used to read to me at night, year-round. I hope my own kid grows up with dreams as darkly wonderful as mine.