A friend beginning to write fiction asked me how to write effective dialogue. I think I said some platitude or another, something like, "Listen to people," and then changed the subject. Then Jed Hartman wrote up some observations of dialogue, and I got to thinking about it again.
The question has stuck with me because it's so difficult to give any good advice about it, and difficult even to argue about it. For instance, Jed really liked the dialogue in The Strange Adventures of Rangergirl, while I did not. Who's right? Both and neither. Certainly there are times when dialogue is obviously clunky or too expository, etc., but once a writer has moved past a beginner's mistakes, dialogue becomes entirely a matter of "ear" -- the writer's and the reader's. (For some good basic lessons on dialogue, check out Robert Sawyer's advice, Holly Lisle's points and excercises, and this list of pitfalls and exercises.)
To move beyond the most basic points about how dialogue works, we need to think about it as a transaction between the writer and the reader, one that relies on the other elements of the writing. "Effective dialogue" is not a single, universal entity. A style of dialogue that is effective in one story may not be effective in another, and whether dialogue is effective depends not just on the writer's skill, but the reader's sense of tone, as well as how the reader constructs an idea of what function the dialogue serves in the story.
Dialogue is not just about how characters talk. It is, more than anything else, a writer's trick. Dialogue in even the most "realistic" story is not how real people talk -- it is a convention created by the writer and accepted by the reader. The most effective realistic dialogue is dialogue that supports the illusion to such an extent that the reader forgets they are reading and begins to hear the words in their mind's ear, convinced that this is, indeed, how real people talk.
If you want to see the difference between real dialogue and fictional dialogue, record a conversation, and transcribe exactly what you hear. Written down, it will usually seem confusing, repetitive, fragmentary, and, paradoxically, unrealistic (or at least mannered), because our minds are not used to converting pure speech into text.
If writers want to improve their dialogue-writing skills, the best advice may not be to eavesdrop on conversations, but rather to look at a wide variety of fiction and decide what seems effective and what doesn't. Look at something from the 19th century, then something from today. Read a variety of contemporary playwrights -- David Mamet, Caryl Churchill, Tony Kushner, Wole Soyinka, Edward Albee, Suzan-Lori Parks, Wallace Shawn -- to see the range of possibilities, everything from the most colloquial to the most ritualized and declamatory (each is effective when it serves the overall purposes of the play). Compare Elmore Leonard and Alice Munro, Carole Maso and Stephen King, Toni Morrison and Kelly Link, Hemingway and Faulkner, Yasunari Kawabata and Louisa May Alcott, Don DeLillo and E.E. "Doc" Smith, J.M. Coetzee and Grace Paley, The Graduate and JR. And and and.
If you read and pay attention to the dialogue in the work of various writers from different cultures and historical periods, you'll find some that impress you and some you find almost unreadable. This is a test of you as a reader, but you can apply it to the sense of yourself as a writer, because it's likely that the rhythms and styles that most appeal to you as a reader do so because they match the conventions your brain has most deeply absorbed, and that will be the style that is easiest for you to write well in. (I'm not saying you should always write in that style, or that good writers never question and challenge what comes easiest to them, but rather that you might as well start with what is most comfortable and work out from there.)
Dialogue can serve many purposes. It is as much about tone and pacing in a story as it is about conveying information about the situation or characters. A lot of short conversations speed up the reading, while long speeches are generally slower to read, as are conversations broken up by narration. Dialogue filled with content is slower than dialogue that is suggestive and apparently empty. Blunt dialogue is different in effect from subtle or evasive dialogue. The use of speech tags is important, too -- often, we want readers to know exactly who is speaking, but not always, and a strategically deleted "he said" can be useful in some situations: it forces the reader to stop and think, or it plunges them into a confusion that forces them to sort things out, or to see characters as identical. Punctuation is often vitally important: there is a difference between a character whose words dissolve in... and a character whose discussion gets slashed with-- The use or lack of quotation marks or (as James Joyce and Hal Duncan prefer) opening dashes can create different tonal effects. Stories where the dialogue is not set apart from the narrative with quotation marks, such as many of Grace Paley's and Jose Saramago's, feel different from stories that use the conventional punctuation; there is a somewhat dreamier feel, one where the talking and the narrative are not as distanced from each other. Unrealistic, stilted, "literary" dialogue -- dialogue that works against the dominant conventions of realistic fiction -- can be valuable for creating irony or humor, or to highlight disconnections between elements of the fiction.
Certainly for many writers dialogue is a means to a narrative end more than a true skill of its own, but while dialogue is one item among many, it rewards careful thought and attention, because it opens up quite a range of possible effects for a writer sensitive to its potential.