SF is often derided by critics as being thin on characterization (which is not quite the same thing as saying it lacks compelling characters -- where would all those fantasy trilogies be without at least a couple such characters, and most of us can name characters from SF stories and novels who have held our imagination for some time). Writers have addressed the issue of characterization at various times, some of them, like Isaac Asimov, saying characterization doesn't need to be the central element of SF stories, while others, such as various people who got labelled as "humanists" in the '80s, have stood up for strong characterization in SF and have exhorted their fellow writers to do better.
I used to agree completely with the characterization-espousers. I thought SF writers needed to work harder to create characters the way mainstream writers do, because often what I most enjoyed in mainstream writing, I thought, was the depth of character, the sense of living along with someone in a story.
However, I think I was wrong. Or, rather, I think I was looking in the wrong direction, and I discovered this when I had trouble defining what it is that differentiated good characterization from bad -- it almost always seemed to have to do with something other than psychology, which is the fundamental engine of the nebulous entity most readers define as characterization of depth and weight.
First, let's think about characterization in mainstream, academically-endorsed literary fiction. Let me go out on a limb and make a wild generalization: the best mainstream writers may create vivid characterizations within their work, but what makes their work great rather than mediocre is seldom the psychology, and what usually makes competent work mediocre is an overattention to psychology.
I've dug myself a pit here from which it may be difficult to climb out, but as evidence I offer a few almost-random names of writers I consider to have worked or be working at the highest level of the art: Nathaniel Hawthorne, Anton Chekhov, Virginia Woolf, William Faulkner, Flannery O'Connor, Ralph Ellison, Donald Barthelme, Alice Munro. (Except for Chekhov, writers whose native language was English, because I want the option here of looking at their exact words.) A sampling, not a representative list of any sort, but it will do.
Many of these writers are acclaimed for their characterization, or at least for characters who have been fascinating and compelling for many readers. Barthelme would be the only clear exception, which is one reason I included him.
What is the nature of the supposedly vivid characterizations, though? With many of those writers, if not all of them, removing the characters from their environment (both physical and social) would leave us with very little to keep us reading. Yes, some of those writers specialized in complex psychological characterization at times, but in every case which comes immediately to my mind, that characterization was either a result of the environment and situation of the story or was directly supported by the writer's flair with language and form.
What SF writers get criticized for when they are criticized for lacking complex characterization is not necessarily "characterization" as a broad term, but rather they are being scolded for not developing enough psychological background details for their characters. Often, their characters are said to be stereotypes -- the strong and stoic starship captain, the intrepid scientist, the muscle-bound swordsman with a heart of gold, etc.
Plenty of great books and, even more so, stories have had characters who were in some way or another stereotypes, though seldom as broad as the ones I described. Hawthorne's work is full of characters who could be judged as stereotypes, as was Flannery O'Connor's. Even Chekhov's. Chekhov is cited often as the epitome of the writer of realistic short stories, but his work, when looked at closely, is only barely that. Even in a long story such as "My Life", he builds his characters, including the protagonist, from a foundation of stereotypes, though we may notice it less today than did readers during Chekhov's era (indeed, much of 19th century Russian literature could be said to be a dialogue with the stereotype of the "dissipated" hero, and "My Life" is an entry in this dialogue). What makes the work interesting and even timeless is the author's attention to a wide variety of details, and the language with which those details are conveyed to the reader.
What we tend to have in the best SF stories are vivid settings, intriguing and even exciting situations, and characters who are the bridge allowing action to develop in those settings and situations. This is not fundamentally different from what characters do in mainstream narratives which claim a certain degree of realism and verisimilitude.
What gets forgotten both by SF fans and by people who denigrate SF in favor of officially literary fiction is that both are operating on a basic grounding of realism, a platform which states that the reader should be brought into the story, made to identify with or feel for the characters, and care about the action. In this way, SF and traditional mainstream fiction share many techniques and values, though SF generally has to work harder at it because the reader needs not only to identify with and believe in the characters, but in an entire imaginary world.
The writers I've listed above, and most of the writers I find most interesting throughout the history of writing, whether SF or not, have not clung to the above ideas as a be-all and end-all, but rather as a starting point. Great writers pay attention to everything that matters within their work, and often, whether consciously or unconsciously, this brings them back to having only one central value: language.
Language is the writer's raw material, and a writer must use it the way a potter uses clay. Shaping the language and working with it is what produces great art. The patterns of imagery and event all originate with words. No other value is absolute.
Scooting myself out onto a shaky limb once again, here's another proposition: Writing which fails, or which is merely mediocre, is so not because of a lack of psychological detail, which any story can live without, but because of a lack of attention to language and how it is structured.
Words create sentences which create (usually) paragraphs, and how those words are chosen and then arranged, how sentences are constructed and then positioned into paragraphs, and the relationship paragraphs have to each other -- this is what determines the success of a story, because all other values are contained within the words, sentences, and paragraphs.
It may be that this is an incredibly stupid observation (though it is one which Gertrude Stein might have supported, having said, "Sentences are not emotional but paragraphs are.") But I find it much more helpful to use this potentially stupid observation as a lens through which to evaluate how writing works, because it moves away from abstract concepts posing as eternal verities ("characterization") and lets us look at the most basic material of storytelling.
To state my position more specifically: A writer who employed all of the psychological material of William Faulkner's As I Lay Dying would not create characters who were as vivid as Faulkner's, because what allows Faulkner's characters to be vivid is the situation in which they are placed and, perhaps more importantly, the manner with which the reader gains information about the characters -- the diction and structure of the sentences, the placement of the sentences in relation to each other, the length and order of the paragraphs, and the shape of the chapters. The success of Faulkner's form is inextricable from the success of the content. Even with a writer such as Chekhov who was not formally innovative in the way Faulkner was seems to me to succeed because of the same considerations -- indeed, read any of his stories in multiple translations, and you will see how important syntax and diction are to the power of the tales, even when the overall structure is maintained. (Perhaps, then, we should say that we read translators as much as we read the writers they translate.)
Alice Munro's best work seems more vivid and compelling than that of so many of her contemporaries because of the way she employs English -- the words she chooses and the way she arranges them. Flannery O'Connor could easily be convicted of creating thin characters, except that the language those characters are created with is (in her best work) vivid and precise. Donald Barthelme wrote entire stories without anything resembling good characterization, and his stories are effective and brilliant because of the words, the sentences, and the way those words and sentences convey information.
I don't think I'm going to banish the word "characterization" from my vocabulary as either a reader or a writer, because at times it has its uses, but I am going to remind myself to look more closely at the language which allows that characterization its illusions.