Do all stories need a villain?
Sure, almost any story needs an antagonist, but that antagonist doesn't need to be a person. Since we are talking about speculative fiction, I should add that by "person" I do not necessarily mean "human." A villain can be an alien, a robot, a fairy, a sentient stone, a god, God, or an evil book. However, to be a villain, the antagonist does need to have at least the potential for personality.
Whether the villain has personality or not depends on how well you write him. In my last post on villains, I talked about this, why it's important that a villain be relatable. In this post, I'll talk about how a villain-driven conflict differs from others.
First, let's fall back into high school English class and talk about conflict. According to literature books everywhere, there are seven basic types of conflict: Man against Man, Man against Nature, Man against Self, Man against God, Man against Society, Man against Technology, and Man against the Supernatural.
I'd like to simplify that list a bit, even though it's already a simplification. Man against God, Man against Society, and Man against Technology could all potentially fall under "Man against Nature." In these conflicts, the protagonist is fighting against the overarching way of things. Whether that be the elements of the wild, the pressures of the human world, the unthinking repetition of machinery, or the will of Heaven, the protagonist has to strive against a vast, unindividualized enemy. The enemy does not recognize the character as an enemy; indeed, it does not even set itself against him. Rather, the enemy simply exists, and by that existence, causes conflict for the protagonist.
Similarly, Man against the Supernatural is not significantly different from "Man against Man." If the enemy be a ghost, a vengeful angel, or an evil fairy, there is a single, individuated enemy. It likely knows the protagonists name, or at the very least will learn to hate that name by the end of the story. The enemy has its own sense of self, its own sense of personality, and could easily have a section of the book written from its point of view. If the technology in "Man against Technology" was a robot, or an AI within the realm of human understanding, then it would fall under this conflict. Similarly, if God himself took a personal interest in the protagonist and set himself up as an enemy, then too would that conflict fall here.
So we have three basic conflicts left. I'll even make them politically correct, since I'm already changing things around: Individual against Individual (IvI), Individual against Self (IvS), and Individual against Nature (IvN).
Only one of these conflicts can contain a villain, and that is IvI.
You might think that recognizing a conflict as IvI would be easy, but we're working in speculative fiction, and we love screwing around with straight lines. A good way to tell if a conflict is IvI is to ask a simple question: Could I write a POV from the antagonist's point of view? If so, then you've got yourself an IvI conflict, and you've got yourself a villain. If not, then you're working in another conflict, and you aren't really dealing with the concept of a villain at all.
Sometimes these lines become blurred. Take The Briar King, by Gregory Keyes. The eponymous creature could definitely be described as an antagonist. Certainly the Briar King creates all sorts of trouble for a few of the characters of that book. But could any reasonable POV be written from the Briar King? Not really. He's just . . . there. He exists, and as such, he creates conflict for those characters. That would be an IvN plot, and the Briar King is therefore not a true villain.
The inner system AI's from Charles Stross's Accelerando are not villains either, really. Their existence comes just short of being anathema to the humans living in the outer system, but there would be no way to write anything meaningful from their POV. They are vast, unknowable gods. There is no way to comprehend them, let alone engage in anything that could be called Man versus Man, or Individual versus Individual.
Robert Jordon's Dark One in The Wheel of Time comes close to losing his status as a villain, as well. While we do have evidence that the Dark One has a personality, and occasionally we actually hear him speak, there is little doubt that we are encountering an evil god. The Dark One is not simply a superpowered, evil person, he is something beyond our ken.
Let's go back to high school for a non-sf example. Jack London's "To Build a Fire" has no villain. The conflict is IvN at its most pure. (If you have not read this, you should. It has to be available online somewhere.) Could a POV have been written for the cold, for the snow? No. At least not in any reasonable fashion. Nature, like the Dark One, like the Briar King, is not a villain.
Yes, none of these stories required a villain for their conflict. There could be a perfectly workable plot without one. However, almost all of these examples would be strengthened by the addition of a villain. While the Dark One provides us a certain ineffable terror, we cannot hate him. Not personally. How can we? Might as well hate the drought for thirsting us to death, or the cold for killing our crops. (Yes, I recognize that people hate such things, but there's a difference. You may hate the winter for causing the ice that made the car skid out of control and killed your grandmother, but it is not the same as the hatred you would have for the person who drove her off the road purposefully, or even by accident!)
Without villains, the stories would still work. But would they be as interesting? In most cases, probably not. Readers want an enemy that they can relate to. They want a chance to see the protagonist standing triumphantly above the villain. They want to see the villain realize his mistake, all too late. These things are impossible with an IvN conflict.
The authors I referenced realized this as well, and wrote accordingly. The Wheel of Time has the Forsaken. Though subservient to the Dark One, they are much more human. They can be hated. Accelerando had multiple villains, various little IvI's throughout the larger IvN of the novel. The Briar King also had human villains, or at least human-like ones. They keep the characters busy, and distract them from their primary task of surviving the harsh realities of nature.
Yes, "To Build a Fire" had no villain, and it's a classic. But Jack London is one of the best IvN writers there has ever been, and even then he rarely eschewed IvI completely.
In the end, don't worry if your primary antagonist is something other than a person/evil spirit/robot/transdimensional death-flower. But if it's a anything more than a short, it's probably a good idea to throw a villain in there as well. The villain doesn't have to supply the main conflict, but his presence will definitely liven things up.