These thoughts about making players feel threatened were sparked off by Shamus Young's excellent article on The Escapist about the Survival Horror genre. I often use horror elements in my games and I've been told they're quite effective. However, I have real difficulties inspiring a healthy level of dread when I want the players to do things like make extra preparations for combat or run away from a monster.
The reason for this is that I forget what actually inspires players to be nervous or careful, so I wind up rediscovering it all over again every time that I run. Fear in games runs contrary to what most people are used to seeing in, say, movies. In games--even tabletop games--people aren't really afraid that their character will be killed, no matter what they might say about it. The more you warn them that their character may die, the LESS frightened they will be about the prospect. I've found this out, to great annoyance, in circumstances when I put OBVIOUS death-traps in the party's path. Not only were they not frightened into caution, but they were so incautious that they set the trap off on themselves and almost killed the entire party.
I think there are a number of reasons for this. Some players have a tendency to mistake dithering for caution. If it takes a long time for them to come to a decision about what to do (as it did in the above trap situation) they seem to subconsciously assume that some sort of precautions have taken place and thus no harm will result. There is also a tendency for people to forget that they may roll poorly just at the wrong moment. Gaming is like gambling in many respects, in that you're almost always playing the odds and nearly all actions involve some element of risk. The players become inured to this risk over time because they usually do succeed and almost all the time, when they fail, the consequences are negligible. Your character's life usually doesn't depend on a single swing of the sword or damage roll, but you're doing exactly the same thing (rolling the dice), as when your character's life DOES depend on it.
Secondly, the characters exist in a world which is organized so they can succeed--and usually, succeed just via the expedient of not dying. If the GM wants to kill off the characters, there's nothing to stop them. The GM runs the entire universe. If they want to slap in an impossible monster or situation, they can. (Some GM's get a kick out of putting in tricks and letting the players kill their own characters, but this doesn't change the fact that they can't even get as far as the trick without the GM organizing the universe for their benefit.) In computer games, this phenomenon is even WORSE, because even if your character does die, you can just load the game.
So, if the threat of death doesn't inspire dread in a game, what does? The answer that I've seen, over and over, is uncertainty. A huge monster stumbling around in the open, no matter HOW big it is or how many vicious teeth and claws it has, does not inspire fear in itself. You may get a measure of fear if the players aren't sure that they can defeat this monster, which is why the tradition is to hide the creature's statistics. It's the uncertainty that generates nervousness and thus cautious behavior. Likewise a sudden sneak attack inspires no dread. You have to build suspense by first telegraphing some alarming event and then hiding the details. Don't have a big fanged monstrosity stomping around in the open and roaring. Hide it in mist and give the characters only the thought of "something large moving around". This is so effective that they'll continue to be nervous even after you've shone light in the corners and proven there's nothing there.
One video game that did this quite well was the original Gothic. The forests in that game were so nerve-wracking that I avoided them even when my character was quite high-level and could maul just about anything in the game. Why? Because they were built up in this way. In Gothic, the game was full of groups of creatures that, especially early on, could annihilate your character. This wasn't fear-inspiring in itself (it was more annoying, but also rewarding when you managed to clear them out). In the open, though, you could see them quite some distance away and avoid them easily because they didn't move around all that much. Heck, even if you did accidentally anger them, you still had a chance to get away because they'd go through a little "growling at the PC" ritual before they outright charged.
In the forest, however, you couldn't see the creatures. (Once you got past the first rank of trees, they pulled a graphical trick that made it very dark and hid objects even in the middle distance--you had to almost trip over things in order to see them.) Due to the forest noises and the eerie music that played, you often couldn't hear the faint rumble that preceded "I charge you, sir!". It was terribly scary to be in the forest for even a short period.
One last note: this doesn't mean you should just take the expedient of hiding everything in the game in order to make it scary. Pitch black games are not scary, they are annoying. They don't inspire the player to caution, they make them turn up the gamma. In order for this sort of thing to work, there has to be a transition from "I'm safe here, I can see things coming" to "where'd everybody go?" Likewise, in tabletop games, if you never tell the players what is going on, you'll just frustrate them and they'll begin randomly acting out in order to provoke some kind of reaction. All you want is a touch of built-up uncertainty at the moments when they have a strong reason to push forward, and they will shake in their boots.