I have been on the "verge" of switching to Firefox for a few months. I ran Firefox for quite a while. Since before version 1 final up to just before 1.5 I believe. The frustration with it just kept building and building. It had memory leaks gallore and just got to be very very slow.
Then I switched to Camino which I ran for quite some time.
I only started using Safari after that, with the version 3 betas. The release version of 3 wasn't as good as the betas. The beta never crashed on me. The release version crashed at least once every two weeks for a long time and ate memory like crazy.
Version 4 improves on 3 in terms of speed and sustainability (maintaining that speed after having the app up for a day or more). So far so good.
It lacks plugins. But at the same time, with Firefox, I have to install a bunch of plugins and themes just to get it to be as basic as Safari.
The one big thing I'm missing in Safari right now is a nice Ad Blocker. SafariBlock and Adblock are nowhere near as nice as AdBlock+ for Firefox. Adblock especially really has no controls and has no ability to reformat the pages - you just get huge empty spaces where there should be an ad.
Some features that are built-into Firefox which I dearly miss: Pop-up blocker with exceptions and notifications, ability to disable specific javascript actions (like window moving and resizing).
I'll probably give Chrome a try once it's available in a stable public beta. I'm hoping Apple will see fit to implement tabs as unique processes at some point. Safari still seems very single-threaded from a user-interaction standpoint.