To be fair, there is such a thing as a 3" DVD. All the new DVD-Camcorders use them. Presumably, you could do exactly the same thing with HD-DVD or Blu-Ray, giving up space, of course, in return for size. There were even commercial releases on 3" CDs, way back in the day. I have several CD singles that I bought in 1987. Today, any CD single you buy will still be on a 5" disc.

For both pre-packaged movies and audio, I think the industry simply likes the standard 5" format, and I imagine the insane production volumes make the cost difference between 5" and 3" negligible. And, of most interest to the industry, bigger discs have bigger packaging that is nicely readable on a shelf from a reasonable distance. Remember casette tapes? I don't think anybody in the industry wants to go back to that form factor.

The proper answer, of course, is to get rid of these shiny discs altogether and go with digital downloads...