It has to do with the network layers in which the IP and port number are contained in the TCP/IP protocol stack. In
OSI parlance, HTTP operates at layer 7 (application layer) and SSL operates at layers 6 and 5 (presentation/session), with port numbers in the TCP header (layer 4/transport layer) and IP addresses in the IP header (layer 3 / network layer.)
The key thing to keep in mind is that encryption at the higher levels in the protocol stack can't do anything about the fields in the other protocol layers. If you're an apache server using SSL, you're just encrypting the layer 7 (HTTP) content and sending it down to the lower layers. TCP picks up the encrypted SSL data and adds the port number, IP picks up the TCP segment and adds the IP address, and then the data link layer (probably Ethernet) adds the physical address (e.g. Ethernet MAC address) and sends it across the wire. The other side does the same process in reverse, eventually getting to the IP layer, where the packet is directed to the correct IP address, the TCP layer, where the segment is sent to the correct port, and then up to SSL, where the decryption happens.
For the IPs and port numbers to be encrypted, the encryption would have to happen at a lower layer in the TCP stack.