#233609 - 16/09/2004 13:57
IP routing question
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
I thought I knew all I would ever need to know about IP routing, but I was wrong.
I am in the process of migrating from one DSL supplier to another. So at the moment I have two DSL connections, one with 16 IP addresses and the other with 32. I have two routers, one connected to each DSL line.
I host my own domains (www, dns, mx, smtp) and a few for other people. All my domains are currently using my servers that are hanging off of ISP1. They will eventually all move over to ISP2, but I had planned that to be a gradual process.
I am finding however that my understanding of IP routing, where you have dual homed servers is probably too naive.
I had thought that if I had a server with two IP addresses, one with ISP1 and the other with ISP2 that I could give it two different names via dns and then access it via those two different names. So it would have two names, say www-isp2.norman.cx and www-isp2.norman.cx which would both work.
It looks like I was wrong and I think I understand why. I think it comes down to the fact that a IP host can only have one default gateway.
Assuming the default gateway is via ISP2, if a request is sent to www-isp2.norman.cx then it gets sucessfully routed all the way to the server via ISP2. However when the server replies it has no way of knowing which route the request took so it just sends the reply back via ISP1 which then gets dropped by a filter somewhere along the line.
Is my understanding correct ?
Is there anything easy I can do about it ? The servers in question are a mixture of Win2k and Linux boxes.
I'm guessing that I could probably solve it using a NAT setup of some sort ? I'm guessing I would need 1-to-1 NAT, which my routers don't do (I suppose I could use the Linux box to do it).
Maybe I should just take the plunge and switch to ISP2 in one go. However I am planning to end up with two DSL connections long term (with a cheap line from ISP3) so it would be good to get a solution to dual homed servers anyway.
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233610 - 16/09/2004 14:31
Re: IP routing question
[Re: andy]
|
pooh-bah
Registered: 12/02/2002
Posts: 2298
Loc: Berkeley, California
|
I think you're overthinking the problem, it should work fine. Any Real OS (and even the Fisher Price(R) one) allows multiple IPs. If a request comes in on one IP it should be smart enough to send it back out on that IP.
Occasionally you'll have to tell the server program which interfaces/addresses you want it to listen to. You may have confused the programs by adding a seccond IP address and not telling it what to do with it. (See: Apache's "listen" directive)
Matthew
|
|
Top
|
|
|
|
|
#233611 - 16/09/2004 15:01
Re: IP routing question
[Re: matthew_k]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Ok.
It isn't working that way though and it is now down to IIS/Apache/bind etc not listening on the right IP addresses, because it effects ping and traceroute as well.
If I ping www-isp2.norman.cx from a remote host and sniff on my local network I see the incoming packet arrive via ISP2. I see www-isp2.norman.cx reply, using the correct ISP2 IP address. The reply never reaches the remote host though.
The only explanation I can think of for this at the moment is that it is being sent out via ISP1 (which is the default gateway).
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233612 - 16/09/2004 15:08
Re: IP routing question
[Re: andy]
|
addict
Registered: 02/04/2002
Posts: 691
|
I had the same problem about a year ago. Switched ISP's, and wanted to put my DNS servers on both networks. If (at least with windows) you have 2 different subnets/gateway addresses, windows has the option to set one as default. The default gateway will be the ISP that all outgoing traffic is set to. This is a huge PITA, one would think an OS could figure out which NIC routes which connection. The way I finally moved, was created ns1 on ISP1, and moved ns2 to ISP2. Each one of these servers I setup to listen to both ns1 and ns2 ips on their ISP's Now, if this doesn’t make any sense, I’m sorry 
_________________________
Oliver
mk1 30gb: 129 | mk2a 30gb: 040104126
|
|
Top
|
|
|
|
|
#233613 - 16/09/2004 16:40
Re: IP routing question
[Re: andy]
|
Carpal Tunnel
Registered: 08/02/2002
Posts: 3411
|
Quote:
If I ping www-isp2.norman.cx from a remote host and sniff on my local network I see the incoming packet arrive via ISP2. I see www-isp2.norman.cx reply, using the correct ISP2 IP address. The reply never reaches the remote host though.
The only explanation I can think of for this at the moment is that it is being sent out via ISP1 (which is the default gateway).
If you're sniffing on the server then it's possible that you'd see the reply coming from the correct address but going via the default gateway where (presumably) it gets dropped upstream because the address appears to be spoofed.
Try adding two routes to 0.0.0.0, one via each gateway, without the default flag, and then remove the existing default route. This should allow the server to choose the appropriate route.
ie;
route del default
route add -net 0.0.0.0 gw isp1.gateway.ip metric 1 dev isp1.ethernet.interface
route add -net 0.0.0.0 gw isp2.gateway.ip metric 1 dev isp2.ethernet.interface
_________________________
Mk2a 60GB Blue. Serial 030102962
sig.mp3: File Format not Valid.
|
|
Top
|
|
|
|
|
#233614 - 16/09/2004 17:37
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
Quote:
I think it comes down to the fact that a IP host can only have one default gateway.
I dunno how it's done in Linux, but in Windows you just create two separate TCP protocols. The trick is that you have to deliberately add the second TCP protocol by hand. This second TCP protocol has its own completely independent set of gateway, netmask, and DNS settings.
|
|
Top
|
|
|
|
|
#233615 - 16/09/2004 17:46
Re: IP routing question
[Re: genixia]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
The "dev" option is a Linux-specific option, and will only work there. In fact, I'm not sure that what you've said will work there, either.
The tecnical term for what you're trying to do is "source-based routing". I'm too lazy to look it up right now. It's too bad that network providers these days are Nazis. You used to be able to stuff like this with no problem.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#233616 - 16/09/2004 18:04
Re: IP routing question
[Re: tfabris]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
Quote:
I think it comes down to the fact that a IP host can only have one default gateway.
I dunno how it's done in Linux, but in Windows you just create two separate TCP protocols. The trick is that you have to deliberately add the second TCP protocol by hand. This second TCP protocol has its own completely independent set of gateway, netmask, and DNS settings.
That sounded very promising. However, when I try and add one TCP/IP is not listed. Appletalk, Netbui etc are there but not TCP/IP.
This is on Win2k server, but I also can't see how to do it on WinXP either.
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233617 - 16/09/2004 18:17
Re: IP routing question
[Re: genixia]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
If you're sniffing on the server then it's possible that you'd see the reply coming from the correct address but going via the default gateway where (presumably) it gets dropped upstream because the address appears to be spoofed.
Yeah, unfortunately it is the only place I can sniff. Both routers and both servers are all connected to the same Ethernet segment.
I tried putting some firewall rules on the routers to spot the packets, but couldn't see them on either router.
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233618 - 16/09/2004 18:20
Re: IP routing question
[Re: andy]
|
Carpal Tunnel
Registered: 08/02/2002
Posts: 3411
|
Buy yourself a cheap hub?
_________________________
Mk2a 60GB Blue. Serial 030102962
sig.mp3: File Format not Valid.
|
|
Top
|
|
|
|
|
#233619 - 16/09/2004 18:21
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
Quote:
This is on Win2k server, but I also can't see how to do it on WinXP either.
Argh, I think it's tricky to do in Win2k server. I remember having a problem adding multiple TCP entries in Win2k. But I remember it was do-able in XP, I just forget how.
They are both fixed addresses, right? I think my problem in Win2k was that I couldn't mix a fixed and a dynamic, but was able to do two "fixed"s.... Only now I don't remember how I did it.
|
|
Top
|
|
|
|
|
#233620 - 16/09/2004 18:22
Re: IP routing question
[Re: genixia]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Eh ? How would a cheap hub help me ?
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233621 - 16/09/2004 18:25
Re: IP routing question
[Re: tfabris]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
I see the deal with it. In Win2k, you only get the one TCP stack, and then hit the "advanced" button and add multiple addresses and multiple gateways to that TCP stack. I don't know how to bind one gateway to one specific address exclusively, I think it just tries one then tries the other.
|
|
Top
|
|
|
|
|
#233622 - 16/09/2004 18:26
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
|
You'd be able to sniff all packets on the seqment. Whereas with a switch you'd only get the packets for that port or the broacast messages. I think that's what he meant.
_________________________
Cheers,
Andy M
|
|
Top
|
|
|
|
|
#233623 - 16/09/2004 18:26
Re: IP routing question
[Re: tfabris]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
I wonder if you could do it with two ethernet cards, having a separate TCP stack for each card? Yeah, I think that works 'cuz that's how our proxy server is set up...
|
|
Top
|
|
|
|
|
#233624 - 16/09/2004 18:33
Re: IP routing question
[Re: andym]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
You'd be able to sniff all packets on the seqment. Whereas with a switch you'd only get the packets for that port or the broacast messages. I think that's what he meant.
I can sniff all the packets on the segment, but that doesn't help me. I can see the packets on the LAN leaving the server, but have no way of seeing (using the packet sniffer) which router they then go through. I can't sniff the other side of the routers.
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233626 - 16/09/2004 18:43
Re: IP routing question
[Re: genixia]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
Try adding two routes to 0.0.0.0, one via each gateway, without the default flag, and then remove the existing default route. This should allow the server to choose the appropriate route.
ie;
route del default
route add -net 0.0.0.0 gw isp1.gateway.ip metric 1 dev isp1.ethernet.interface
route add -net 0.0.0.0 gw isp2.gateway.ip metric 1 dev isp2.ethernet.interface
Spot on. Though I didn't have to delete the original route, just add a new default for the second gateway
Right, Linux down only Win2k to go...
I think I'll go and investigate the Win2k route command (though I have only one interface there so that might be a problem)
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233627 - 16/09/2004 18:48
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
The easiest way to do this, probably, is to have one router for both DSL lines and have that router use some rules to determine where to send the packets. This is pretty easy to do with commercial-grade routers (CIscos and such), but probably impossible with consumer-grade ones. You might be able to do this with a PC running a quality OS but it's going to be more difficult than if you had a "real" router. Here's a page on how to do source-based routing under Linux.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#233628 - 16/09/2004 18:49
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
Quote:
I think I'll go and investigate the Win2k route command (though I have only one interface there so that might be a problem)
It looks promising, if you use the "advanced" button to add another IP address to the TCP stack. Interested in seeing how it works out for you, I'd like to know myself.
|
|
Top
|
|
|
|
|
#233629 - 16/09/2004 18:51
Re: IP routing question
[Re: tfabris]
|
Carpal Tunnel
Registered: 08/02/2002
Posts: 3411
|
I assumed that he was using two cards. I can't see why that should make a difference though, Ok, the linux answer is to use iproute2. Check http://lartc.org/ This has the ability to route traffic according to provider on a dual-home server ( 'Split Access', section 4 of the documention) Can't help with the 'doze.
_________________________
Mk2a 60GB Blue. Serial 030102962
sig.mp3: File Format not Valid.
|
|
Top
|
|
|
|
|
#233630 - 16/09/2004 20:19
Re: IP routing question
[Re: tfabris]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
Quote:
I think I'll go and investigate the Win2k route command (though I have only one interface there so that might be a problem)
It looks promising, if you use the "advanced" button to add another IP address to the TCP stack. Interested in seeing how it works out for you, I'd like to know myself.
I have exhausted my enquires. It looks very much as if this is impossible to do on Windows without a second physical network card.
I had assumed that Windows had the equivalent of Linux's IP Alias functionality, that allows you to add a virtual network interface for just this sort of thing. I can't find any reference to it anywhere though.
Chalk another one up to Linux's network stack...
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233631 - 16/09/2004 20:36
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
Funny thing is, Windows 98 *did* have this. All you had to do was add another TCP entry. Argh.
I have a command-line utility that lets me add a second IP address to Windows 2000, I don't recall if it lets me change parameters like Gateway, but buzz me on ICQ and I'll see if I can get it to you.
|
|
Top
|
|
|
|
|
#233632 - 16/09/2004 20:47
Re: IP routing question
[Re: andy]
|
addict
Registered: 02/04/2002
Posts: 691
|
Even when i had two NICs in my 2000 server box, windows couldn't handle 2 gateways. It would always default to the onboard nic for outgoing packets.
_________________________
Oliver
mk1 30gb: 129 | mk2a 30gb: 040104126
|
|
Top
|
|
|
|
|
#233633 - 17/09/2004 03:14
Re: IP routing question
[Re: andy]
|
old hand
Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
|
if i recall, all you have to do is goto the nic -> properties -> tcp/ip -> properties -> advanced -> and add another ip address and default gateway. thats how to do it in xp and 2k.
|
|
Top
|
|
|
|
|
#233634 - 17/09/2004 03:27
Re: IP routing question
[Re: image]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
Quote:
if i recall, all you have to do is goto the nic -> properties -> tcp/ip -> properties -> advanced -> and add another ip address and default gateway. thats how to do it in xp and 2k.
Right. But how do you tell it: This address uses only this gateway, and that address uses only that gateway?
|
|
Top
|
|
|
|
|
#233635 - 17/09/2004 09:23
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
Chalk another one up to Linux's network stack...
Looks like I was a little hasty in announcing my Linux sucess. I have just discovered that my Linux box has been unreachable from the rest of the world via ISP1 since last night. I've just had to endure a very painful ssh session on my pocket pc to rip out the duff route to fix the problem.
I did test external connectivity last night, but from another box I have access to on ISP2. I guess it was working from there because their "it isn't our traffic" must be filtered out further upstream.
I'll have to have another go over the weekend and this time dial into ISP3 to test it all...
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233636 - 17/09/2004 13:36
Re: IP routing question
[Re: tfabris]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
I have discovered that my Win2k that currently has two different gateways set, one on ISP1 and the other on ISP2 is behaving very oddly.
It is choosing which gateway to use seemingly randomly. If you ping both its IP addresses from a remote host you see returns from one IP address for a few minutes then that stops and you see returns from the other. It switches back and forth every few minutes.
Very odd. Time to go and remove that second gateway.
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#233637 - 17/09/2004 13:48
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Unfortunately, that's expected behavior. I doubt that you have a default gateway assigned to two different interfaces or IP addresses. I'm sure that you've just got two default gateways for the whole machine, and it's choosing which one to use via some internal algorithm that doesn't take source into account.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#233639 - 17/09/2004 14:28
Re: IP routing question
[Re: andy]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
What it probably ought to do is keep track of which gateway responds faster and then use the faster one, probably with some debounce logic in there. As you use one more, the response times for it get slower, as you're using up bandwidth with actual traffic. So it eventually switches over to the other apparently faster one, when it all starts again. Of course, that's pure speculation. I don't think that there's any way to really know what algorithm MS uses, and I'm too lazy to look to see how they do it in BSD, which is probably where MS stole it from to begin with anyway. You might be able to specify a metric, but that wouldn't do anything more than prefer one and end up using it all the time. It might notice when that gateway goes down, and switch over, if you're lucky.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
|
Previous Topic
Index