#269089 - 08/11/2005 21:31
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
To be honest, you're not providing quite enough information. The important thing here is whether the external IP addresses are doing incoming NAT for a web server that exists on the private network or if the web server is actually on the machine that has those external IP addresses.
I think my initial assumption was incorrect (that it was NATting the web server), but I added another possibility with an edit. Maybe you read my post before I finished the edit. Go back and see if it makes any sense to you: the unintentional and unneeded NAT of the internal machines to the web server addresses.
Edited by wfaulk (08/11/2005 21:32)
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#269090 - 08/11/2005 21:45
Re: Help: Is my site accessible?
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Quote:
To be honest, you're not providing quite enough information. The important thing here is whether the external IP addresses are doing incoming NAT for a web server that exists on the private network or if the web server is actually on the machine that has those external IP addresses.
The web server is actually on the machine that has those external IP addresses, and normally responds to requests from any addresses.
This is different from how "the masses" do it, because "the masses" would be using a little internet gateway/router to do it all, and they would have no choice but to DNAT their servers. Here, we have a full fledged Linux box, so it can self-host whatever.
Thanks Bitt!
|
|
Top
|
|
|
|
|
#269091 - 08/11/2005 23:40
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Did that help? Nothing else pops into my head about why it wouldn't work, but the fact that the only thing you did to make it break was add an interface doesn't really help my hypothesis.
If that's not it, you could try a network snoop and see what packets happen. The router might be getting confused and sending it out of the wrong interface for some reason, or maybe it doesn't want to generate traffic from external addresses pointed to the internal interface, despite that being a perfectly valid thing to do.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#269092 - 08/11/2005 23:54
Re: Help: Is my site accessible?
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
It's pretty weird.
The same firewall rules work when I deconfigure the second external interface. So I think the firewall is okay.
tcpdump shows the packets arriving at the gateway on the LAN interface, but they appear to die there, even before the Linux iptables firewall gets to see them (it NEVER sees them).
So, a routing problem.
Cheers
|
|
Top
|
|
|
|
|
#269093 - 09/11/2005 00:17
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Huh? You see the packets enter the machine that's intended to receive the packets, but no one answers them? That's not a routing problem. If the packet reaches the machine that it's destined for, routing is done.
Reasons the packet is not being processed could be that the IP stack doesn't think it's destined for that machine or the firewall could be dropping the packet. Sometimes there are rules that drop packets that come in on the "wrong" interface in order to prevent an attacker from sending a packet to your public interface with a private address on it. You might want to look into that.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#269094 - 09/11/2005 00:45
Re: Help: Is my site accessible?
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Quote:
Huh? You see the packets enter the machine that's intended to receive the packets, but no one answers them? That's not a routing problem. If the packet reaches the machine that it's destined for, routing is done.
Not quite. The packet arrives on the LAN interface of the gateway machine, but is not targeted to the LAN IP address, rather it is destined for one of the external IP addresses. As a result, the kernel might think it needs to forward it, or it might be killing it off due to low-level IP filtering. Or at least I think so.
When I first setup the twin interfaces, I had the same issue with packets coming in from the outside --> all incoming connection attempts were being dropped on the floor, and the firewall could NOT see them arriving, even though tcpdump could see them.
I fixed the routing tables (left the firewall config as-was), and that problem went away.
I really don't understand routing, or perhaps it just doesn't happen the way I think it should.
Cheers
Edited by mlord (09/11/2005 00:47)
|
|
Top
|
|
|
|
|
#269095 - 09/11/2005 01:25
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Well, I think you're including kernel IP stack filtering in with routing. I'm not really all that familiar with the Linux IP stack, and whatever they're calling the NAT/firewall module these days changes it a lot anyway. On most OSes I know of, the kernel doesn't care what interface a packet came in on when it receives it; it either deals with it if it's an IP it has, forwards it if not and forwarding is enabled, and otherwise it drops it. Firewalls change that a lot, and not knowing the ins and outs of the firewall you're using, I can't tell you exactly what. Of course, firewalls also modify the OS's normal routing, and it's certainly possible that modifying the firewall routing also modifies other parts of the stack, too. Basically what I'm saying at this point is that all normal, established IP knowhow is thrown out of the window when you're dealing with a firewall and you have to know the ins and outs of the firewall itself.
Nothing you've described is wrong. It's just that the Linux firewall doesn't like it for some reason. I've had virtually the same setup with OpenBSD and not had this problem at all. It's just quirky.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#269096 - 09/11/2005 10:39
Re: Help: Is my site accessible?
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Okay, this is WAAAAYYYY too weird now.
It all works today, after a mere good night's sleep, with no changes made.
Cheers
|
|
Top
|
|
|
|
|
#269097 - 09/11/2005 10:53
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
It all works today, after a mere good night's sleep, with no changes made.
That would be the Routing Fairy at work. You did leave a cross-over cable under your pillow as payment didn't you ?
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#269098 - 09/11/2005 11:04
Re: Help: Is my site accessible?
[Re: andy]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
I guess "ip route flush" doesn't do what I thought it should do, eh! Now I really don't understand routing..
Quote:
That would be the Routing Fairy at work. You did leave a cross-over cable under your pillow as payment didn't you ?
Yes, of course! One of my super special cross-over cables, too!
Edited by mlord (09/11/2005 11:08)
|
|
Top
|
|
|
|
|
#269099 - 09/11/2005 11:19
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 19/01/2002
Posts: 3584
Loc: Columbus, OH
|
Quote:
Cool. Does it work now?
Yes.
_________________________
~ John
|
|
Top
|
|
|
|
|
#269100 - 11/11/2005 02:29
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Okay, today rebooted the server (kernel upgrade), and rebooted my notebook computer too (also a kernel change).
Now, my notebook can no longer ping/access rtr.ca again!
Maybe it will cure itself overnight again.. ?
|
|
Top
|
|
|
|
|
#269101 - 11/11/2005 05:41
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
Maybe it will cure itself overnight again.. ?
Now your Routing Fairy has its own cross-over cable you will need to leave out an alternative offering...
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#269102 - 11/11/2005 05:57
Re: Help: Is my site accessible?
[Re: mlord]
|
pooh-bah
Registered: 13/09/1999
Posts: 2401
Loc: Croatia
|
Works for me...
_________________________
Dragi "Bonzi" Raos
Q#5196
MkII #080000376, 18GB green
MkIIa #040103247, 60GB blue
|
|
Top
|
|
|
|
|
#269103 - 11/11/2005 14:47
Re: Help: Is my site accessible?
[Re: andy]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Hehh Okay, my education in routing continues.. I just discovered the multiple "OUTPUT" chains in the kernel firewall, and found my missing packets getting DROPed on one of them. The routing tables were trying to send replies from my "external IPs" out the external NICs, as they normally should do. But when my internal LAN clients connect to my external IPs, the replies have to be sent back via the internal NIC, not the external NICs. A routing nightmare for a novice such as myself. So I patched in the ipt_ROUTE target module to my kernel, and then did this: Code:
iptables -A OUTPUT -t mangle -s $EXT_IP1 -d $LAN_SUBNET -j ROUTE --oif $LAN_NIC
iptables -A OUTPUT -t mangle -s $EXT_IP2 -d $LAN_SUBNET -j ROUTE --oif $LAN_NIC
And all is well again. I just wish I understood the "ip route/rule" syntax well enough to do it properly that way, rather than via mangle rules in the firewall script. Cheers
|
|
Top
|
|
|
|
|
#269104 - 11/11/2005 15:29
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
The reason you can't do that, as I've said many times, is that it's the firewall (or probably, more precisely, the NATting) that's causing the problem, not the routing. If you didn't have the firewall/NAT in place, it would all work fine as you configured it. Since it's the firewall/NAT that's causing the problem, you have to fix the firewall/NAT to fix the problem.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#269105 - 11/11/2005 15:42
Re: Help: Is my site accessible?
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Quote:
The reason you can't do that, as I've said many times, is that it's the firewall (or probably, more precisely, the NATting) that's causing the problem, not the routing. If you didn't have the firewall/NAT in place, it would all work fine as you configured it. Since it's the firewall/NAT that's causing the problem, you have to fix the firewall/NAT to fix the problem.
That's not consistent with observed behaviour.
The reason I'm having this problem is that my site has TWO external NICs, with individual external IP addresses. To make that work, I had to add source routing rules, to ensure that connections initiated on one of those external NICs, would have their entire connection happen on that same NIC. Otherwise, clients from the internet were unable to visit my servers.
But a consequence of using those routing entries, was that it cut off access to my external IP addresses from within our internel LAN. Even with all firewall rules removed, and the policies set to ACCEPT, internal clients were still unable to access the external IP addresses. EDIT: there is no NAT happening for LAN access to servers running on my external IP addresses.
When I only had one external NIC, the routing table was much simpler, with no source based routing entries. So my internal LAN clients had no issues accessing the external IPs.
So, I've fixed it with a firewall rule kludge, simply because that's the hammer I (mostly) understand. But a routing table fix would be far better.
Cheers
Edited by mlord (11/11/2005 15:50)
|
|
Top
|
|
|
|
|
#269106 - 11/11/2005 15:47
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Oh, and just to add to the confusion:
I also have anti-IP-address-spoofing enabled in the kernel. This was also getting in the way , because my PPPoE DSL modem (the new external IP connection) loops internally to eth2. If I turn on the spoofing filter for eth2, this then prevents the LAN clients from talking with the external IPs of the firewall machine.
This led to the following kernel settings:
Code:
# Enable(1) IP spoofing filters
for nic in /proc/sys/net/ipv4/conf/* ; do
echo 0 > $nic/accept_source_route # disabled source routed packets
echo 1 > $nic/rp_filter # prevent IP spoofing
done
echo 0 >/proc/sys/net/ipv4/conf/all/rp_filter # The DSL nic *must* allow spoofing
echo 0 >/proc/sys/net/ipv4/conf/${DSL_NIC}/rp_filter # The DSL nic *must* allow spoofing
Note that the last two lines were only necessary, because I have a non-zero IP address assigned to the ${DSL_NIC} (eth2), so that I can access the management interface of the DSL modem itself. If instead I used 0.0.0.0 as the IP address (etc..), then I don't thing the spoofing filter would have cared.
Whew!
Edited by mlord (11/11/2005 15:48)
|
|
Top
|
|
|
|
|
#269107 - 11/11/2005 15:56
Re: Help: Is my site accessible?
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Oh, that's the part I was missing. Source-based routing is not a normal part of any standard IP stack. I'm not even sure if it's available on commercial-grade routers by default. It goes against the IP specification. If it's not part of the NAT module for Linux, it's another addition beyond the normal IP stack. That said, I suppose if you have it set up to route all packets from an IP address out of the external interface, of course it's going to go out fo the external interface. You need a rule that would take precedence. I don't know how the source-based routing works under Linux, but see if you can get it to apply only for the default destination route so that any static routes you have (like your directly-connected 10.whatever network) will take predecence in the routing table. It's bound to interact with the normal priority roung system in some way.
In fact, you shouldn't have to do that source-based routing except for that your ISPs (or theirs) have intentionally broken open routing by denying packets whose source addresses aren't in their whitelist.
I suppose it's hard to say which of these things is the most broken.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#269108 - 11/11/2005 18:25
Re: Help: Is my site accessible?
[Re: wfaulk]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Quote:
In fact, you shouldn't have to do that source-based routing except for that your ISPs (or theirs) have intentionally broken open routing by denying packets whose source addresses aren't in their whitelist.
In the UK this applies pretty much all broadband users. British Telecom, who supply the lion's share of ADSLs in the UK decided some time ago to add source filtering to much of their core network, thereby breaking the setup for a whole load of people binding two lines.
_________________________
Remind me to change my signature to something more interesting someday
|
|
Top
|
|
|
|
|
#269109 - 11/11/2005 18:36
Re: Help: Is my site accessible?
[Re: andy]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Yeah, I'd be surprised if there were any residential providers over here who don't block that way. It'd certainly be better to assume that that's the way they do it. That doesn't mean that they haven't intentionally broken standards-based routing on the Internet, though.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
|
Previous Topic
Index
