[linux] Connections time out, need help [SOLVED]

[Brad Willson]

As Paul Harvey would say "Now for the rest of the story..."

Two firewalls were humming along seemingly content with each other's
presence until 10:09am Friday. Something happened to start them flapping
and everything after that was bad news. Several services were blocked
while others worked as normal. The quick fix became clear today when I
turned off one of them resulting in a total blockage of the services
behind the other. I turned off the offending firewall, turned on 'the
good one' and now everything is back to normal. The solution going
forward will be to mirror the Heartbeat/LVS setup I implemented in our
development subnet. Only one firewall is online at any time so the whole
contention issue is eliminated. The haresources file is a few thousand
lines long, but well worth the effort.

To address your bits Meryll, the existing rules are, in a word, ugly;
another motivation to convert them as soon as possible. I document the
rules I write so they're easily located and the structures are organized
by service. Also I will get to do away with the firewall marks! SELinux
crossed my mind, but after having dealt with it in development, ruled it
out near the start. I did run 'fixfiles restore' as a matter of
precaution. As for the logs, well...nothing pointed to the flapping that
was taking place.

Thanks for the responses, Evan and Meryll!

Meryll Larkin wrote:
> If it was my box, I'd check the firewall again. If the ssh is
> configured the same on bad-box as on good-box and ssh works in and out
> on your intranet with bad-box, it doesn't sound to me like the problem
> is with ssh. For example, if it was pam.d or smrsh I don't think you'd
> be able to connect at all. tcp wrappers is a possibility - worth a
> third or fourth look.
>
> When looking at the firewall I'd check the following:
> The configuration has a typo?
> The configuration was made correctly but not saved?
> Maybe there is a buffer or something that needs to be flushed?
>
> Have you tried turning off SELinux (if you have that on this box) just
> to see if that might be the problem? For a quick test (and to be close
> up immediately after the test) you might try allowing All:All in
> hosts.allow
>
> If you grep for ssh in /var/log or in /var/spool/mail/root do you get
> any meaningful messages?
>
> Meryll
>
>
> On Sat, 17 Mar 2007, Brad Willson wrote:
>
>> I would be writing from my GeneTests account but alas, it is having
>> issues.
>>
>> Here's the deal;
>> * reverse DNS is working, which I confirmed with host on all systems on
>> the intranet.
>> * internet SSH sessions into and out of the box time out without
>> connection.
>> * internet SSH sessions into and out of other boxes on the subnet
>> succeed.
>> * intranet SSH sessions succeed on the same subnet, both into and out of
>> the box.
>> * ping connections to internet destinations time out, but ip resolution
>> does take place properly.
>> * runs of chkrootkit and rkhunter revealed nothing unusual.
>> * nsswitch.conf, resolv.conf, hosts, hosts.allow, hosts.deny all appear
>> to be proper compared to other systems.
>> * the most vexing piece of this puzzle; the firewall rules for the bad
>> box mirror those of working boxes.
>>
>> Obviously I am overlooking something, but what?
>>
>>


-- 
Brad Willson, Sr. Computer Specialist
UW GeneTests, UW Box: 358735
EM: bwil150n at u.washington.edu
W: 206.221.4674, C: 425.891.2732
http://www.genetests.org