Weird networking problem

From: Stig Bull (stig.bull.no.spam_at_broadpark.no)
Date: 04/18/05

  • Next message: Tristram Scott: "Re: Keyboard problems with Java Desktop System"
    Date: Mon, 18 Apr 2005 11:05:30 +0200
    
    

    We're experiencing som extremely weird networking problems at our
    office, now I know this isn't a Sun related problem, but I'm trying this
    group in order to get an idea of what to do next, or which forums/web
    sites I really should direct my request at, since I haven't found any
    dedicated, *good* networking forum while googing.

    Here is the problem:
    Last Friday morning, I was unable to SSH into the Sun servers at our
    office. It worked as normal Thursday evening when we left so I thought
    this was a Windows problem, but after rebooting the laptop as well as
    another employee complained about the same problem, it turns out there's
    something weird going on with out network.

    What happens is this:
    1: (Everything is on the same subnet, behind a Checkpoint firewall. The
    switch is layer 2 so it doesn't route anything. No configuration has
    been changed on the switch, the firewall, servers or office PC's)

    2: We can SSH into 1 of 4 Sun servers

    3: The same servers are reachable by FTP, ping and HTTP (not tried
    anything else, Telnet is disabled), but not SSH

    4: What turns up in var/adm/messages when we try SSH is this:
    "Apr 18 10:04:29 jupiter sshd[15028]: [ID 800047 auth.crit] fatal:
    Timeout before authentication for x.x.x.x" indicating that the packet
    from the servers *seems* to be routed somewhere else

    5: If we use SSH from our 3 remote production sites or even using a
    remote PC with VPN, we can reach these servers

    6: We can SSH from one server to the other between these 3, we can even
    SSH into the fourth server from these 3, which is also reachable from
    our office PC's, but SSH from this fourth server also times out to the
    other 3

    7: If I debug packets on the firewall, I can't see any SSH packets which
    are routed elsewhere, so everything seems to be going on locally on the
    switch/backbone, not going via the gateway at all

    8: I've rebooted the switch twice, I even rebooted the servers *and* the
    firewall and the problem still persists

    9: I unplugged every cable from the switch one by one, while another
    employee tried SSH into a server. SSH failed every single time

    10: I replaced the switch this morning with an identical model and
    configuration, but the problem still persists

    11: On 2 of 3 servers I can sometimes get lucky and a SSH session is
    successfully established -- after a looooong time. None of the other
    employees gets a session established

    12: SSH to the firewall and a Linux box which is on the same subnet
    works perfectly

    13: Just poking around I tried route -f on one of the servers. I was
    able to SSH into it every single time, but still none of the other
    employees were. Of course the gateway address disappeared with the route
    flushed. After a reboot the server went back to "normal" and I was
    unable to SSH into it again

    14: This happens with SecureSSH (4.1/5.0), Putty and OpenSSH (various
    versions) installed on servers.

    So that's it, I've tried about everything I can think of, but use a
    packet sniffer, even though I'm not sure if it will produce any results.
    I do suspect this to be a routing issue even though I can't see any
    'mal-routed' packets on the firewall.

    Has anyone an idea of what I can try next, or perhaps provide me with a
    link to some networking forum where I can ask about this?

    -- 
    Stig Bull
           | remove .no.spam from my email address to reply by mail |
    No animals were hurt or killed in the process of creating this 
    electronic message. To reduce download time, this message is made of 
    100% recycled bytes.
    

  • Next message: Tristram Scott: "Re: Keyboard problems with Java Desktop System"

    Relevant Pages

    • Analysis of SSH crc32 compensation attack detector exploit
      ... Analysis of SSH crc32 compensation attack detector exploit ... detector vulnerability to remotely compromise a Red Hat Linux ... Active Internet connections (servers and established) ...
      (Incidents)
    • Re: Duplex/Speed Hardcoding
      ... I am between the ages of 45 and 55 and have been in the IT field a very long time I am not nor have been a programmer, started out designing communications and networking equipment as an electronics design engineer. ... We follow the same rules, hard coding servers and infrastructure devices, and letting clients negotiate on all 100M ports. ... All fiber switch to switch connections are always hard set with all flow control disabled. ...
      (comp.dcom.sys.cisco)
    • Re: Agent Forwarding Question for the list
      ... I provided a suggestion (invoking ssh with -vvv) as to how to further troubleshoot the problem. ... I was determined to ask the experts in case it was a common mistake or something that simply is not possible under openssh. ... Say in the ideal setup for development servers I'd have a cronuser, scriptuser, monitoruser, cvsuser, and root all configured with my public key and that I could jump in and out of each not only from my own Linux Desktop, but through each user to each user on other servers in the development chain. ... After reading all the documentation and FAQs I could find, I had assumed ssh-agent on the desktop and agent forwarding on the servers would be sufficient, but something is blocking the forwarding, or I'm way off and this isn't how it's meant to work. ...
      (SSH)
    • Re: Anti-virus Programs
      ... As for SMB/CIFS filesystems shared with Windows systems I use Clamav to ... SSH vulnerability. ... users who have remote shell access. ... But poorly setup mail servers are possibly the worst because your ...
      (Fedora)
    • RE: ssh login protection
      ... Then allow SSH from that server to yours. ... I often ssh into two mail servers from dialup(thus dynamic ... I specify which IPs that can ssh into the two ...
      (Security-Basics)