Help request: problems with a 5.1 server and large numbers of ssh users.

From: Len Sassaman (rabbi_at_anonymizer.com)
Date: 11/20/03

  • Next message: Terry Lambert: "Re: Conflict between <sys/sysproto.h> & <stdio.h> ... ?" 
    Date: Wed, 19 Nov 2003 21:26:10 -0800
To: freebsd-current@freebsd.org, freebsd-hackers@freebsd.org

    Hi folks,

    I have a problem, and I am unable to find previous discussions of it.
    Any pointers or clues would be much appreciated.

    I have a FreeBSD 5.1 server that needs to be able to handle several
    thousand simultaneous ssh sessions from distinct users. (I am using
    FreeBSD 5.1 because I need to be able to support ldap authentication.)

    Hardware info:
    CPU: AMD Athlon(tm) MP 2000+ (1666.74-MHz 686-class CPU)
    real memory = 1610088448 (1535 MB)
    avail memory = 1558822912 (1486 MB)

    My version of ssh is 3.6.1p2 patched to address the security concerns.
    (I am not using 3.7.1p because it dropped support for password
    authentication with PAM, and I cannot assume keyboard-interactive
    authentication will be present in my users' clients.)

    All of these users are doing ssh port forwarding, and are not assigned
    ptys.

    I have not modified login.conf in any way -- the defaults of "no
    limits" remain.

    The kernel tunables in /boot/loader.conf are set to:

    kern.maxfiles="49312"
    kern.maxproc="24656"
    kern.maxprocperuid="11094"
    kern.ipc.maxsockets="24656"
    kern.ipc.somaxconn="8192"

    The kernel is compiled with NMBCLUSTERS=65536 and maxusers=0 (which
    defaults to 384).

    The problem is that after about 150 users log in (300ish sshd sessions,
    since I am using privsep), incoming connections start getting dropped.
    i.e.,

    bash-2.05b# ssh -v localhost
    OpenSSH_3.6.1p2, SSH protocols 1.5/2.0, OpenSSL 0x0090701f
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: Rhosts Authentication disabled, originating port will not be
    trusted.
    debug1: Connecting to localhost [::1] port 22.
    socket: Protocol not supported
    debug1: Connecting to localhost [127.0.0.1] port 22.
    debug1: Connection established.
    debug1: identity file /root/.ssh/identity type -1
    debug1: identity file /root/.ssh/id_rsa type -1
    debug1: identity file /root/.ssh/id_dsa type -1
    ssh_exchange_identification: Connection closed by remote host
    debug1: Calling cleanup 0x805f010(0x0)
    bash-2.05b#

    It is my intuition from this behavior that the sshd master process
    listening for connections is unable to spawn a new process to complete
    the authentication step, and thus the connection is being dropped.
    There is no information of use in dmesg, nor in the system logs. (I've
    cranked up LogLevel to DEBUG3 in sshd_config).

    I have a RedHat Linux server running the 2.4.18-3smp kernel on a dual
    Athlon MP 1800+ and 2048MB RAM that is known to handle 1000 users
    without issue -- so I have to believe the FreeBSD box, though not as
    beefy hardware-wise, should be able to do better than a few hundred
    users. I believe this to be some sort of resource limit issue, but I
    have addressed everything I could think of.

    Here's the sysctl vm.zone output:

    vm.zone:
    ITEM SIZE LIMIT USED FREE REQUESTS

    FFS2 dinode: 256, 0, 1089, 21, 1359
    FFS1 dinode: 128, 0, 0, 0, 0
    FFS inode: 144, 0, 1089, 59, 1359
    SWAPMETA: 276, 121576, 0, 0, 0
    unpcb: 140, 65548, 329, 63, 31364
    ripcb: 228, 49317, 0, 0, 0
    syncache: 136, 15370, 0, 58, 36747
    tcptw: 48, 49385, 3812, 255, 89831
    tcpcb: 360, 49313, 1048, 63, 195072
    inpcb: 228, 49317, 4921, 94, 195072
    udpcb: 228, 49317, 1, 33, 114497
    socket: 256, 49320, 1383, 102, 340934
    KNOTE: 64, 0, 0, 124, 114453
    PIPE: 176, 0, 622, 68, 17402
    DIRHASH: 1024, 0, 138, 6, 138
    NAMEI: 1024, 0, 9, 11, 451791
    VNODEPOLL: 76, 0, 0, 0, 0
    VNODE: 292, 0, 1473, 35, 1473
    g_bio: 144, 0, 259, 49, 186276
    VMSPACE: 256, 0, 424, 26, 11035
    UPCALL: 44, 0, 0, 0, 0
    KSE: 64, 0, 496, 62, 496
    KSEGRP: 120, 0, 496, 62, 496
    THREAD: 292, 0, 496, 11, 496
    PROC: 480, 0, 461, 35, 11074
    Files: 60, 0, 6051, 153, 89241268
    65536: 65536, 0, 3, 3, 3
    32768: 32768, 0, 3, 3, 32
    16384: 16384, 0, 56, 22, 1733
    8192: 8192, 0, 2, 4, 50
    4096: 4096, 0, 736, 44, 11965
    2048: 2048, 0, 71, 5, 359215
    1024: 1024, 0, 408, 20, 284756
    512: 512, 0, 102, 18, 43908
    256: 256, 0, 5166, 84, 131327
    128: 128, 0, 6784, 253, 535182
    64: 64, 0, 3032, 68, 87489
    32: 32, 0, 2155, 182, 211243
    16: 16, 0, 4485, 295, 844397
    DP fakepg: 72, 0, 0, 0, 0
    PV ENTRY: 28, 5324340, 347293, 45827, 7291251
    MAP ENTRY: 60, 0, 27235, 89, 560399
    KMAP ENTRY: 60, 32802, 817, 107, 22218
    MAP: 176, 0, 8, 38, 6
    VM OBJECT: 148, 0, 18751, 122, 318636
    UMA Buckets: 512, 0, 741, 1, 0
    UMA Hash: 128, 0, 1, 30, 0
    UMA Slabs: 34, 0, 1092, 8, 0
    UMA Zones: 284, 0, 48, 8, 0

    Am I missing anything?

    Thanks,

    Len

    _______________________________________________
    freebsd-hackers@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
    To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

  • Next message: Terry Lambert: "Re: Conflict between <sys/sysproto.h> & <stdio.h> ... ?"

    Relevant Pages

    • Re: SSH commands to HMC
      ... debug1: Reading configuration data ... debug1: connect to address 10.26.10.21 port 22: ... Connection timed out ... ssh: connect to host 10.26.10.21 port 22: Connection ...
      (AIX-L)
    • Re: Problem connecting to SSH, OpenVMS7.3-2, HP TCP/IP v5.4 patched
      ... warning: Authentication failed. ... connection lost (Connection closed by remote host.). ... So, as you can see, I am trying to make an ssh connection with the ...
      (comp.os.vms)
    • [SLE] Slow SSH login
      ... A> ssh B ... second delay no matter the authentication mechanism ... debug1: Authentication succeeded. ...
      (SuSE)
    • SSH TCP forwarding: works with v1, not with v2 ssh
      ... that they're setting up the tunnels with no problem, ... I can get to the work ssh daemon: ... debug1: Connections to remote port 65002 forwarded to local address palimpsest:22 ... something answers (if I get "connection refused" there's no listener); ...
      (FreeBSD-Security)
    • Re: SSH TCP forwarding: works with v1, not with v2 ssh
      ... >that they're setting up the tunnels with no problem, ... >I can get to the work ssh daemon: ... > debug1: Entering interactive session. ... > Connection closed by foreign host. ...
      (FreeBSD-Security)
    • Quantcast