Re: stress test deadlock involving vm and/or geom

From: Don Lewis (truckman_at_FreeBSD.org)
Date: 09/29/05

  • Next message: Poul-Henning Kamp: "Re: stress test deadlock involving vm and/or geom" 
    Date: Thu, 29 Sep 2005 00:31:02 -0700 (PDT)
To: phk@phk.freebsd.dk

    On 29 Sep, Poul-Henning Kamp wrote:
    > In message <200509290214.j8T2EogR006609@gw.catspoiler.org>, Don Lewis writes:
    >
    >>Also, g_io_check() can only return ENOMEM if that is the provider error
    >>status, but the provider seems to be happy:
    >
    > ENOMEM can also be returned by the code in the GEOM class.

    Ok, that was obscure enough.

    It looks like the
            uma_zalloc(biozone, M_NOWAIT | M_ZERO)
    call in g_clone_bio() is the likely culprit. Lots of allocation
    failures for this zone:

    (kgdb) print *biozone
    $2 = {uz_name = 0xc086938c "g_bio", uz_lock = 0xc10295a8, uz_keg = 0xc10295a0,
      uz_link = {le_next = 0x0, le_prev = 0xc10295d8}, uz_full_bucket = {
        lh_first = 0x0}, uz_free_bucket = {lh_first = 0x0}, uz_ctor = 0,
      uz_dtor = 0, uz_init = 0, uz_fini = 0, uz_allocs = 22457718,
      uz_frees = 22456210, uz_fails = 6603729, uz_fills = 1, uz_count = 128,
      uz_cpu = {{uc_freebucket = 0x0, uc_allocbucket = 0x0, uc_allocs = 0,
          uc_frees = 0}}}

    Any easy way to figure out why the memory allocation is repeatedly
    failing?

    Here's the output of vmstat -z:

    ITEM SIZE LIMIT USED FREE REQUESTS

    UMA Kegs: 140, 0, 62, 10, 62
    UMA Zones: 120, 0, 62, 28, 62
    UMA Slabs: 64, 0, 2169, 132, 51890
    UMA RCntSlabs: 104, 0, 104, 266, 37549
    UMA Hash: 128, 0, 1, 29, 3
    16 Bucket: 76, 0, 12, 38, 45
    32 Bucket: 140, 0, 30, 26, 69
    64 Bucket: 268, 0, 14, 14, 72
    128 Bucket: 524, 0, 2409, 244, 53217
    VM OBJECT: 132, 0, 5249, 17255, 2323529
    MAP: 192, 0, 7, 33, 7
    KMAP ENTRY: 68, 56616, 867, 1317, 797654
    MAP ENTRY: 68, 0, 1881, 3271, 615413
    PV ENTRY: 24, 1494950, 186575, 68335, 77309488
    DP fakepg: 72, 0, 0, 0, 0
    mt_zone: 64, 0, 194, 101, 194
    16: 16, 0, 2357, 282, 2618883
    32: 32, 0, 36199, 19962, 9905089
    64: 64, 0, 5020, 1883, 3448783
    128: 128, 0, 29552, 4078, 5969903
    256: 256, 0, 2556, 309, 6529179
    512: 512, 0, 71, 33, 5456
    1024: 1024, 0, 139, 65, 67843
    2048: 2048, 0, 3123, 57, 775663
    4096: 4096, 0, 216, 97, 22551
    Files: 72, 0, 151, 326, 2846128
    PROC: 524, 0, 143, 130, 14242
    THREAD: 372, 0, 273, 77, 5308
    KSEGRP: 88, 0, 273, 47, 707
    UPCALL: 44, 0, 0, 156, 866
    VMSPACE: 300, 0, 92, 142, 14191
    mbuf_packet: 256, 0, 66, 51, 82560936
    mbuf: 256, 0, 3, 540, 231234746
    mbuf_cluster: 2048, 25600, 117, 91, 12760461
    ACL UMA zone: 388, 0, 0, 0, 0
    g_bio: 132, 0, 1508, 0, 22457718
    ata_request: 200, 0, 0, 0, 0
    ata_composite: 192, 0, 0, 0, 0
    VNODE: 272, 0, 29373, 6523, 4183368
    VNODEPOLL: 76, 0, 0, 0, 0
    S VFS Cache: 68, 0, 11060, 2828, 844011
    L VFS Cache: 291, 0, 0, 0, 0
    NAMEI: 1024, 0, 24, 52, 7276418
    NFSMOUNT: 480, 0, 2, 14, 2
    NFSNODE: 424, 0, 25, 902, 7316
    DIRHASH: 1024, 0, 491, 97, 56851
    PIPE: 408, 0, 4, 59, 4381
    KNOTE: 68, 0, 0, 112, 80
    socket: 356, 25608, 66, 55, 807098
    unpcb: 140, 25620, 17, 39, 217
    udpcb: 180, 25608, 25, 41, 805728
    inpcb: 180, 25608, 24, 64, 1152
    tcpcb: 460, 25600, 24, 32, 1152
    tcptw: 48, 5148, 0, 156, 237
    syncache: 100, 15366, 0, 78, 348
    hostcache: 76, 15400, 1, 99, 2
    tcpreass: 20, 1690, 0, 0, 0
    sackhole: 20, 0, 0, 0, 0
    ripcb: 180, 25608, 0, 0, 0
    rtentry: 132, 0, 14, 44, 14
    SWAPMETA: 276, 121576, 2620, 5542, 354141
    FFS inode: 132, 0, 29314, 4123, 4175811
    FFS1 dinode: 128, 0, 183, 1497, 13155
    FFS2 dinode: 256, 0, 29124, 3441, 4162649

    It looks like the problem might be caused by having too much I/O in
    progress.

    (kgdb) print runningbufspace
    $3 = 1585152
    (kgdb) print lorunningspace
    $4 = 524288
    print hirunningspace
    $5 = 1048576

    so runningbufspace >> hirunningspace

    The bufdaemon and the syncer both ignore the runningbufspace limit.
    Maybe they should obey the limit, but each be guaranteed a minimum
    quota.

    _______________________________________________
    freebsd-current@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-current
    To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

  • Next message: Poul-Henning Kamp: "Re: stress test deadlock involving vm and/or geom"