Re: 7.0-Beta 3: zfs makes system reboot

On Dec 2, 2007, at 14:15 , Johan Ström wrote:

On Dec 2, 2007, at 13:33 , Johan Ström wrote:

On Nov 30, 2007, at 17:27 , Michael Rebele wrote:

Hello,

i'm testing the zfs since 7.0-Beta 1.
First, i had only access to an 32 Bit Machine (P4/3GHz with 2GB RAM, 2xHD for RAID1 and 2xHD for ZFS Raid 0).

While running iozone with the following call:
iozone -R -a -z -b file.wks -g 4G -f testile

(This is inspired by Dominic Kay from Sun, see http:// blogs.sun.com/dom/entry/zfs_v_vxfs_iozone for details).

the well known "kmem_malloc" error occured and stopped the system.
(panic: kmem_malloc (131072): kmem_map too small: 398491648 total allocated cpuid=1)

I tested several optimizations as suggested in the ZFS Tuning Guide and several postings on this list.
The problem stayed mainly the same, it stopped with a "kmem_malloc" or rebooted without warning. This depends on the configuration, if i raised the vm.kmem_-sizes or only the KVA_PAGES or both.
But it never ever made the benchmark. With more memory in vm.kmem_size and vm.kmem_size_max, the problem came later.



But ok, the main target for the ZFS is to use amd64, not i386. Now i have access to an Intel Woodcrest-System, it's a Xeon 5160 with 4GB RAM and 1xHD. It has UFS for the System and Home and one ZFS only for data (for the iozone-Benchmark).
It has a vanilla kernel, i haven't touched it. I've tested the default settings from Beta 3 and applied the tuning tips from the Tuning Guide.
It shows the same behaviour as on the 32 Bit machine. One major difference: it makes always a reboot. There's no kmem_malloc error message (which made the system hang).

The problem is the "-z" option in the iozone-Benchmark. Without it, the benchmark works (on the i386 and on the amd64-Machine). This option makes iozone testing small record sizes for large files. On an UFS-Filesystem, iozone works with the "-z" option. Though, it seems to me, that this is a problem with ZFS.

Here are some more informations (from the amd64-System):

1. The captured iozone output

[root@zfs /tank/iozone]# iozone -R -a -z -b filez-512M.wks -g 4G - f testile
...


For the record, I can reproduce the same thing on amd64 FreeBSD RELENG_7 (installed from beta3 2 days ago) from 2 days ago. Its a c2d box with 2Gb of memory and two satadrives in zpool mirror. No special tweaking whatsoever yet..
The panic was Page fault, supervisor read instruction page not present.. so not the (apparently) regular kmem_malloc? So I doubt the other patch that was linked to by Alexandre would help?

iozone got to
Run began: Sun Dec 2 13:11:53 2007

Excel chart generation enabled
Auto Mode
Cross over of record size disabled.
Using maximum file size of 4194304 kilobytes.
Command line used: iozone -R -a -z -b file.wks -g 4G -f testile
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
64 4 122584 489126 969761 1210227 1033216 503814 769584 516414 877797 291206 460591 703068 735831
64 8 204474 735831 1452528 1518251 1279447 799377 1255511 752329 1460430 372410 727850 1087638 1279447
......
131072 4 65734 71698 1011780 970967 755928 5479 1008858 494172 931232 65869 68155 906746 910950
131072 8 79507 74422 1699148 1710185 1350184 10907 1612344 929991 1372725 34699 74782 1407638 1429434
131072 16 82479 74279 2411000 2426173 2095714 25327 2299061 1608974 2038950 71102 69200 1887231 1893067
131072 32 75268 73077 3276650 3326454 2954789 70573 3195793 2697621 2987611
then it died

No cores dumped however.. Altough I'm running on a gmirror for swap, if I recall correct at least 6.x couldnt dump to a gmirror, I guess 7.x cant either then.. Altought the dump message DID say it dumped memory (and it did say Dump complete), savecore didnt find any dumps at boot..

The box didnt do anything else during this test, and is not running any apps yet. Havent encounterd the problem before, but then again I've only been playing with it for 2 days without any real hard test (just scp'ed about 50 gigs of data to it, but thats it)

Ehr.. im sorry, I think i missread the dump.. or rather the whole dump wasnt on screen.. just did the same test again:

panic: kmem_malloc(131082): kmem_map too small: 412643328 total allocated
and page fault..

I'll test that VM patch.

Back again after > 24h testing. Without the patch it crashes (two out of two times).

After applying the patch I havent had a single crash, and it was runnig almost through the full test (After 10 hours I didnt realy think it would crash anyway so :P)
Tried with no special loader.conf-settings, then with vm.kmem_size (_max)="1G", and then also with vfs.zfs.arc_max="512M".. No crashes anywhere..

So this patch seems to make it not crash.. But as Michael Rebele pointed out in another post, it was written 4 weeks before beta3, and yet not applied?

_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: 7.0-Beta 3: zfs makes system reboot
    ... I've applied the mentioned patch to the Beta3. ... I guess, the kmem_map error is the cause, because the symptoms are the same as on the BETA3-system before the applied patch. ... The last output i have from the benchmark, showed that the file size were in the 4GB area with a reclen in the 8192 area (well, short before iozone should finish). ... You should really apply the mentioned patch if you plan to use ZFS on your Box for more than just testing and experimenting. ...
    (freebsd-current)
  • 7.0-Beta 3: zfs makes system reboot
    ... While running iozone with the following call: ... I tested several optimizations as suggested in the ZFS Tuning Guide and several postings on this list. ... Now i have access to an Intel Woodcrest-System, it's a Xeon 5160 with 4GB RAM and 1xHD. ... This option makes iozone testing small record sizes for large files. ...
    (freebsd-current)
  • Re: 7.0-Beta 3: zfs makes system reboot
    ... I tested several optimizations as suggested in the ZFS Tuning Guide and several postings on this list. ... With more memory in vm.kmem_size and vm.kmem_size_max, ... This option makes iozone testing small record sizes for large files. ... Altough I'm running on a gmirror for swap, if I recall correct at least 6.x couldnt dump to a gmirror, I guess 7.x cant either then.. ...
    (freebsd-current)
  • Re: 7.0-Beta 3: zfs makes system reboot
    ... I tested several optimizations as suggested in the ZFS Tuning Guide and several postings on this list. ... With more memory in vm.kmem_size and vm.kmem_size_max, ... This option makes iozone testing small record sizes for large files. ... Altough I'm running on a gmirror for swap, if I recall correct at least 6.x couldnt dump to a gmirror, I guess 7.x cant either then.. ...
    (freebsd-current)
  • Change in default vm_dirty_ratio
    ... for the 2.6.22-rc kernels in this patch: ... IOZone write drops by about 60% when test file size is 50 percent of ...
    (Linux-Kernel)