Re: ZFS melting under postgres...

On Dec 12, 2007, at 10:25 PM, Benjamin Close wrote:

Hugo Silva wrote:
Benjamin Close wrote:
Peter Losher wrote:
Hi,

As part of our testing 7.0/ZFS we tried putting it thru it's paces
having ZFS act as our storage medium for some test pgsql db's (like for
sqlgrey, etc) and in both BETA2 and BETA4 (amd64) we get the same
results with a RAIDZ2 container:

-=-
Dec 12 14:24:12 nsa sqlgrey: fatal: setconfig error at
/usr/local/sbin/sqlgrey line 186.
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=vault error=86
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa postgres[50527]: [5-1] PANIC: could not write to
log file 2, segment 53 at offset 7864320, length 8192: Input/ output error
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=vault error=86
Dec 12 16:49:53 nsa postgres[50596]: [1-1] FATAL: the database system
is starting up
Dec 12 16:49:53 nsa kernel: pid 50527 (postgres), uid 70: exited on
signal 6 (core dumped)
-=-

It basically corrupts the container from the inside until it fails
completely (usually withing 24-48 hours depending on how busy the db is)

I had thought it was a bad SATA replicator/controller, but we had that
replaced w/ one from Supermicro. So it's either the disks, or something
in ZFS. Anyone used ZFS to backend any db's (mysql or pgsql?)

If you need more info, let me know...


Try turning of zil, whilst I don't use a db, I have zfs under high load. I've found without zil turned off I see checksum corruption as well:

/boot/loader.conf

vfs.zfs.zil_disable=1

Cheers,
Benjamin

Wouldn't it be a bad idea to disable ZIL ?

http://www.solarisinternals.com/wiki/index.php/ ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29

A good read is:

http://blogs.sun.com/perrin/entry/the_lumberjack

Which shows why zil exists.

Cheers,
Benjamin

So does anybody know of a battery backed NVRAM card that can be used with FreeBSD that the ZIL could be offloaded to?

--
DaveD

Relevant Pages

  • RE: Help me select hardware....Some real world data that might help
    ... Less than 1 minute into a backup run of only 4 machines, I've got a fresh ZFS wedgie. ... As I understand it, the ZIL is not as much of an integrity boost as a speed boost, especially since we already have checksum-per-block. ... Array drives all live on the LSI right now. ... in the raidz datasets, and the pool as a whole. ...
    (freebsd-current)
  • Re: Panic on ZFS startup after crash
    ... first e-mail about ZFS not beeing able to replay ZIL. ... # zpool export ... I fixed 'zpool import -o ro' problem in HEAD, but you can also patch ...
    (freebsd-stable)
  • Re: NFS hangs
    ... All exported file systems are ZFS; ... I can't see how ZIL can make any difference visible for NFS clients, ...
    (freebsd-current)
  • Re: Panic on ZFS startup after crash
    ... up enough to trigger an assert while ZFS tries to parse/replay it. ... first e-mail about ZFS not beeing able to replay ZIL. ... We can also try to destroy the ZIL without replaying it. ...
    (freebsd-stable)
  • Re: ZFS & Bittorent -> Hang?
    ... related to using raidz or not. ... Prefetch seems to me to be a little ... Not sure whether the zil just helped a lot there or if the ... I'm using ZFS on my Mac OS X laptop as well, ...
    (freebsd-stable)