Re: ZFS melting under postgres...

Peter Losher wrote:
Hi,

As part of our testing 7.0/ZFS we tried putting it thru it's paces
having ZFS act as our storage medium for some test pgsql db's (like for
sqlgrey, etc) and in both BETA2 and BETA4 (amd64) we get the same
results with a RAIDZ2 container:

-=-
Dec 12 14:24:12 nsa sqlgrey: fatal: setconfig error at
/usr/local/sbin/sqlgrey line 186.
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=vault error=86
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa postgres[50527]: [5-1] PANIC: could not write to
log file 2, segment 53 at offset 7864320, length 8192: Input/output error
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad4 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad6 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad8 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad10 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad12 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad14 offset=3665128448 size=22016
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad16 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault
path=/dev/ad18 offset=3665128448 size=21504
Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=vault error=86
Dec 12 16:49:53 nsa postgres[50596]: [1-1] FATAL: the database system
is starting up
Dec 12 16:49:53 nsa kernel: pid 50527 (postgres), uid 70: exited on
signal 6 (core dumped)
-=-

It basically corrupts the container from the inside until it fails
completely (usually withing 24-48 hours depending on how busy the db is)

I had thought it was a bad SATA replicator/controller, but we had that
replaced w/ one from Supermicro. So it's either the disks, or something
in ZFS. Anyone used ZFS to backend any db's (mysql or pgsql?)

If you need more info, let me know...

Try turning of zil, whilst I don't use a db, I have zfs under high load. I've found without zil turned off I see checksum corruption as well:

/boot/loader.conf

vfs.zfs.zil_disable=1

Cheers,
Benjamin
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • ZFS melting under postgres...
    ... having ZFS act as our storage medium for some test pgsql db's (like for ... results with a RAIDZ2 container: ... in ZFS. ...
    (freebsd-current)
  • Re: ZFS melting under postgres...
    ... having ZFS act as our storage medium for some test pgsql db's (like for ... results with a RAIDZ2 container: ... It basically corrupts the container from the inside until it fails ... in ZFS. ...
    (freebsd-current)
  • Re: ZFS melting under postgres...
    ... having ZFS act as our storage medium for some test pgsql db's (like for ... in ZFS. ... It is hard for me to believe that this is FreeBSD-specific bug, ... I'd start from configuring UFS on top of GELI with authentication. ...
    (freebsd-current)
  • Re: ZFS melting under postgres...
    ... results with a RAIDZ2 container: ... It basically corrupts the container from the inside until it fails ... in ZFS. ... Try turning of zil, whilst I don't use a db, I have zfs under high load. ...
    (freebsd-current)
  • Re: ZFS melting under postgres...
    ... having ZFS act as our storage medium for some test pgsql db's (like for ... in ZFS. ... Try turning of zil, whilst I don't use a db, I have zfs under high ... FreeBSD suffers from deadlocks under load if ZIL is enabled. ...
    (freebsd-current)
Loading