Re: 7.1-PRERELEASE: arcmsr write performance problem



Replying to my own post ...

I have done a test on the same machine comparing 6.3-p1 to 7.1-PRE. The performance is the expected ~6MB/s (because of the lack of cache) on 6.3-p1, so the BIOS change doesn't seem to be at fault.

This seems to be a regression somewhere between 6.3 to 7.1. The Areca driver is the same in 6.3 and 7.1, so the problem seems to be elsewhere.

I think this is more than just a "performance" problem. The observations with gstat showing extremely high ms/w values (I have seen them as high as 22000) makes it look like IO completion interrupts are being lost.

Any suggestions on where to look next? Are there obvious candidates?


Jan Mikkelsen wrote:
Hi,

I am seeing extremely poor performance (~100kB/s) when untaring large tar files into fresh ufs filesystems. I see the problem with softupdates and without softupdates but with an async mount. This is a Supermicro X7DB8 board, 4GB, 2 x Xeon 5140.

Sample gstat output:

dT: 1.033s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
585 61 0 0 0.0 61 170 13812.0 100.1| da2

I see ms/w start at about 200ms with a ~3MB/s throughput, and then I see ms/w rise and kBps drop. ms/w goes as high as 16-20s, and then suddenly drops back down to about 200ms. Using iostat, while the performance is high(er), kb/t is 64kB, as the problem starts it drops towards 2kB.

Copying a single large file doesn't exhibit this problem, although throughput isn't great (~3-5MB/s). However, that's better that 100kB/s.

arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0 on pci10
ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06
arcmsr0: [ITHREAD]

There are eight disks connected in a RAID-6 configuration. The controller's cache is write-through and the disks' write caches are disabled. NCQ is enabled on the drives.

The same hardware when it ran 6.3-p1 didn't have this problem. However, the system BIOS was updated at the same time as the operating system (in an attempt to solve a recent em problem), so it is possible that it is a BIOS related problem. The same build on an entirely different machine with an aac controller and SAS disks also doesn't show this problem.

Running 'devinfo -r' doesn't list arcmsr as having an interrupt at all. (see below). That strikes me as odd; checking another machine that is still running 6.2 with an arcmsr controller, I can see the interrupt just fine.

So:

- Does anyone have any suggestions?

- Is it normal for arcmsr to not show an interrupt in the output from devinfo in 7.1?

Full dmesg, devinfo below.

Thanks,

Jan Mikkelsen


Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-PRERELEASE #0: Mon Dec 1 14:53:12 EST 2008
root@xxxxxxxxxxxxxxxxxxxxxxxxxxxx:/home/janm/p4/freebsd-image-std-2008.2/work/base-freebsd/home/janm/p4/freebsd-image-std-2008.2/FreeBSD/src/sys/TW-SMP

Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2333.35-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>

AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
usable memory = 4280651776 (4082 MB)
avail memory = 4117843968 (3927 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <SMCI SMCISLP2> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x2400-0x24ff,0x2000-0x20ff mem 0xd8500000-0xd8501fff irq 16 at device 2.0 on pci4
ahd0: [ITHREAD]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x2c00-0x2cff,0x2800-0x28ff mem 0xd8502000-0xd8503fff irq 17 at device 2.1 on pci4
ahd1: [ITHREAD]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
bge0: <Altima Gigabit Ethernet Controller, ASIC rev. 0x105> mem 0xd8600000-0xd860ffff irq 16 at device 1.0 on pci5
miibus0: <MII bus> on bge0
brgphy0: <BCM5701 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:40:f4:66:b1:56
bge0: [ITHREAD]
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3000-0x301f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:30:48:31:67:86
em1: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3020-0x303f mem 0xd8420000-0xd843ffff irq 19 at device 0.1 on pci6
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:30:48:31:67:87
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <PCI-PCI bridge> at device 0.0 on pci9
pci10: <PCI bus> on pcib10
arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0 on pci10
ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06
arcmsr0: [ITHREAD]
pcib11: <PCI-PCI bridge> at device 0.2 on pci9
pci11: <PCI bus> on pcib11
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib12: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci12: <ACPI PCI bus> on pcib12
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd8c00400-0xd8c007ff irq 17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 63XXESB USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 6 ports with 6 removable, self powered
pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci13: <ACPI PCI bus> on pcib13
vgapci0: <VGA-compatible display> port 0x4000-0x40ff mem 0xd0000000-0xd7ffffff,0xd8800000-0xd880ffff irq 18 at device 1.0 on pci13
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model Generic PS/2 mouse, device ID 0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: DVDR <PIONEER DVD-RW DVR-111D/1.23> at ata0-master UDMA66
Waiting 5 seconds for SCSI devices to settle
(probe46:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <Areca ARC-1220-VOL#00 R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da0: 77247MB (158201856 512 byte sectors: 255H 63S/T 9847C)
da1 at arcmsr0 bus 0 target 1 lun 0
da1: <Areca ARC-1220-VOL#01 R001> Fixed Direct Access SCSI-5 device
da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da1: 953673MB (1953122304 512 byte sectors: 255H 63S/T 121576C)
da2 at arcmsr0 bus 0 target 2 lun 0
da2: <Areca ARC-1220-VOL#02 R001> Fixed Direct Access SCSI-5 device
da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da2: 800131MB (1638669312 512 byte sectors: 255H 63S/T 102002C)
sa0 at ahd1 bus 0 target 6 lun 0
sa0: <SEAGATE DAT 04106-XXX 7600> Removable Sequential Access SCSI-2 device
sa0: 10.000MB/s transfers (10.000MHz, offset 15)
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/da0s2a
This module (opensolaris) contains code covered by the
Common Development and Distribution License (CDDL)
see http://opensolaris.org/os/licensing/opensolaris_license/
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6
bge0: link state changed to UP
em0: link state changed to UP
em1: link state changed to UP


nexus0
acpi0
Interrupt request lines:
9
I/O ports:
0x10-0x1f
0x24-0x25
0x28-0x29
0x2c-0x2d
0x2e-0x2f
0x30-0x31
0x34-0x35
0x38-0x39
0x3c-0x3d
0x4e-0x4f
0x50-0x53
0x63
0x65
0x67
0x72-0x77
0x80
0x90-0x9f
0xa4-0xa5
0xa8-0xa9
0xac-0xad
0xb0-0xb5
0xb8-0xb9
0xbc-0xbd
0x295-0x296
0x4d0-0x4d1
0x800-0x80f
0xca2-0xca3
0xca8-0xcaf
0x1000-0x107f
0x1180-0x11bf
0xfe00
I/O memory addresses:
0xe0000000-0xefffffff
0xfe000000-0xfe01ffff
0xfe600000-0xfe6fffff
0xfec80000-0xfec80fff
0xfed1c000-0xfed1ffff
0xfee00000-0xfee0ffff
cpu0
acpi_perf0
est0
p4tcc0
cpufreq0
cpu1
acpi_perf1
est1
p4tcc1
cpufreq1
cpu2
acpi_perf2
est2
p4tcc2
cpufreq2
cpu3
acpi_perf3
est3
p4tcc3
cpufreq3
pcib0
pci0
hostb0
pcib1
pci1
pcib2
pci2
pcib3
pci3
pcib4
pci4
ahd0
Interrupt request lines:
16
I/O ports:
0x2000-0x20ff
0x2400-0x24ff
I/O memory addresses:
0xd8500000-0xd8501fff
ahd1
Interrupt request lines:
17
I/O ports:
0x2800-0x28ff
0x2c00-0x2cff
I/O memory addresses:
0xd8502000-0xd8503fff
pcib5
pci5
bge0
I/O memory addresses:
0xd8600000-0xd860ffff
miibus0
brgphy0
pcib6
pci6
em0
Interrupt request lines:
256
I/O ports:
0x3000-0x301f
I/O memory addresses:
0xd8400000-0xd841ffff
em1
Interrupt request lines:
257
I/O ports:
0x3020-0x303f
I/O memory addresses:
0xd8420000-0xd843ffff
pcib7
pci7
pcib8
pci8
pcib9
pci9
pcib10
pci10
arcmsr0
I/O memory addresses:
0xd8000000-0xd83fffff
0xd8900000-0xd8900fff
pcib11
pci11
hostb1
hostb2
hostb3
hostb4
hostb5
hostb6
hostb7
pcib12
pci12
uhci0
I/O ports:
0x1800-0x181f
usb0
uhub0
uhci1
Interrupt request lines:
19
I/O ports:
0x1820-0x183f
usb1
uhub1
uhci2
Interrupt request lines:
18
I/O ports:
0x1840-0x185f
usb2
uhub2
ehci0
I/O memory addresses:
0xd8c00400-0xd8c007ff
usb3
uhub3
pcib13
pci13
vgapci0
I/O ports:
0x4000-0x40ff
I/O memory addresses:
0xd0000000-0xd7ffffff
0xd8800000-0xd880ffff
isab0
isa0
sc0
vga0
I/O ports:
0x3c0-0x3df
I/O memory addresses:
0xa0000-0xbffff
orm0
I/O memory addresses:
0xc0000-0xcafff
atapci0
I/O ports:
0x170-0x177
0x1f0-0x1f7
0x376
0x3f6
0x1860-0x186f
ata0
Interrupt request lines:
14
acd0
acpi_sysresource0
atdma0
fpupnp0
attimer0
attimer1
pci_link0
pci_link1
pci_link2
pci_link3
pci_link4
pci_link5
pci_link6
pci_link7
atkbdc0
I/O ports:
0x60
0x64
atkbd0
Interrupt request lines:
1
psm0
Interrupt request lines:
12
psmcpnp0
sio0
Interrupt request lines:
4
I/O ports:
0x3f8-0x3ff
sio1
Interrupt request lines:
3
I/O ports:
0x2f8-0x2ff
fdc0
Interrupt request lines:
6
DMA request lines:
2
I/O ports:
0x3f0-0x3f5
0x3f7
ppc0
Interrupt request lines:
7
DMA request lines:
3
I/O ports:
0x378-0x37f
ppbus0
plip0
lpt0
ppi0
acpi_button0
acpi_timer0
ACPI I/O ports:
0x1008-0x100b
apic0
I/O memory addresses:
0xfec00000-0xfec0001f
ram0
I/O memory addresses:
0x0-0x9dfff
0x100000-0xcff4ffff
0x100000000-0x12fffffff

_______________________________________________
freebsd-stable@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@xxxxxxxxxxx"

_______________________________________________
freebsd-stable@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@xxxxxxxxxxx"



Relevant Pages