Re: Very low disk performance on 5.x (some tweaking and more results)

From: Mike Tancsa (mike_at_sentex.net)
Date: 05/04/05

Next message: Steven Hartland: "Re: Very low disk performance on 5.x (some tweaking and more results)"

Previous message: Steven Hartland: "Re: Very low disk performance on 5.x"
Next in thread: Steven Hartland: "Re: Very low disk performance on 5.x (some tweaking and more results)"
Reply: Steven Hartland: "Re: Very low disk performance on 5.x (some tweaking and more results)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Date: Wed, 04 May 2005 07:57:15 -0400
To: "Poul-Henning Kamp" <phk@phk.freebsd.dk>

At 05:31 PM 02/05/2005, Poul-Henning Kamp wrote:
>In message <6.2.1.2.0.20050502163426.02c4e678@64.7.153.2>, Mike Tancsa writes:
>
> >Using an amrc controller in RAID5, it doesnt make a difference really on
> >the dd stuff - read or write-- perhaps 2-5MB difference on the faster
> >side. Raw reads and writes whether I use da0 or da0s1d, the speed is
> >pretty well the same as before... HOWEVER, one measurement is really odd on
> >bonnie-- the random seeks .... and postmark shows a *big* difference as
> well.
>
>This is very likely the alignment thing.

OK, some further tests, trying to control for this. I am not sure what
values to actually fiddle with via bsdlable as this is the entire disk so I
will just vary mounting da0 vs da0s1d. I also cvsup'd to current as of
yesterday (FreeBSD nfs.sentex.ca 6.0-CURRENT FreeBSD 6.0-CURRENT #0: Mon
May 2 15:03:53 EDT 2005). I will also try with Scott's changes to the
driver which went in after.

Also, the newfs params does seem to make a very big difference and
vfs.read_max as well.

[nfs]# fdisk da0
******* Working on device /dev/da0 *******
parameters extracted from in-core disklabel are:
cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
     start 63, size 1464838767 (715253 Meg), flag 80 (active)
         beg: cyl 0/ head 1/ sector 1;
         end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
[nfs]# bsdlabel da0s1
# /dev/da0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
   c: 1464838767 0 unused 0 0 # "raw" part,
don't edit
   d: 1464838767 0 4.2BSD 2048 16384 28552
[nfs]# newfs -b 32768 -f 4096 /dev/da0

[nfs]# bsdlabel da0
bsdlabel: /dev/da0: no valid label found
[nfs]#
even though I can mount it just fine

[nfs]# mount /dev/da0 /mnt
[nfs]# df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/ad0s1a 1012974 356028 575910 38% /
devfs 1 1 0 100% /dev
/dev/ad0s1e 1012974 580870 351068 62% /tmp
/dev/ad0s1d 15231278 5848016 8164760 42% /usr
/dev/ad0s1f 19564270 192176 17806954 1% /var
/dev/da0 720910352 8 663237516 0% /mnt
[nfs]#

-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
da0s1 4000 61175 53.3 61832 16.3 31829 8.7 63008 60.4 79551 11.2
224.2 0.7
da0s1 v 4000 61278 54.3 61938 16.4 31828 8.8 73566 71.7 81778 12.6
216.5 0.7
da0 4000 62625 54.5 62154 15.7 31907 8.3 63297 60.8 80139 11.0
226.2 0.7
da0 v 4000 62315 54.3 61759 15.4 32094 8.3 74209 72.3 84470 12.6
223.5 0.7
da0 N 4000 75239 62.7 75771 15.5 33596 8.4 81909 78.6 98453 13.3
214.8 0.8
da0 vN 4000 72786 61.7 74939 15.3 33672 8.4 82944 79.8 108106 14.9
214.7 0.8
v = sysctl -w vfs.read_max=16
N = newfs -b 32768 -f 4096 /dev/da0

[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set location /mnt
pm>set size 300 100000
pm>set transactions 400000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
1418 seconds total
1418 seconds of transactions (282 per second)

Files:
         200107 created (141 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (140 per second)
         199905 read (140 per second)
         199384 appended (140 per second)
         200107 deleted (141 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (140 per second)

Data:
12715.55 megabytes read (8.97 megabytes per second)
12728.92 megabytes written (8.98 megabytes per second)
pm>quit
[nfs]#

create a junk file
[nfs]# time dd if=/dev/urandom of=/mnt/big-4g-rand bs=1024k count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 124.694915 secs (33636528 bytes/sec)
0.022u 116.020s 2:04.70 93.0% 21+2910k 124+32000io 0pf+0w

wait 2 min for disks to totally flush

[nfs]# time dd if=/dev/zero of=/mnt/big-4g-zero bs=1024k count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 66.463634 secs (63106751 bytes/sec)
0.000u 11.265s 1:06.50 16.9% 21+2926k 126+32000io 0pf+0w

sysctl -w vfs.read_max=16

(clear out cache by reading via md5 4G of other files) and then wait a bit

[nfs]# time dd if=/mnt/big-4g-rand of=/dev/null bs=1024k count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 59.178470 secs (70875506 bytes/sec)
0.000u 7.677s 0:59.18 12.9% 20+2784k 32128+0io 0pf+0w

[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set locatime /mnt
Eh?
pm>set location /mnt
pm>set size 300 100000
pm>set transactions 400000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
1369 seconds total
1369 seconds of transactions (292 per second)

Files:
         200107 created (146 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (145 per second)
         199905 read (146 per second)
         199384 appended (145 per second)
         200107 deleted (146 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (145 per second)

Data:
12715.55 megabytes read (9.29 megabytes per second)
12728.92 megabytes written (9.30 megabytes per second)
pm>

With vfs.read_max=8

[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set location /mnt
pm>set size 300 100000
pm>set transactions 400000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
1013 seconds total
1013 seconds of transactions (394 per second)

Files:
         200107 created (197 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (197 per second)
         199905 read (197 per second)
         199384 appended (196 per second)
         200107 deleted (197 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (196 per second)

Data:
12715.55 megabytes read (12.55 megabytes per second)
12728.92 megabytes written (12.57 megabytes per second)
pm>
pm>quit
[nfs]#

---------------------------------------------------------------------
newfs -b 32768 -f 4096 /dev/da0

[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set location /mnt
pm>set transactions 400000
pm>set size 300 100000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
542 seconds total
541 seconds of transactions (739 per second)

Files:
         200107 created (369 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (368 per second)
         199905 read (369 per second)
         199384 appended (368 per second)
         200107 deleted (369 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (368 per second)

Data:
12715.55 megabytes read (23.46 megabytes per second)
12728.92 megabytes written (23.49 megabytes per second)
pm>
[nfs]# sysctl -w vfs.read_max=16
vfs.read_max: 8 -> 16

pm>set location /mnt
pm>set transactions 400000
pm>set size 300 100000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
542 seconds total
541 seconds of transactions (739 per second)

Data:
12715.55 megabytes read (23.46 megabytes per second)
12728.92 megabytes written (23.49 megabytes per second)
pm>

[nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 19.298799 secs (108667489 bytes/sec)
0.000u 3.603s 0:19.32 18.6% 21+2965k 16019+0io 0pf+0w
[nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 20.188331 secs (103879414 bytes/sec)
0.015u 3.515s 0:20.24 17.3% 20+2865k 16020+0io 3pf+0w
[nfs]# sysctl -w vfs.read_max=8

vfs.read_max: 16 -> 8
[nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 21.421867 secs (97897722 bytes/sec)
0.007u 3.482s 0:21.43 16.2% 21+2873k 16018+0io 0pf+0w
[nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 23.306603 secs (89981024 bytes/sec)
0.000u 3.590s 0:23.37 15.3% 19+2687k 16020+0io 3pf+0w
[nfs]#

_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Next message: Steven Hartland: "Re: Very low disk performance on 5.x (some tweaking and more results)"

Previous message: Steven Hartland: "Re: Very low disk performance on 5.x"
Next in thread: Steven Hartland: "Re: Very low disk performance on 5.x (some tweaking and more results)"
Reply: Steven Hartland: "Re: Very low disk performance on 5.x (some tweaking and more results)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Relevant Pages

Re: 2.6.6 breaks kmail (nfs related?)
... My problem occurs already with kernel 2.6.5, and it is indeed NFS related (It ... I reproduced the crash with a server exporting an ext2 partition and one ... This succeeds in the 2.6.4 trace: ... send the line "unsubscribe linux-kernel" in ...
(Linux-Kernel)
Re: Trash
... there is a .local in your home directory. ... The Trash behaves slightly different when deleting a file that's on a NFS ... where the files and info dirs are for that NFS share or partition. ...
(alt.os.linux.suse)
Re: NFS4 Rephrase
... NFS doesn't see it. ... If you NFS mount a partition, only that partition will be available on ... the NFS server system within that partition. ...
(Fedora)
Re: XFS over 7.7TB LVM partition through NFS
... >> over NFS. ... >> this partition over NFS, and after a short while we get this call trace, ... > extra lines before the Call Trace ... ... send the line "unsubscribe linux-kernel" in ...
(Linux-Kernel)
Re: FreeBSD 5.3 I/O Performance / Linux 2.6.10 | Continued Discussion
... RELENG_5 and HEAD, at least as it relates to the 3ware driver. ... I could not run the full NFS tests, and async nfs mounts are broken in ... pm>set transactions 400000 ... Deleting files...Done ...
(freebsd-performance)