Lesson learned from working with slow HW crypto card
From: Youlin Feng (youlinfeng_at_hotmail.com)
Date: 03/19/05
- Previous message: Peter C. Lai: "Re: PPP routing failure [fixed]"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
To: freebsd-net@freebsd.org Date: Sat, 19 Mar 2005 02:13:48 +0000
This is a lesson I learned from working with a slow HW crypto card in
the presence of the GbE interface. Hope it is helpful to others when
they run into the same issue.
We are using a 200Mpps HW crypto card on a FreeBSD 5.3 system as a GbE
gateway. With ~700Mbps traffic inbound for encryption, the system
dropped into a livelock state and the throughput is barely ~50Mbps.
The console doesn't respond to key strokes and the system complains
about the lack of memory.
The problem turns out to be related to the fact that the crypto
request queue doesn't have a limit on the queue length. Due to the
async nature of HW crypto processing and the speed mismatch between
the crypto card and the GbE interface, the request queue becomes so
long that CPU will be pretty much monopolized by the crypto_proc
thread running at SWI_NET pri level, leaving no cycles for lower pri
kernel tasks and applications. Because the crypto callback thread can
only run as fast as the HW crypto speed, memory resources are freed at
a much slower rate than when they were allocated, leading to
out-of-memory condition.
The workaround I used is to cap the crypto request queue, ie crp_q.
Capping it to 64 works fine with the 200Mbps crypto card, increasing
the throughput to ~200Mbps. And livelock condition is also gone.
The asymmetric op queue, ie crp_kq, can be left as unlimited because
there are much fewer such requests.
The limited request queue shouldn't have any impact on the SW crypto
since it is a synchronous operation.
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
- Previous message: Peter C. Lai: "Re: PPP routing failure [fixed]"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
|
