Re: Deadlock in Solaris 8

From: Dug Birdzell (dug_at_iris.emitter.net)
Date: 07/24/03 

Date: Thu, 24 Jul 2003 02:53:48 GMT

Hi,

Here's what I'd do.. Hope it helps!

Run the application through gdb, dbx or some such, and set
a break point at some of the addresses in the pstack output.
Calls to lwp_sema_wait and _thread_start might be good.
Make sure you have built the application with debug information
so you can display source. This should give you an idea
of where the different threads are when deadlock occurs.

Other thoughts...

Are you using stl strings? These can be subtlely problematic.
I have seen cases where a function taking an stl string as an
arg was passed a character constant. The stl string's type-conversion
operator created a new string instance from the constant data.
Unfortunately the function call was inside a loop that was
hit pretty hard by a number of threads. All that unnecessary string
instantiation was causing the system heap manager to work very hard,
which killing performance due to the multiprocessor environment.
The app actually ran slower when given more processors to run on.
When the code was rewritten to avoid the string instantiation via the
function call, the performance problem went away.

Put another way, frequent heap allocations should be avoided
in a multithreaded application running on a multiprocessor system.

HTH
--Dug

In comp.programming.threads Indira <indira1005@yahoo.com> wrote:
> Hi,
> We have developed a "multi-threaded" application in Solaris 8 running
> in E-255 with 2 processor.The application hangs after running for
> sometime. I'm attaching pstack output. We are using mutex for
> locking.(wondering if it is causing the deadlock)
> Any help/pointers to resolve the problem would be great.

> Pstack:
> ----------------- lwp# 46 / thread# 3 --------------------
> ffffffff7d2a3be8 lwp_sema_wait (ffffffff7c20dd18)
> ffffffff7c90c834 _park (ffffffff7c20dd18, ffffffff7ca22000, 0,
> ffffffff7c20dc10, 1, ffffffff7ca23cc4) + 114
> ffffffff7c90c500 _swtch (ffffffff7c20dc10, ffffffff7ca28150,
> ffffffff7ca22000, 0, 5, 1000) + 424
> ffffffff7c910c68 _reap_wait (ffffffff7ca28150, ffffffff7ca2d6e0, 0,
> 0, 0, 0) + 38
> ffffffff7c9109b8 _reaper (ffffffff7ca23d68, ffffffff7ca28150, 0,
> ffffffff7ca23d40, 0, 1000) + 38
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 47 / thread# 47 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff74e05910, 208, 2f, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70a05361, 1, ffffffff7ca23c80, 0,
> ffffffff70a05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 48 / thread# 48 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff74c03910, 208, 30, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70905361, 1, ffffffff7ca23c80, 0,
> ffffffff70905c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 49 / thread# 49 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff74a01910, 208, 31, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70805361, 1, ffffffff7ca23c80, 0,
> ffffffff70805c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 50 / thread# 50 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7440f910, 208, 32, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70705361, 1, ffffffff7ca23c80, 0,
> ffffffff70705c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 51 / thread# 51 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7420d910, 208, 33, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70605361, 1, ffffffff7ca23c80, 0,
> ffffffff70605c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 52 / thread# 52 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7400b910, 208, 34, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70505361, 1, ffffffff7ca23c80, 0,
> ffffffff70505c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 53 / thread# 53 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73e09910, 208, 35, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70405361, 1, ffffffff7ca23c80, 0,
> ffffffff70405c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 54 / thread# 54 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73c07910, 208, 36, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70305361, 1, ffffffff7ca23c80, 0,
> ffffffff70305c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 55 / thread# 55 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73a05910, 208, 37, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70205361, 1, ffffffff7ca23c80, 0,
> ffffffff70205c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 56 / thread# 56 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73803910, 208, 38, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70105361, 1, ffffffff7ca23c80, 0,
> ffffffff70105c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 57 / thread# 57 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73601910, 208, 39, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70005361, 1, ffffffff7ca23c80, 0,
> ffffffff70005c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 58 / thread# 58 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7300f940, 208, 3a, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6ff05361, 1, ffffffff7ca23c80, 0,
> ffffffff6ff05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 59 / thread# 59 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff72e0d940, 208, 3b, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6fe05361, 1, ffffffff7ca23c80, 0,
> ffffffff6fe05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 60 / thread# 60 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff72c0b940, 208, 3c, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6fd05361, 1, ffffffff7ca23c80, 0,
> ffffffff6fd05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 61 / thread# 61 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff72a09940, 208, 3d, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6fc05361, 1, ffffffff7ca23c80, 0,
> ffffffff6fc05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40

> Thanks
> Indira

Relevant Pages

  • Re: Fast string operations
    ... the char[] for each string cached with the string, or is a new one created ... the string is full of whitespace. ... >> ToLowergenerates a new string instance as does Trim. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: System.String: It doesnt seem to act like a reference type.
    ... First, because Strings are immutable, changing the value of a string ... doesn't really change the string _in situ_ in memory. ... creates a whole new string and points the String reference to it. ... the new String instance instead of the old ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Fast string operations
    ... Regardless, even with your suggestion, the .Trim() still creates a new ... > My assumption here is that you already intend to trim the string before it ... >> ToLowergenerates a new string instance as does Trim. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Strings odd one out
    ... > My understanding of classes is that a class serves as a wrapper for ... or they can be called by a literal String. ... A Java String ... just like any other String instance when the program runs. ...
    (comp.lang.java.programmer)
  • Re: Need Container for Large Binary String
    ... > I need to manipulate large binary strings. ... > family) or STL (vector or STL strings). ... > The STL string class would be ideal, ... > an efficient way in initializing the vector with my binary data (i.e. ...
    (microsoft.public.vc.mfc)
Quantcast