xmodmap not working from xinitrc

Hi!
My .xinitrc has an xmodmap tweak to switch the Ctrl and Caps keys.
For some reason these changes are not applied when I start X, Ctrl is still Ctrl
and Caps is still Caps.
However I have to run xmodmap twice when X has started to actually make it work.
It seems like the changes were applied but somehow not used by X, when I rerun
xmodmap it seems like it switches the keys back to their original states and when I
then run it again it finally works.

Here's my .xinitrc:

------------------------
# X tweaks
xsetroot -cursor_name left_ptr
xset -b
numlockx on
xmodmap ~/.xmodmap

# Desktop environment
feh --bg-center ~/.themes/FreeBSD.jpg
mydzen &

xmonad &

# Wait
wait $!
pkill dzen2
wait
------------------------

and here's my .xmodmap:

------------------------
remove Lock = Caps_Lock
remove Control = Control_L
keysym Control_L = Caps_Lock
keysym Caps_Lock = Control_L
add Lock = Caps_Lock
add Control = Control_L
------------------------

uname -a:
FreeBSD nirvana.my.domain 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #0: Wed Jan 30 10:06:18 CET 2008 root@xxxxxxxxxxxxxxxxx:/usr/obj/usr/src/sys/HYPOCENTER i386

Installed versions:
xorg-7.3_1
xmodmap-1.0.3

--
:wq Jona

Attachment: pgp1zacUQ448v.pgp
Description: PGP signature