I'll dig into this more on monday, but it looks like we may have hit a ulimit problem. ulimit -n is 1024 and not set in the CC startup scripts. Restarting the services brought it back up. Authentication was never down on both auth servers. 33229 ERROR Operational - 004-007-007 2011/04/29 15:15:36:977 [ ] Type com.cryptocard.utilities.CCException, message ' Error creating Topic Connection!' resulting from ... 33230 ERROR Operational - 001-006-001 2011/04/29 15:15:36:978 [ ] Raw exception type: org.jboss.mq.SpyJMSException, me ssage: 'Cannot authenticate user; - nested throwable: (java.net.SocketException: Too many open files)'. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Dan Olson" <[email protected]> To: "Schmitz Corby" <[email protected]> Cc: "Core Admins" <[email protected]> Sent: Friday, April 29, 2011 3:16:15 PM Subject: Re: auth/RADIUS_AUTH_A_USER PROBLEM This was a valid alert.. auth is dead for some reason, but auth2 is still working. I'm looking at it now. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Schmitz Corby" <[email protected]> To: "Dan Olson" <[email protected]> Cc: "Core Admins" <[email protected]> Sent: Friday, April 29, 2011 2:59:39 PM Subject: Fwd: auth/RADIUS_AUTH_A_USER PROBLEM -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I thought you said this doesn't go down! Page has gone off! - -corby Begin forwarded message:
From: [email protected] (Nagios on Watcher) Date: April 29, 2011 2:57:14 PM CDT To: [email protected] Subject: auth/RADIUS_AUTH_A_USER PROBLEM
***** Nagios on watcher *****
Notification Type: PROBLEM
Service: RADIUS_AUTH_A_USER Host: auth Address: 140.221.6.49 State: CRITICAL
Date/Time: Fri Apr 29 14:57:13 CDT 2011
Additional Info: Timeout USER: corby NOTES:
URL: https://watcher.mcs.anl.gov/mcs/nagios/cgi-bin/extinfo.cgi?type=2&host=auth&...
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) iD8DBQFNuxirQhpwH3ALVFERAtJZAJ9XqlIumB0gmfqbFBUj0Hdd/mT7EQCgnsjD 5jZu6h0aMxJCKwsEOd3hen4= =Vfzv -----END PGP SIGNATURE-----