Things I discovered this morning during the cooling outage
* Mike Jackson seemed to think he should wait until 6AM to notify me of the loss of cooling in the DC. I'm going to make sure he understands I need to know when he knows. * The certificate on watcher.mcs.anl.gov is expired. This prevents some browsers from connecting. For self-signed certificates, I recommend making them last at least 10 years. * console cert is also expired.
stace@namby% console sto01 ~ console: SSLVerifyCallback(): error with certificate at depth: 0 console: SSLVerifyCallback(): issuer = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): subject = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): error #10: certificate has expired console: SSL negotiation failed 21890:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:951:
* I have no idea how to login or acknowledge alerts on the new nagios server. I'm guessing based on the mails headers it comes from peer.mcs.anl.gov, but the URL eludes me. Is it possible to have the URL for the ack in the message? * I don't know the current root pw. (or, the root pw I have doesn't work on sto01.) * I get nagios mails, but they're so frequent and so often flapping that it's not useful to me, and thus I filter them away. I suspect I'm not alone. I'm going to try to craft a Zimbra filter that will send a message to my SMS gateway if the nagios alert is a critical heat warning. * Whoever has the pager either didn't get or didn't respond to alerts about overtemp in the room. Are we sending critical temp alerts to the pager? I understand we don't want the pager to get everything (as I said above, nagios is super spammy right now), but I think critical heat alerts are important enough to trigger it. == Craig
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I renewed the cert (now good through 2013, I don't have the ability to select the duration on the ANL signed certs). I have the pager, and it didn't go off. I own nagios and I agree that it is really chatty on the temp/humidity side and needs some love. It currently doesn't send to the pager as it is just not reliable enough. I will work on this issue this week. Critical temperature events will go to my phone via sms. Once its stable, I will send to the pager and anyone else that needs it. - -corby On Jul 25, 2011, at 10:08 AM, Craig Stacey wrote:
* Mike Jackson seemed to think he should wait until 6AM to notify me of the loss of cooling in the DC. I'm going to make sure he understands I need to know when he knows.
* The certificate on watcher.mcs.anl.gov is expired. This prevents some browsers from connecting. For self-signed certificates, I recommend making them last at least 10 years.
* console cert is also expired.
stace@namby% console sto01 ~ console: SSLVerifyCallback(): error with certificate at depth: 0 console: SSLVerifyCallback(): issuer = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): subject = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): error #10: certificate has expired console: SSL negotiation failed 21890:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:951:
* I have no idea how to login or acknowledge alerts on the new nagios server. I'm guessing based on the mails headers it comes from peer.mcs.anl.gov, but the URL eludes me. Is it possible to have the URL for the ack in the message?
* I don't know the current root pw. (or, the root pw I have doesn't work on sto01.)
* I get nagios mails, but they're so frequent and so often flapping that it's not useful to me, and thus I filter them away. I suspect I'm not alone. I'm going to try to craft a Zimbra filter that will send a message to my SMS gateway if the nagios alert is a critical heat warning.
* Whoever has the pager either didn't get or didn't respond to alerts about overtemp in the room. Are we sending critical temp alerts to the pager? I understand we don't want the pager to get everything (as I said above, nagios is super spammy right now), but I think critical heat alerts are important enough to trigger it.
== Craig
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) iD8DBQFOLYozQhpwH3ALVFERArOWAKCXPZmoZQRA+WzAuchDNtgLVWSZFACdG7Di /TnSlhq/VvmqDDmcxSUzfwA= =xSb4 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I believe that I have fixed the problem with the radix PDU chattyness. It was a humidity thing. I am no longer monitoring humidity with alerts, just temperature, but they won't hit the pager at this point. - -corby On Jul 25, 2011, at 10:22 AM, Schmitz Corby wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I renewed the cert (now good through 2013, I don't have the ability to select the duration on the ANL signed certs).
I have the pager, and it didn't go off.
I own nagios and I agree that it is really chatty on the temp/humidity side and needs some love. It currently doesn't send to the pager as it is just not reliable enough. I will work on this issue this week. Critical temperature events will go to my phone via sms. Once its stable, I will send to the pager and anyone else that needs it.
- -corby
On Jul 25, 2011, at 10:08 AM, Craig Stacey wrote:
* Mike Jackson seemed to think he should wait until 6AM to notify me of the loss of cooling in the DC. I'm going to make sure he understands I need to know when he knows.
* The certificate on watcher.mcs.anl.gov is expired. This prevents some browsers from connecting. For self-signed certificates, I recommend making them last at least 10 years.
* console cert is also expired.
stace@namby% console sto01 ~ console: SSLVerifyCallback(): error with certificate at depth: 0 console: SSLVerifyCallback(): issuer = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): subject = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): error #10: certificate has expired console: SSL negotiation failed 21890:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:951:
* I have no idea how to login or acknowledge alerts on the new nagios server. I'm guessing based on the mails headers it comes from peer.mcs.anl.gov, but the URL eludes me. Is it possible to have the URL for the ack in the message?
* I don't know the current root pw. (or, the root pw I have doesn't work on sto01.)
* I get nagios mails, but they're so frequent and so often flapping that it's not useful to me, and thus I filter them away. I suspect I'm not alone. I'm going to try to craft a Zimbra filter that will send a message to my SMS gateway if the nagios alert is a critical heat warning.
* Whoever has the pager either didn't get or didn't respond to alerts about overtemp in the room. Are we sending critical temp alerts to the pager? I understand we don't want the pager to get everything (as I said above, nagios is super spammy right now), but I think critical heat alerts are important enough to trigger it.
== Craig
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
iD8DBQFOLYozQhpwH3ALVFERArOWAKCXPZmoZQRA+WzAuchDNtgLVWSZFACdG7Di /TnSlhq/VvmqDDmcxSUzfwA= =xSb4 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) iD8DBQFOLcLnQhpwH3ALVFERAm3RAKCmWMr0uBNybwMGcCP5U5poQoDALACdGXlF vkLwZAh9ld6t9yjx/2/80jM= =5rvj -----END PGP SIGNATURE-----
New console cert is in place. Pointing a browser at http://peer.mcs.anl.gov now gets you to the main Nagios page. Login using your MCS creds. Ken Raffenetti Systems Administration Associate MCS Division - Argonne National Laboratory On 07/25/2011 10:08 AM, Craig Stacey wrote:
* Mike Jackson seemed to think he should wait until 6AM to notify me of the loss of cooling in the DC. I'm going to make sure he understands I need to know when he knows.
* The certificate on watcher.mcs.anl.gov is expired. This prevents some browsers from connecting. For self-signed certificates, I recommend making them last at least 10 years.
* console cert is also expired.
stace@namby% console sto01 ~ console: SSLVerifyCallback(): error with certificate at depth: 0 console: SSLVerifyCallback(): issuer = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): subject = /C=US/ST=Illinois/L=Lemont/O=ANL/OU=MCS/CN=ROOT/[email protected] console: SSLVerifyCallback(): error #10: certificate has expired console: SSL negotiation failed 21890:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:951:
* I have no idea how to login or acknowledge alerts on the new nagios server. I'm guessing based on the mails headers it comes from peer.mcs.anl.gov, but the URL eludes me. Is it possible to have the URL for the ack in the message?
* I don't know the current root pw. (or, the root pw I have doesn't work on sto01.)
* I get nagios mails, but they're so frequent and so often flapping that it's not useful to me, and thus I filter them away. I suspect I'm not alone. I'm going to try to craft a Zimbra filter that will send a message to my SMS gateway if the nagios alert is a critical heat warning.
* Whoever has the pager either didn't get or didn't respond to alerts about overtemp in the room. Are we sending critical temp alerts to the pager? I understand we don't want the pager to get everything (as I said above, nagios is super spammy right now), but I think critical heat alerts are important enough to trigger it.
== Craig
participants (3)
-
Craig Stacey -
Ken Raffenetti -
Schmitz Corby