February 2016 - Core-other - mailman3.cels.anl.gov

[CELS Sysadmins] FW: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
by Stacey, Craig 17 Feb '16

17 Feb '16

Pretty sure you’ve all seen this, but in case you haven’t. Need to make sure the login nodes are patched for sure, along with nginx, wilbur, gust, collab, xgitlab, jira. I mean, everything needs patching, obviously, but we need to hit the externally exposed machines first. -- Craig On 2/16/16, 4:51 PM, "ihw_cspr-bounces(a)lists.anl.gov on behalf of Gene Rackow" <ihw_cspr-bounces(a)lists.anl.gov on behalf of rackow(a)anl.gov> wrote: >For anyone running Linux... > >In case you haven't seen it yet... A hole has been found in glibc >that allows remote execution of code. It's not easy, but possible. >Note that glibc is in every tool on the system, so anything with >a network component is vulnerable... ssh, apach, wget, sudo, ... > >More likely attempts to exploit this will cause the applications to >crash. Please check your patch levels to get this installed ASAP. > >=================================================== >https://googleonlinesecurity.blogspot.com/2016/02/cve-2015-7547-glibc-getad… > >https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html > >Proof of concept: https://github.com/fjserna/CVE-2015-7547 > >- From the notes... > >Our initial investigations showed that the issue affected all the >versions of glibc since 2.9. You should definitely update if you are >on an older version though. If the vulnerability is detected, >machine owners may wish to take steps to mitigate the risk of an attack. > >The glibc DNS client side resolver is vulnerable to a stack-based >buffer overflow when the getaddrinfo() library function is used. >Software using this function may be exploited with >attacker-controlled domain names, attacker-controlled DNS servers, >or through a man-in-the-middle attack. > >The vectors to trigger this buffer overflow are very common and can >include ssh, sudo, and curl. We are confident that the exploitation >vectors are diverse and widespread; we have not attempted to >enumerate these vectors further. > >Remote code execution is possible, but not straightforward. It >requires bypassing the security mitigations present on the system, >such as ASLR. > >=================================================== >https://googleonlinesecurity.blogspot.com/2016/02/cve-2015-7547-glibc-getad… > >https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html > >Proof of concept: https://github.com/fjserna/CVE-2015-7547 > >- From the notes... > >Our initial investigations showed that the issue affected all the >versions of glibc since 2.9. You should definitely update if you are >on an older version though. If the vulnerability is detected, >machine owners may wish to take steps to mitigate the risk of an attack. > >The glibc DNS client side resolver is vulnerable to a stack-based >buffer overflow when the getaddrinfo() library function is used. >Software using this function may be exploited with >attacker-controlled domain names, attacker-controlled DNS servers, >or through a man-in-the-middle attack. > >The vectors to trigger this buffer overflow are very common and can >include ssh, sudo, and curl. We are confident that the exploitation >vectors are diverse and widespread; we have not attempted to >enumerate these vectors further. > >Remote code execution is possible, but not straightforward. It >requires bypassing the security mitigations present on the system, >such as ASLR. > > >--Gene > >--Gene >_______________________________________________ >cyber911 mailing list >cyber911(a)lists.anl.gov >https://lists.anl.gov/mailman/listinfo/cyber911

1 0

Where's Tina- Wednesday
by Stanton-Piersanti, Tina Joy 17 Feb '16

17 Feb '16

Just in case you didn’t see my message in Chat, I am home again today with Vinnie. Will work from here as much as I can. Today, I plan on working on tickets, accounts and I have a call at 10 am with David Martin re: 593’s Tomorrow I have a scheduled vaca day and will head to Indiana for our last attempt to fight our taxes. T.

1 0

[CELS Sysadmins] where's Linda 2/16-18
by Winkler, Linda 11 Feb '16

11 Feb '16

I will be at FNAL next Tues-Thur for project meetings. lw

1 0

[CELS Sysadmins] Where's Dan?
by Murphy-Olson, Daniel E. 11 Feb '16

11 Feb '16

I have a parent/teacher meeting this morning from 10am-noon. I'm working from home prior to that, and I'll be in the office this afternoon. Dan

1 0

[CELS Sysadmins] Where's Dan?
by Murphy-Olson, Daniel E. 07 Feb '16

07 Feb '16

Vacation Monday 2/8 and Tuesday 2/9. Dan

1 0

Re: [CELS Sysadmins] Network Outage Overnight - Card swap required
by Stacey, Craig 04 Feb '16

04 Feb '16

Does this affect LCF? -- Craig (mobile) From: Schmitz, Corby B. Sent: Feb 4, 2016 7:11 PM To: helpinfo; CELS Admins (sysadmins(a)lists.cels.anl.gov) Sidorowicz, Kenneth V.; Leibfritz, David W.; cspo(a)anl.gov Cc: noc(a)anl.gov; Schmidt, Jack C.; anchor(a)anl.gov; Dust, Jim Subject: Re: [CELS Sysadmins] Network Outage Overnight - Card swap required All: I finally have resolution on this particular issue and I wanted to share what we know, and what we are doing with this knowledge. JTAC took a good deal of time working through the logs and debugging data. It was identified that the memory on the MPC in slot 3 in the border router is experiencing errors. Many of these are correctable. In the instance noted below, it was unable to correct the hardware issue. JTAC ran through other cases where this has happened and decided that the best course of action was to replace the card and move forward. We spent a ittle time ensuring that the errors were not coming from the fabric cards, but that turned out not to be the case. I have taken delivery of the replacement card under RMA and have submitted a change request (CHG#31762) for 0530hrs CST on Tuesday 2/9/2016. It is going through the final approval process within the CAB, but I wanted to check in, in parallel, to ensure that there were no concerns with swapping out the card at that time. The affected connections are listed below: Xe-3/0/0.803,804 – MREN Backup Link (R&E and TR/CPS) – Will not be visible Xe-3/0/0.345,346 – ESnet Tertiary Link – Will not be visible Xe-3/0/1.675 – MCS-240rtr – May experience a momentary blip of under 5 seconds Xe-3/2/3 – CyberSpan - Will lose replicated traffic during the maintenance window (expected to be less than 10 minutes) Xe-3/3/0.1300 – APS-ScienceDMZ – Will lose connectivity during the maintenance window (expected to be less than 10 minutes) Please let me know if there are any issues or concerns. If there is a more suitable timeframe for the swap, please suggest. corby ________________________________ From: Schmitz, Corby B. Sent: Wednesday, January 27, 2016 7:56 AM To: helpinfo; CELS Admins (sysadmins(a)lists.cels.anl.gov) Sidorowicz, Kenneth V. (kvs(a)aps.anl.gov) Dave Leibfritz; cspo(a)anl.gov Cc: anchor(a)anl.gov; noc(a)anl.gov; Schmidt, Jack C.; Dust, Jim Subject: Network Outage Overnight All: At 0050hrs last night, the border router in 541b (noni) suffered a hardware-related event. Linecard #3 clocked a number of errors on traffic received from the chassis backplane. At 0057.16hrs, the chassis health monitor set the system status to Yellow, noting an impact to some traffic flows through linecard #3. At 0057.23hrs, the chassis health monitor set the system status to Red, indicating that the errors were unrecoverable in the current situation and that all traffic through linecard #3 was impacted. This culminated in a chassis health process initiating a module reset at 0057.38hrs, which ceased all traffic across linecard #3. At 0100.11hrs, linecard #3 had restored to operational state and by 0103.05hrs, all related network services had been fully restored. The following links were impacted by this outage: Xe-3/0/0.803,804 – MREN Backup Link (R&E and TR/CPS) – not active at the time of the event Xe-3/0/0.345,346 – ESnet Tertiary Link – not active at the time of the event Xe-3/0/1.675 – MCS-240rtr – networking on alternate path via lulo was not impacted, short outage of 15s was likely perceived Xe-3/0/2.668 – T1-Firewall link deprecated during maintenance weekend – not in production state Xe-3/1/1 – 308Core1-L2 – Layer 2 service backup link, all primary services were unaffected on 100G link to Core541 Xe-3/2/3 – CyberSpan, during this window the cyber monitoring on noni was non-functional – 0050-0103hrs Xe-3/3/0.1300 – APS-ScienceDMZ – impacted throughout the window from 0050-0103hrs We are working with Juniper TAC to identify the root cause of the hardware failure, and take whatever action needed to rectify the situation. There have been no additional indications of fabric errors since the card reset this morning, and no indication prior to the log messages at 0500hrs this morning. More information will be provided as we move through the TAC process. If you have any questions or concerns, please reach out to me directly, or send a note to anchor(a)anl.gov. -- Corby Schmitz Network Communication Operations and Support Manager Computer and Information Systems Division - Network Engineer MREN/Starlight -- Argonne National Laboratory 9700 S. Cass Ave. Argonne, IL 60439 Desk: 630-252-7664 Cell: 630-296-4252 E-mail: cschmitz(a)anl.gov

2 2

Re: [CELS Sysadmins] Network Outage Overnight - Card swap required
by Schmitz, Corby B. 04 Feb '16

04 Feb '16

All: I finally have resolution on this particular issue and I wanted to share what we know, and what we are doing with this knowledge. JTAC took a good deal of time working through the logs and debugging data. It was identified that the memory on the MPC in slot 3 in the border router is experiencing errors. Many of these are correctable. In the instance noted below, it was unable to correct the hardware issue. JTAC ran through other cases where this has happened and decided that the best course of action was to replace the card and move forward. We spent a little time ensuring that the errors were not coming from the fabric cards, but that turned out not to be the case. I have taken delivery of the replacement card under RMA and have submitted a change request (CHG#31762) for 0530hrs CST on Tuesday 2/9/2016. It is going through the final approval process within the CAB, but I wanted to check in, in parallel, to ensure that there were no concerns with swapping out the card at that time. The affected connections are listed below: Xe-3/0/0.803,804 – MREN Backup Link (R&E and TR/CPS) – Will not be visible Xe-3/0/0.345,346 – ESnet Tertiary Link – Will not be visible Xe-3/0/1.675 – MCS-240rtr – May experience a momentary blip of under 5 seconds Xe-3/2/3 – CyberSpan - Will lose replicated traffic during the maintenance window (expected to be less than 10 minutes) Xe-3/3/0.1300 – APS-ScienceDMZ – Will lose connectivity during the maintenance window (expected to be less than 10 minutes) Please let me know if there are any issues or concerns. If there is a more suitable timeframe for the swap, please suggest. corby ________________________________ From: Schmitz, Corby B. Sent: Wednesday, January 27, 2016 7:56 AM To: helpinfo; CELS Admins (sysadmins(a)lists.cels.anl.gov) Sidorowicz, Kenneth V. (kvs(a)aps.anl.gov) Dave Leibfritz; cspo(a)anl.gov Cc: anchor(a)anl.gov; noc(a)anl.gov; Schmidt, Jack C.; Dust, Jim Subject: Network Outage Overnight All: At 0050hrs last night, the border router in 541b (noni) suffered a hardware-related event. Linecard #3 clocked a number of errors on traffic received from the chassis backplane. At 0057.16hrs, the chassis health monitor set the system status to Yellow, noting an impact to some traffic flows through linecard #3. At 0057.23hrs, the chassis health monitor set the system status to Red, indicating that the errors were unrecoverable in the current situation and that all traffic through linecard #3 was impacted. This culminated in a chassis health process initiating a module reset at 0057.38hrs, which ceased all traffic across linecard #3. At 0100.11hrs, linecard #3 had restored to operational state and by 0103.05hrs, all related network services had been fully restored. The following links were impacted by this outage: Xe-3/0/0.803,804 – MREN Backup Link (R&E and TR/CPS) – not active at the time of the event Xe-3/0/0.345,346 – ESnet Tertiary Link – not active at the time of the event Xe-3/0/1.675 – MCS-240rtr – networking on alternate path via lulo was not impacted, short outage of 15s was likely perceived Xe-3/0/2.668 – T1-Firewall link deprecated during maintenance weekend – not in production state Xe-3/1/1 – 308Core1-L2 – Layer 2 service backup link, all primary services were unaffected on 100G link to Core541 Xe-3/2/3 – CyberSpan, during this window the cyber monitoring on noni was non-functional – 0050-0103hrs Xe-3/3/0.1300 – APS-ScienceDMZ – impacted throughout the window from 0050-0103hrs We are working with Juniper TAC to identify the root cause of the hardware failure, and take whatever action needed to rectify the situation. There have been no additional indications of fabric errors since the card reset this morning, and no indication prior to the log messages at 0500hrs this morning. More information will be provided as we move through the TAC process. If you have any questions or concerns, please reach out to me directly, or send a note to anchor(a)anl.gov. -- Corby Schmitz Network Communication Operations and Support Manager Computer and Information Systems Division - Network Engineer MREN/Starlight -- Argonne National Laboratory 9700 S. Cass Ave. Argonne, IL 60439 Desk: 630-252-7664 Cell: 630-296-4252 E-mail: cschmitz(a)anl.gov

1 0

Tina-WFH
by Stanton-Piersanti, Tina Joy 02 Feb '16

02 Feb '16

I'm going to WFH today so I don't spread my cold germs further. T.

1 0

[CELS Sysadmins] where's Linda 2/1 update
by Winkler, Linda 01 Feb '16

01 Feb '16

Working from home this afternoon lw

1 0

[CELS Sysadmins] Getting support for computer issues
by Stacey, Craig 01 Feb '16

01 Feb '16

Hi. First, let me introduce myself. I’m Craig Stacey, the IT manager for CELS. I’ve met some of you already, and for those of you I haven’t yet met, I’m sure we’ll cross paths in the coming weeks. First off, I apologize for not sending this mail sooner, but we’ve been working on the details and I wanted to have something meaningful to tell you. As you’re no doubt aware, Rocky left Argonne a couple of weeks ago for what I’m sure we all hope is a wonderful opportunity elsewhere. No doubt many of you are wondering what to do for your IT support issues. As of January 25, my group (The CELS Systems team) took on responsibility of the IT support needs for BIO. We’re still figuring out the general lay of the land and investigating how to best roll you into what we’re doing over here, but for now things won’t change a whole lot, except it might take us a little longer to figure out issues and address them for you. Rocky knew this environment like the back of his hand, and we’re still in the discovery stages here. So, how do you go about getting support? The short answer is you can send mail to help(a)cels.anl.gov. That will create a support ticket for the CELS Systems Team to look into the issue. Phone support can be found at x6813. We’re still in the process of updating our documentation to include more relevant info for the Bio division, but these links might help you get a handle on what we do: * http://virtualhelpdesk.cels.anl.gov – A good starting out point. Links to most of the below, and includes forms for commonly requested items. * https://mcssys.wordpress.com/ - This is our blog where we post announcements of relevance to our users. (The MCS name comes from when we were the IT support for only MCS.) * https://twitter.com/mcssys – Mostly links to blog posts above, helpful for being notified during critical outages. * http://www.mcs.anl.gov/help – Our user documentation page. Most of this applies to the existing IT infrastructure in building 240, but we’ll update it with BIO-specific docs as we get more acquainted with the environment. I’ll have more information for you soon, and we’re planning a division meeting on the topic as well, so stand by for that. Thanks! — Craig Stacey IT Manager, CELS

1 0