April 2011 - Core-sysadmins - mailman3.cels.anl.gov

pam changes
by Kenneth Raffenetti 27 Apr '11

27 Apr '11

Just a heads-up, I put my PAM config changes into bcfg2 this afternoon. They be on all the Lucid desktops tomorrow. I've successfully tested sshkey, krb5 passwords, and cryptocard with this setup, but if you see/hear anything unusual from users, let me know. -- Ken Raffenetti Systems Administration Associate MCS Division - Argonne National Laboratory

1 0

Mail delivery tuning
by Craig Stacey 27 Apr '11

27 Apr '11

Ever since the load balancer came into play, all hosts starting sending through mailrelay.mcs.anl.gov, including our list servers. I would like owney (and lists.ci.uchicago.edu) to be able to send directly to the internet again. Going through mailrelay is adding an extra hop to an already long hop list, and it's creating a bottleneck on mailrelay. I had a complaint yesterday that an internal mail message took 10 minutes to get delivered, and it looks like it got caught in a busy mail queue on mailrelay. I've also tweaked some settings on mailrelay (and in bcfg, natch) to up the number of concurrent delivery attempts to hosts designated "relay" in /etc/postfix/transport. The default is 20. I've checked, and zimbra and owney are both configured to accept up to 50 concurrent connections from an SMTP client, so I added relay_destination_concurrency_limit to be 50. I'm hoping the combination of these two things will help. If it still seems slow, we can configure zimbra, owney, and lists.ci to exempt mailrelay from the connection rate limits and up the concurrency again.

1 0

Fwd: [Rt-announce] Security vulnerabilities in RT
by Craig Stacey 22 Apr '11

22 Apr '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 FYI Begin forwarded message: > From: Alex Vandiver <alexmv(a)bestpractical.com> > Date: April 14, 2011 8:59:18 AM CDT > To: rt-announce(a)lists.bestpractical.com > Subject: [Rt-announce] Security vulnerabilities in RT > > In the process of preparing the release of RT 4.0.0, we performed an > extensive security audit of RT's source code. During this audit, > several vulnerabilities were found which affect earlier releases of RT. > We are releasing versions 3.6.11, 3.8.10, and 4.0.0rc8 to resolve these > vulnerabilities, as well as patches which apply atop 3.6.10 and all > versions of RT 3.8. > > RT versions 3.8.0 and above with the "external custom field" feature > enabled and configured are vulnerable to a remote code execution > vulnerability. An authenticated user (either privileged or > unprivileged) can use this vulnerability to execute arbitrary code with > the permissions of the webserver; they may also be tricked into doing so > via cross-site request forgery (CSRF). The external custom field option > is disabled by default; if you have not explicitly enabled > "CustomFieldValuesSources" in your RT configuration, your RT instance is > not vulnerable. We have been assigned CVE-2011-1685 for this > vulnerability. > > RT versions 2.0.0 and above are vulnerable to multiple SQL injection > attacks. We do not believe these attacks to be capable of directly > inserting, altering or removing data from the database, but an > authenticated user (either privileged or unprivileged) could use them to > retrieve unauthorized ticket data. Deployments since 3.6.0 are > additionally vulnerable to a more complex attack, which can be used by a > privileged user to retrieve arbitrary data from the database. We have > been assigned CVE-2011-1686 for this vulnerability. > > RT versions 3.0.0 and higher are vulnerable to an information leak > wherein an authenticated privileged user could gain sensitive > information, such as encrypted passwords, via the search interface. We > have been assigned CVE-2011-1687 for this vulnerability. This > vulnerability is particularly notable given RT's previous vulnerability > with insecure hashing (CVE-2011-0009). > > RT versions 3.6.0 through 3.8.7, as well as 3.8.8 to a more limited > degree, are vulnerable to a malicious attacker tricking the user into > sending their authentication credentials to a third-party server. We > have been assigned CVE-2011-1690 for this vulnerability. > > RT versions 3.2.0 and above are vulnerable to a directory traversal > attack where an unauthenticated attacker can read any file which is > readable by the webserver. While some servers (Apache, nginx) have > safeguards which mitigate this attack, preventing such traversals from > accessing files outside of RT's document root, many others (including > the standalone server provided with RT, plackup, starman, twiggy, and > lighttpd) are vulnerable to this exploit. We have been assigned > CVE-2011-1688 for this vulnerability. > > RT versions 2.0.0 and above are vulnerable to javascript > cross-site-scripting vulnerabilities, which allow an attacker to run > javascript with the user's credentials. We have been assigned > CVE-2011-1689 for this vulnerability. > > > In addition to releasing RT versions 3.6.11, 3.8.10, and 4.0.0rc8, we > have collected patches for 3.6.10 and all releases of 3.8 into a > distribution available for download at this link: > > http://download.bestpractical.com/pub/rt/release/security-2011-04-14.tar.gz > http://download.bestpractical.com/pub/rt/release/security-2011-04-14.tar.gz… > > 7d09b1315785a90d915bdbc86da1a0c9bd017a03 security-2011-04-14.tar.gz > 7898a45b15474641a0f9c381d0f6f58fb34afcc3 security-2011-04-14.tar.gz.sig > > The README in the tarball contains instructions for applying the > patches. > > If you need help resolving this issue locally, we will provide > discounted pricing for single-incident support. Please contact us at > sales(a)bestpractical.com for more information. > > - Alex > > _______________________________________________ > RT-Announce mailing list > RT-Announce(a)lists.bestpractical.com > http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-announce == Craig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAk2x4EAACgkQbVNGA3HOzQvykgCeIGjaXomYQTIpJ6S7uhSeQdkZ cGYAn1eMI0tpqjF5TtJBfrI51mAjTuz6 =3Hfg -----END PGP SIGNATURE-----

1 0

LDAP SSL certs
by Craig Stacey 19 Apr '11

19 Apr '11

I need to get our new Zimbra domains authenticating against our LDAP servers. This requires either certs signed by trusted roots, or me to add the certs manually to Zimbra. Since these aren't generally externally exposed certs, I'm okay with them being self-signed. We also need the zimbra servers (zimbra.anl.gov, zimbra-mb1.anl.gov, zimbra-mb2.anl.gov) to be able to talk to our LDAP servers, so there may be firewall rules that need to be put in place. I can gather the certificates and attempt to get the right holes placed to allow the authentication. Can someone point me at the LDAP server(s) I should be using?

2 1

Monitoring
by Craig Stacey 18 Apr '11

18 Apr '11

Someone *please* tell me we're going to be monitoring the load balancer from nagios before end of day. And someone also tell me we don't have other critical services that aren't being monitored. == Craig

1 0

Where's Corby (reminder)
by Corby Schmitz 18 Apr '11

18 Apr '11

I have some bloodwork to deal with first thing tomorrow (0800hrs). I will work from home tomorrow until the appointment and then be in when I am done being poked. I expect to be in by 0900hrs. The rest of my week is also a mess, but more on that later. Corby Schmitz Sent from my mobile office

2 1

Announcements to users
by Craig Stacey 14 Apr '11

14 Apr '11

From here on out, any time you need to send an outage announcement, use "systems-announce(a)mcs.anl.gov". This will send mail to MCS, ALCF Staff, and CELS Office. Mail to mcs(a)mcs.anl.gov alone isn't sufficient to catch all our users. Eventually, the CI notification list will be put under this as well (once more of the infrastructure is integrated). Thanks.

1 0

Fwd: [mcs-systems #70367] Urgent: www.alcf.anl.gov is down
by Craig Stacey 14 Apr '11

14 Apr '11

dhclient was running on the box. At 23:10, DHCP it somehow managed to get itself a DHCP address of 140.221.8.36 (ie, vserver1's address). All the interfaces broke, and we were offline. Gory details in mud scrollback. Short story is we eventually figured out which server was wilbur-1 (vserver1) and started looking at possible causes. After much self doubt as to whether I was on the safe wilbur or the compromised wilbur, I figured this out, killed dhclient, updated hostbase to remove that MAC addr from DHCP, crossed my finger and did an /etc/init.d/networking restart. And things magically came back to life. Begin forwarded message: > From: Craig Stacey <stace(a)mcs.anl.gov> > Date: April 14, 2011 12:47:03 AM CDT > To: "David E. Martin" <dem(a)alcf.anl.gov> > Cc: systems(a)mcs.anl.gov, Max trefonides <maxadam(a)mcs.anl.gov>, uso-staff(a)lists.alcf.anl.gov, Grey Council <gc(a)alcf.anl.gov> > Subject: Re: [mcs-systems #70367] Urgent: www.alcf.anl.gov is down > > Will do, though we're at the point of turning in for the night since we're not making progress and are unable to raise the right people without waking families. > > I'll just remind folks that we're not a 24 hour support organization, but we'll do our best to get this (and the other affected hosts) up as quickly as possible. > > > On Apr 14, 2011, at 12:29 AM, David E. Martin wrote: > >> Keep me posted on the progress and fix. >> >> -David >> >> On 4/14/11 12:22 AM, Craig Stacey wrote: >>> No. (Not to be facetious, but if I could, I would have.) >>> >>> On Apr 14, 2011, at 12:21 AM, David E. Martin wrote: >>> >>>> Can you switch to the backup server? >>>> >>>> -David >>>> >>>> On 4/14/11 12:21 AM, Craig Stacey wrote: >>>>> It's down, and we're not sure what the cause is at this point with the people we've been able to raise. It's likely to be down until the morning at this point. >>>>> >>>>> On Apr 13, 2011, at 11:53 PM, David E. Martin wrote: >>>>> >>>>>> Thanks. Call me at 312 952 7232 if I can do anything to help. >>>>>> >>>>>> -David >>>>>> >>>>>> On 4/13/11 11:38 PM, Max trefonides wrote: >>>>>>> Looking in to it, >>>>>>> >>>>>>> -- >>>>>>> max >>>>>>> >>>>>>> On Apr 13, 2011, at 11:30 PM, David E. Martin wrote: >>>>>>> >>>>>>>> www.alcf.anl.gov is down. wiki.alcf.anl.gov and www.anl.gov are fine. >>>>>>>> Please make this a high priority. >>>>>>>> >>>>>>>> -David >>>>>>>> >>>>>>>> >>>>>>>> Quick check for: www.alcf.anl.gov (http) >>>>>>>> >>>>>>>> *Date* *City* *Country* *Time (s)* *Result* >>>>>>>> April 13, 2011 21:29:08 San Francisco United States 0.640 *Failed* >>>>>>>> April 13, 2011 21:29:09 Chicago United States 3.120 *Failed* >>>>>>>> April 13, 2011 21:29:12 New York United States 3.064 *Failed* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -David >>>>>>>> >>>>>>>> www.alcf.anl.gov is down. wiki.alcf.anl.gov and www.anl.gov are fine. Please make this a high priority. >>>>>>>> >>>>>>>> -David >>>>>>>> Quick check for: www.alcf.anl.gov (http) >>>>>>>> >>>>>>>> Date City Country Time (s) Result >>>>>>>> April 13, 2011 21:29:08 San Francisco United States 0.640 Failed >>>>>>>> April 13, 2011 21:29:09 Chicago United States 3.120 Failed >>>>>>>> April 13, 2011 21:29:12 New York United States 3.064 Failed >>>>>>>> >>>>>>>> -David >>>>> == >>>>> Craig >>>>> >>> == >>> Craig >>> >> > > == > Craig > == Craig

1 0

Meeting Notes: Changes to MCS web server infrastructure
by Craig Stacey 13 Apr '11

13 Apr '11

1 0

Student resume
by Craig Stacey 13 Apr '11

13 Apr '11

http://cs.unh.edu/~arr68/resume.pdf This is one of the students we're bringing in. -- Craig

1 0