Fwd: Torque, Maui, Moab, and Adaptive Computing products.
-- Craig Begin forwarded message:
From: [email protected] (Gene Rackow) Subject: Torque, Maui, Moab, and Adaptive Computing products. Date: March 11, 2013 2:22:55 PM CDT To: [email protected], [email protected]
Sounds like anyone using adaptive computing products, there is a hole in the authentication stuff that allows hackers in. This appears tied to some of the cluster infections happening at other sites.
It's not clear which of the products may be vulnerable. At least the Torque product has holes and the recommendation is to upgrade to 4.1.5 or 4.2.1.
They are still investigating if this impacts MAUI or MOAB. It sounds like MOAB may be clean, but just in case.
Gene Rackow writes:
Sounds like anyone using adaptive computing products, there is a hole in the authentication stuff that allows hackers in. This appears tied to some of the cluster infections happening at other sites.
It's not clear which of the products may be vulnerable. At least the Torque product has holes and the recommendation is to upgrade to 4.1.5 or 4.2.1.
More details would be useful, particularly since the authentication stuff in torque changed with torque 4.0, and so maybe pre-4.0 isn't vulnerable? Specificially, before torque 4.0, torque (plus other PBS-based systems, like PBSpro and OpenPBS) used a set-uid root executable, pbs_iff, to "authenticate" users on clients to the PBS server. I'm not exactly sure why that executable needs to be set-uid root (the network ports torque/pbs uses aren't in the reserved range, so that's not why), but torque client commands that talk to the server will complain and not work if it isn't. In torque 4.0 and greater, pbs_iff was replaced by a daemon, trqauthd, that runs as root on each client. Client commands talk to the daemon when "authenticating", and the daemon in turn talks to the server (but once authenticated, I believe the clients then talk directly to the server). Both designs are susceptible to attack, but the former (set-uid root exectuables) usually requires that the attacker is already on the vulnerable system (eg, using a compromised user account). A firewall could protect the latter from remote exploit, but it would still be susceptible to local attack. pbs_iff has been around for a long time, as have attacks on setuid executables, so one would hope any security holes in pbs_iff have been eliminated long ago, but of course, being around for a long time doesn't guarantee security. :) trqauthd is fairly new, so I wouldn't be surprised if there are security related bugs in it, but bugs in root-running daemons have likewise been long known as a vector for remote compromise, so one would expect (well, hope :) ) the designer/programmer of trqauthd took that into account.
They are still investigating if this impacts MAUI or MOAB. It sounds like MOAB may be clean, but just in case.
The maui and moab daemons talk to the pbs server as clients, and none of the maui/moab end-user commands are setuid-root, so I would think that there are no related security issues with maui or moab. Although perhaps one of the end user commands could be leveraged to have maui or moab trigger some sort of overflow on the pbs server, or to cause the pbs server to do something it shouldn't be doing (eg, run a command it shouldn't). So more details would be good. :) John
I don't have more details on this. Torque has been on the possible hit list involved in a number of attacks at the other teragrid/xsede related site compromises. Jim Marsteller at PSC had some preliminary info from Adaptive Computing, but nothing official. There should be a CVE and announcement coming from them soon. It sounded like MOAB may not be vulnerable, but all things considered, I'm not sure how/why they are making that claim. Be on the lookout for software updates in that space. John Valdes made the following keystrokes:
Gene Rackow writes:
Sounds like anyone using adaptive computing products, there is a hole in the authentication stuff that allows hackers in. This appears tied to some of the cluster infections happening at other sites.
It's not clear which of the products may be vulnerable. At least the Torque product has holes and the recommendation is to upgrade to 4.1.5 or 4.2.1.
More details would be useful, particularly since the authentication stuff in torque changed with torque 4.0, and so maybe pre-4.0 isn't vulnerable?
Specificially, before torque 4.0, torque (plus other PBS-based systems, like PBSpro and OpenPBS) used a set-uid root executable, pbs_iff, to "authenticate" users on clients to the PBS server. I'm not exactly sure why that executable needs to be set-uid root (the network ports torque/pbs uses aren't in the reserved range, so that's not why), but torque client commands that talk to the server will complain and not work if it isn't.
In torque 4.0 and greater, pbs_iff was replaced by a daemon, trqauthd, that runs as root on each client. Client commands talk to the daemon when "authenticating", and the daemon in turn talks to the server (but once authenticated, I believe the clients then talk directly to the server).
Both designs are susceptible to attack, but the former (set-uid root exectuables) usually requires that the attacker is already on the vulnerable system (eg, using a compromised user account). A firewall could protect the latter from remote exploit, but it would still be susceptible to local attack.
pbs_iff has been around for a long time, as have attacks on setuid executables, so one would hope any security holes in pbs_iff have been eliminated long ago, but of course, being around for a long time doesn't guarantee security. :) trqauthd is fairly new, so I wouldn't be surprised if there are security related bugs in it, but bugs in root-running daemons have likewise been long known as a vector for remote compromise, so one would expect (well, hope :) ) the designer/programmer of trqauthd took that into account.
They are still investigating if this impacts MAUI or MOAB. It sounds like MOAB may be clean, but just in case.
The maui and moab daemons talk to the pbs server as clients, and none of the maui/moab end-user commands are setuid-root, so I would think that there are no related security issues with maui or moab. Although perhaps one of the end user commands could be leveraged to have maui or moab trigger some sort of overflow on the pbs server, or to cause the pbs server to do something it shouldn't be doing (eg, run a command it shouldn't).
So more details would be good. :)
John
Below is the latest from Adaptive. The problem appears to be trqauthd. Even if you are not using that, but "Munge" as the auth server, if trqauthd is running you have a hole. It's not a remote exploit if you limit access to the torque servers. It sounds like this was found at ORNL and making its way to other sites now that it's been announced. This bug only exists in version 4 of the tree, but indications are that versions back to 2.4(at least) are vulnerable to similar problems. ---Gene From the Adaptive website: < http://marketing.adaptivecomputing.com/acton/rif/3697/s-0029-1303/-/l-sf-cl-...
From: Adaptive Computing <[email protected]> Reply-To: Adaptive Computing <[email protected]> Date: Tuesday, March 12, 2013 9:00 AM To: Bill Kramer <[email protected]> Subject: TORQUE 4.0+ Security Vulnerability
Click here to view this message in a browser window.
A security vulnerability has been discovered in versions of TORQUE released after and including version 4.0.0. The vulnerability is present between the client commands and trqauthd daemon.
Anatomy of the vulnerability
The client commands in TORQUE first connect to pbs_server which listens on port 15001. When the client connects it uses a dynamic port on its side of the connection. The client then connects to trqauthd sending the user name of the person issuing the command and the dynamic port on which it made the connection to pbs_server. trqauthd then makes a connection to pbs_server on a privileged port between 144 and 1023. Because the connection came from a privileged port pbs_server knows the connection came from a root process and assumes it is trqauthd. trqauthd then vouches for the connection on the dynamic port of the client and the server authorizes the connection based on the user name.
The vulnerability is exploited using the LD_PRELOAD environment variable which allows users to tell the OS to process the libraries in the LD_PRELOAD list before any other libraries. The user can then create their own libraries with their own API functions to be called. In the case of this vulnerability the user could create a getuid() command and then claim the connection came from any user including root.
Because TORQUE does not allow jobs to be submitted as root no jobs would have been submitted and executed as root. However, users would be able to use the UID of other users and submit jobs. Commands like qmgr could be called as root or as the TORQUE manager. The command qstat which normally only allows users to see their own jobs could be called as root and users would be able to see all jobs in the system. An examination of each utility considering what could happen using root or another user id will let administrators understand what kind of problems may have occurred if users exploited this vulnerability.
Remedy
Users are encouraged to upgrade to either TORQUE 4.1.5 or 4.2.1. These can be downloaded from GitHub from the adaptivecomputing organization at https://github.com/adaptivecomputing/torque or users can download a tarball at http://www.adaptivecomputing.com/support-old/download-center/torque-download .
The fix for this problem was to replace the use of TCP with Unix Domain Sockets for the communication between client utilities and trqauthd. Unix Domain Sockets give trqauthd the ability to know the process id, user id and group id of the process opening the connection. This cannot be spoofed using the LD_PRELOAD attack. To see the code change to 4.2.1 you can visit https://github.com/adaptivecomputing/torque/commit/fcf9f4cdc87d1a06c5cb95fe0...
To see the code change for 4.1.5 visit https://github.com/adaptivecomputing/torque/commit/9027614d17bd82fce88f5b0bd...
Thanks,
Adaptive Computing Team
(801) 717-3700 Phone
participants (3)
-
Craig Stacey -
John Valdes -
rackow＠anl.gov