More info continues to come in... Turns out the cause of failure is worse than I expected. In the details from other sites, it appears it's the response to a DNS request that is causing it's death. Something caused a message to need to be sent from a .edu site. In the process of their mail server looking up the MX record on the malicious domain their DNS server crashed. Indicactions of an invalid response to the DNS query. Yukkk. My guess is we could pull some interesting bits out of pcap at this time about what was being looked up, then possibly tie that back to the host doing the lookup.
DNS format error from 61.151.253.94#53 resolving jfjb.c0m.cn/MX for client XXXXXXXXXXXXXX#30063: invalid response
--gene Brian Sebby made the following keystrokes:
Gene,
It appears that we were hit with this, too. This morning I noticed that our DNS servers restarted themselves last night, and thanks to your pointers to what to look for, found this in titania's log:
Nov 15 22:31:26 titania named[11364]: query.c:1781: INSIST(! dns_rdataset_isasso ciated(sigrdataset)) failed, back trace Nov 15 22:31:26 titania named[11364]: #0 0x412ceb in assertion_failed()+0x4b Nov 15 22:31:26 titania named[11364]: #1 0x55bfea in isc_assertion_failed()+0xa Nov 15 22:31:26 titania named[11364]: #2 0x419b8f in query_addadditional2()+0x10 9f Nov 15 22:31:26 titania named[11364]: #3 0x4bbae2 in dns_rdata_additionaldata()+ 0x1a2 Nov 15 22:31:26 titania named[11364]: #4 0x4e82cb in dns_rdataset_additionaldata ()+0xdb Nov 15 22:31:26 titania named[11364]: #5 0x4167c3 in query_addrrset()+0x253 Nov 15 22:31:26 titania named[11364]: #6 0x41c922 in query_find()+0x2c42 Nov 15 22:31:26 titania named[11364]: #7 0x41de69 in query_resume()+0x1f9 Nov 15 22:31:26 titania named[11364]: #8 0x575ee4 in isc__taskmgr_dispatch()+0x1 d4 Nov 15 22:31:26 titania named[11364]: #9 0x578cbf in evloop()+0x8f Nov 15 22:31:26 titania named[11364]: #10 0x578ef0 in isc__app_ctxrun()+0x130 Nov 15 22:31:26 titania named[11364]: #11 0x413ced in main()+0x59d Nov 15 22:31:26 titania named[11364]: #12 0x7fc9376c01c4 in _fini()+0x7fc9371383 20 Nov 15 22:31:26 titania named[11364]: #13 0x405789 in _start()+0x29 Nov 15 22:31:26 titania named[11364]: exiting (due to assertion failure) Nov 15 22:32:23 titania named[25139]: starting BIND 9.7.3-P3 -d 1 -c /etc/iscbin d/named.conf Nov 15 22:32:23 titania named[25139]: built with '--prefix=/etc/iscbind/bind/' ' --sysconfdir=/etc/iscbind' '--mandir=/usr/share/man' '--with-openssl=/usr' '--in fodir=/usr/share/info' Nov 15 22:32:23 titania named[25139]: using up to 4096 sockets Nov 15 22:32:23 titania named[25139]: loading configuration from '/etc/iscbind/n amed.conf'
This hit oberon at about the same time, with the same log entries and restart.
Fortunately, we have a script to monitor named, and it immediately restarted it on both servers after they crashed.
Strangely, these are our tier 1 DNS servers that are not accessible from the internet, and nothing hit our tier 1 DNS servers. We have some giant core files from these crashes, are they of any value to you?
I am on the ISC BIND mailing list and will be looking for an update to fix this. Once I have that, I'll install it ASAP and will keep everyone informed of the status.
Brian
On Wed, Nov 16, 2011 at 09:36:54AM -0600, [email protected] wrote:
At least 4 other universities have chimed in that they too have been hit with this attack killing 1 or more of their servers.
Gene Rackow made the following keystrokes:
Pass this on to others in your areas that are running DNS servers.
Hitting the news wires this morning are the fact that someone has found a way of crashing Bind9. Not sure on exact versions yet, but in one of the messages, a .edu site indicated they have run into problems with this already. The log files have:
Nov 15 23:36:31 admdns2 named[33018]: /usr/src/usr.sbin/named/../../contrib/bind9/bin/named/query.c:1781: INSIST(! dns_rdataset_isassociated(sigrdataset)) failed
Nov 15 23:36:31 admdns2 named[33018]: exiting (due to assertion failure)
This was using BIND 9.6-ESV-R3, the version in FreeBSD 8.2's base installation. It's not clear what versions the problem exists in yet. The story doesn't indicate a which, if any are safe. Expect updates across the board soon. -_Gene
------- End of Forwarded Message http://www.isc.org/software/bind/advisories/cve-2011-tbd
BIND 9 Resolver crashes after logging an error in query.c Organizations across the Internet are reporting crashes interrupting = service on BIND 9 nameservers performing recursive queries. Affected = servers crash after logging an error in query.c with the following = message: "INSIST(! dns_rdataset_isassociated(sigrdataset))" Multiple = versions are reported as being affected, including all currently = supported release versions of ISC BIND 9. ISC is actively investigating = the root cause and working to produce patches which avoid the crash. = Further information will be made available soon. CVE: CVE-2011-TBD Document Version: 1.0 Posting date: 16 Nov 2011 Program Impacted: BIND Versions affected: Multiple version of BIND 9. Specific versions under = investigation Severity: Serious Exploitable: Unknown Description:=20 Details are being collected and work with the ISC Engineering and = Support Team.
Workarounds:=20 Under Investigation - a workaround patch is currently being tested.
Active exploits:=20 Under investigation Solution:=20 Fixes for all current versions are being developed or are being = investigated
ISC is receiving multiple reports and working with multiple customers on = this issue. Please E-mail all questions, packet captures, and details = [email protected] - - --
------- End of Forwarded Message
hostmaster mailing list [email protected] https://lists.anl.gov/mailman/listinfo/hostmaster
-- Brian Sebby ([email protected]) | Infrastructure and Operation Services Phone: +1 630.252.9935 | Computing and Information Systems Fax: +1 630.252.4601 | Argonne National Laboratory