Internet Identity

On May 12, DENIC, the .de domain registry, suffered a technical problem that caused roughly two-thirds of the domains registered in the .de TLD to disappear from the Internet for approximately two hours.  IID's ActiveTrust DNS system identified the trouble starting at 04:30 Pacific time (11:30 UTC)  for the .de domains we track as part of our general Internet industry set, including google.de, paypal.de and schlund.de. 

According to our data, the outage lasted for approximately two hours until 6:13 Pacific time (13:13 UTC), during which time some of the .de root servers returned a response of "NXDomain" for those domains, meaning that they technically did not exist for the portion of lookups that went to the affected servers.

(This post updated as of 5/17/10 to reflect DENIC's post-mortem report on the outage.)

DENIC realized that its root nameservers weren't properly answering queries at around 11:30 UTC and dispatched an emergency response team to locate and neutralize the problem. The incomplete zone file ended somewhere after domains starting with fa (so facebook.de resolved, but ford.de and on was reported as NX).  DENIC claims to have had the problem completely solved by 13:45 UTC. ISP servers may have cached the erroneous data for up to two more hours, until at the latest 15:45 UTC.

Here is a message about the incident sent to the DNS community  from DENIC official Peter Koch:

Dear colleagues,

as already discussed on some operational mailing lists, the DE nameservice faced an incident earlier today (2010-05-12). Starting around 11:30 UTC and lasting until 13:45 UTC several of the authoritative nameservers for the DE top level domain returned NXDOMAIN responses for a yet to be determined number of DE domains that existed in our registration database.  At that later point in time, all affected servers had either been disabled or fed with an earlier version of the DE zone.
Regular operations were eventually resumed at 15:00 UTC.

The negative caching TTL for DE is set to 7200, leading to a latest presence of such NXDOMAIN responses at 15:45 UTC, provided regular handling of both NXDOMAIN and negative TTLs was in use at caching resolvers.

The issue has been dealt with by an internal incident response team.
DENIC will provide further details after a root cause analysis.

Please refer to http://www.denic.de/en/denic-in-dialogue/news/ for future announcements during the upcoming days. {News might be provided in English with some delay.}

Kind regards,
Peter Koch, DENIC eG