Anomalous behavior of the DNS on March 24th, 2010

Inicio

Because of the interest of the press about the discovery of anomalies in the functioning of DNS servers located in China, NIC Chile reports:

On Wednesday March 24th, 2010, thanks to information provided by engineers of VTR (a Chilean ISP), Mauricio Vergara, DNS Admin for .CL in NIC Chile, communicated to an operators mailing list managed by DNS-OARC, an anomalous behavior that involved what seems to be a data alteration on DNS responses in one of the servers from the "I" root-server, located in China. The anomalous DNS responses affected domain names such as Facebook.com, Twitter.com and YouTube.com

1.- Is the problem still ongoing?

We now know that Autonomica/Netnod (Sweedish organization, operator of root-server "I") shut down the connection to the node located in China. We haven't detected that the anomalous behavior continues.

2.- Do you know which ISPs where affected?

On March 24th, we were only able to detect this problem in Chile from the ISPs VTR and Telmex.

We also tried to see if this problem was affecting only Chile, so we checked on a California server operated by NIC Chile, and we detected the same behavior.

3.- What would explain that someone made announcements of a root server from China to the rest of the world?

DNS Queries to root servers are made to any of the 13 servers located around the world. These servers are identified by a letter, from "A" through "M". The "I" root server, like almost all the other ones, is a group of more than 40 servers located in different cities on every continent (this is called an "anycast cloud"). It happened that this time, one server located in China seemed to be "closer" (in term of Internet routes) to any other server from the "I" root server (for instance, it seemed closer than a US or a European one). More information about this could be provided by Autonomica/Netnod.

4.- Do you think this is an accident?

To reach a Chinese route from Chile, is something that can happen during normal operation of the Internet. In fact, the Internet works in this way, and it always tries to find a path that is considered "closer". Depending on the configuration of the conections made between Internet providers, China can seem to be closer than other countries that have a shorter physical distance.

Although we can't be 100% sure, we think this anomaly could be an unintended consequence of the use of filters or modifying DNS contents aparently done in China

5.- When did you notice this problem?

On Wednesday March 24th, around 10:00 AM CLST, when Pablo Jimenez (IP Service and monitoring engineer from VTR) contacted Mauricio Vergara Ereche, DNS Admin for .CL in NIC Chile, to ask if NIC Chile was seeing this anomalous behavior. Pablo commented that they have been seeing this for at least a couple days.

VTR seems to be the first organization to detect this behavior, and they came to NIC Chile to see how they could contact people in charge of the "I" root-server, or if NIC Chile have seen something similar.

After that, we started to study the case, we made some tests, and we detected that the problem was probably affecting the I root server located in China. At that moment, Mauricio Vergara contacted directly to Autonomica/Netnod, and explained them what NIC Chile had detected. After that, Pablo Jimenez also contacted Autonomica/Netnod. Almost at that same time, NIC Chile informed this behaviour to a coordination mailing list for DNS hosted by DNS-OARC, an association in which NIC Chile is a member. DNS-OARC has as a main objective to be an Operational and Analysis Research Center for DNS world. Usually DNS-OARC is ahead in case of events like this one.

6.- Did NIC Chile have problems with Facebook, YouTube and Twitter?

When we made DNS queries to those sites, we detected that an odd situation was going on. The problem didn't occur every time, but we were able to see that some answers were anomalous, and they seemed to have been modified along the way. This was clearly an error in the operation of the Internet DNS.

7.- Could this problems be attributed to the Chinese firewall?

We are not certain about that at this point, but it seems likely.

8.- To whom was directed the e-mail from Mauricio Vergara?

As we stated before, the e-mail was directed to a mailing list administered by DNS-OARC.

9.- Is this event something to be worried about?

If this was a situation that continues on time, it would definitely be something to be very worried about. Modifying data or filtering contents on the DNS affects the normal and correct way that the Internet works.

Fortunately this odd behavior has not continued.

10.- Can we prevent this behavior or are we at the mercy of foreign servers?

Today we are in the process of implementing a global system (that NIC Chile is also part of) with security extensions for the DNS protocol known as DNSSEC, which helps to authenticate the origin of the DNS responses using cryptographic alghoritms to assure that the content and origin of the responses are really from whom it corresponds.

Further information:

NIC Chile
Santiago, March 29, 2010.