Investigation into the Issues surrounding CO.ZA

Following Delegation Signer (DS) records to the .za space on the 5th September 2019, in preparation for a Key Signing Key (KSK) rollover we started receiving reports of issues on some validating name servers. 

The issue was eventually traced back to old buggy versions of Unbound that had been standard in older versions of both Ubuntu and Redhat. In order to solve this we removed the higher security SHA-384 DS keys from the .ZA zone and left the SHA-256 keys in place.

Knot has better documentation, better support, is easier for the junior staff to understand, and regular releases.

As part of this process the Key Signing Key (KSK) on both .za and .co.za has to change. The annual KSK rollovers are also due as per the ZADNA and ZACR DNSSEC Policy Statements (DPS).

The KSK has to be anchored in a parent zone using a DS (Delegation signer) record. Since knot signer automatically generates DS keys for both type 2 (SHA-256) and type 4 (SHA-384) these were both added to the .za zone for .co.za 

After it was added, the zone was checked using internal tools and using the popular  http://dnsviz.net/ checking tool. At that time, no issues were found. 

Later that day, we received reports from some sites that the DNSSEC validation of anything under .co.za was failing on some name servers but not others.  Upon further investigation we found that the problem seemed to be occurring on sites running the popular recursive name server package Unbound. 

Attempting to reproduce on newer versions of Unbound failed.  With the assistance of several members of the community we eventually isolated the issue to versions of Unbound  1.6.0 and older. We believe it is the bug reported at https://nlnetlabs.nl/bugs-script/show_bug.cgi?id=1449  The fix for this has been out since April 2017. It appears this older version of Unbound was common on both Redhat EL6 and Ubuntu 16.04.

Since both those Operating systems are still relatively common we decided it was safer to remove all SHA-384 digests from the .ZA zone and leave only the SHA-256 hashes.  Once the type 4 digests had been removed the problem immediately went away.

We are in the process of updating our formal policies to only use SHA-256 DS keys until IANA lists its requirement as mandatory, as it is currently optional.

We would like to thank ZADNA and Cedrick Lumadi in particular for his quick assistance in removing the SHA-384 digests. Thanks also go to Ed Pascoe for his quick response from DNS Africa and for compiling the Incident Report following the issue. We also offer grateful thanks to the co.za community at large for helping to solve this problem.

In order to combat this issue before it reoccurs in the future, it is necessary for customers to upgrade their software if they are running validation resolvers. The outdated software that is still in place is unable to handle these keys and hence, issues such as the above continue to arise.