KSK Ceremony 52 Debrief Notes

[Aaron Foley]

Dear Colleagues,

We held a debrief session on 14 February 2024 directly following the recent KSK Ceremony 52. Ten participants took part, and the overall feedback was that the ceremony was successful. Below is a summary of the discussion, along with any applicable action items:

The perspective of ceremony participants was positive.

Clarity and transparency were maintained.

A question was raised about the value of having a PIN associated with the smartcards when it’s printed in the script. We explained that we would avoid the PIN if possible as we rely on physical rather than logical security regarding the credentials.
-It was also noted that the credentials for the new HSM hardware scheduled for introduction do not require a PIN and will not be used.

The topic of ceremony computer equipment was raised. Particularly, the availability of laptops with unique features like removable batteries, no wi-fi card, removable hard drives, optical drives, etc. Whether we have spare laptops in case of hardware failure, and the topic of UEFI as a possible attack vector was also raised in relation.
-RKOS explained the motivation to move toward SD cards for the operating system was driven by the scarce availability of laptops with built-in optical drives, and the laptop form factor is still most desirable to use from a security perspective, as the entire unit can be locked away in the safe with chain of custody proven for all critical computer components between ceremonies. The Dell rugged line of notebooks continues to offer laptops with the other desirable features.
-RKOS also noted that laptop replacements are on our roadmap in the next two years, and if we had a failure before then, we still have one more brand-new laptop of the same exact type that could be introduced, at which point the replacement schedule would be accelerated.

The topic of CO2 levels in the ceremony room was also raised. It was noted that at a recent conference a ceremony attendee had been to in a less densely populated room, CO2 levels had been measured to a level known to have negative cognitive effect. It was also noted the room felt warmer than usual.
-RKOS will raise the issue with the facility provider to test the HVAC and explore options to monitor CO2 levels. More frequent ceremony breaks with the air purifier running should also help to address these concerns.

During the HSMFD backup process in the penultimate act, 2 separate USB drives failed to be recognized by the laptop. 2 brand new spare USB drives stored in the KMF were used to replace the failed units without incident. The failed drives had been formatted and tested by RKOS just a few days prior before they were brought to the key management facility. We assume there could have been a bad batch of USB drives, as these were the first 2 failures we’ve observed since 2019 when we began to use these drives exclusively.
-RKOS further explained that this was a bit troubling, and we plan to incorporate a more thorough testing regime prior to introduction. We also perform HSMFD backups in duplicate to ensure there are no single points of failure for similar reasons.

Jorge Etges’ safe deposit box lock mechanism is not operating as smoothly as it should, and there was some difficulty opening the box both times in this ceremony.
-RKOS will schedule Jorge to migrate to a new safe deposit box at the earliest logical opportunity, and the lock mechanism will be replaced in a future administrative ceremony after the box is vacated.

The topic of the safes was raised, particularly having them mounted directly to the floor, and whether raising them would be an option, and how we would address a situation where a TCR was physically unable to open their safe deposit box.
-RKOS noted that while the location is not ideal for ease of use, they are bolted directly to the floor for security purposes, and due to space constraints and production material, it would be very difficult and costly to update the configuration. We also noted that if the TCR were physically present, they could allow another individual to operate their key and retrieve materials on their behalf.

It was noted during the ceremony that there was some difficulty in determining which way safe deposit box keys should be facing to operate their respective locks, and there should be some direction in that regard.
-RKOS have already made updates to the script templates to ensure that this is well defined in future ceremonies

The topic of conflicting signage was brought up: There are signs that say no food & drink, no cell phones and cameras, although these things are allowed.
-RKOS explained that outside food & drink is still planned upon as all cleaning is performed by CBO staff, which is why we only allow water bottles with lids. We will consider removing the signs about cell phones and cameras.

A question was raised about how we were able to modify the configuration of the copy-hsmfd script before it was executed when the operating system is read-only.
-RKOS explained that COEN uses squashFS to load the operating system into memory on bootup, and commands are executed from a sort of RAM disk while in use, which allows us to modify the files in memory without affecting the SD card or its contents.
-Upon further examination a more thorough explanation is this:
COEN is Debian-based and uses a combination of squashfs mounted in read only to access the system image along with a tmpfs located in memory that’s mounted in read/write mode. The two are combined in what’s called a unionfs. It's a virtual filesystem which combines the contents of two directories, giving you the illusion of a single filesystem, backed by two other directories. By combining the two filesystems (the read-only squashfs and the read-write but blank tmpfs), you get a read-write filesystem in which changes are just stored in the tmpfs. Other files not contained in the tmpfs are read directly from the read-only squashfs. This allows live changes to be made in RAM while not affecting the contents of the operating system SD card.

Best Regards,

Aaron Foley
Cryptographic Key Manager: IANA
Aaron.Foley at iana.org<mailto:Aaron.Foley at iana.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mm.icann.org/pipermail/root-dnssec-announce/attachments/20240220/3ed3428b/attachment.html>