Initial failure
On the evening of December the 15th, we discovered that a few of the floor tiles in our server room has broken. The result is that two of our racks now has a leg with no support, and the racks has become "a bit" tilted (see picture below). Awaiting carpenter work, we have removed five compute nodes from the most tilted rack to reduce weight, and the rack has been supported by a steel beam.
Implemented fix
- Shutdown and remove all servers from all eight racks. Leave the switches and cabling. Move the racks, so the carpenters can rebuild the floor.
- Build support framing for the legs that supports the floor tiles. Replace the floor tiles with thicker tiles.
- Put all servers back in the racks, and make sure everything still works as normal.
Event log
| Time | Event |
|---|---|
| 16.12.23 - 20:22 | The broken floor tile was discovered |
| 16.12.23 - ca. 23:00 | Agreed that we should remove some weight from R3, and contact help on Monday |
| 18.12.23 - ca. 09:00 | Contacted a company that will assess the damage, and come up with a plan to fix the floor |
| 18.12.23 - ca. 12:00 | Placed a steel beam under R3, to support it. Migrated all VMs from five of the compute nodes in R3, and removed them from the rack |
| 20.12.23 | Visit from the carpenter company. Made an initial plan for what needed to be done. Decided that the steel beam would suffice for support. Little to no measured further "sinking". |
| 17.01.24 - 10:00 | Meeting with the carpenter. A plan was made for repairing the floor. They will build support framing between all the floor tile legs, and replace all necessary tiles. The carpenters will need two days, and we schedule two days for removing all servers, and one day to put everything back in after the floor is fixed. Meaning a total downtime of five days. |
| 18.01.24 - 13:00 | Received confirmation from the carpenter, that they can start the repair work on 6th of February. We accepted the offer. |