10/10/22 2:46 PM Grid time - Hardware Crash
OVH support reported a hardware crash of the Discovery Grid Server. It is likely repeated machine check error recovery was happening previous to the crash, causing slow performance.
10/10/22 3:19 PM Grid time - Hardware Repair & Recovery
All the RAM modules in the grid server were replaced by OVH and the grid server was restarted.
Initial checks show no problems. Database integrity checking is underway. All database tables have passed integrity tests. fsassets table is still undergoing integrity checks (it is huge).
10/10/22 4:50 PM Grid time - Database Integrity is OK
All grid service database tables are clean. For complete safety, rolling restarts of regions will commence at 6:30PM grid time. No region is restarted while there are avatars present., so there is no need to logout if you are already inworld.
Monday, October 10, 2022