It Appears That There’s a Mass Emborkening With Windows Server

Tuckerfan · Aug 16, 2023

Basically, there's a random glitch with how the server syncs with atomic clocks, and it throws the servers date settings wildly off.

Windows feature that resets system clocks based on random data is wreaking havoc

A few months ago, an engineer in a data center in Norway encountered some perplexing errors that caused a Windows server to suddenly reset its system clock to 55 days in the future. The engineer relied on the server to maintain a routing table that tracked cell phone numbers in real time as they moved from one carrier to the other. A jump of eight weeks had dire consequences because it caused numbers that had yet to be transferred to be listed as having already been moved and numbers that had already been transferred to be reported as pending.

“With these updated routing tables, a lot of people were unable to make calls, as we didn't have a correct state!” the engineer, who asked to be identified only by his first name, Simen, wrote in an email. “We would route incoming and outgoing calls to the wrong operators! This meant, e.g., children could not reach their parents and vice versa.”

A show-stopping issue
Simen had experienced a similar error last August when a machine running Windows Server 2019 reset its clock to January 2023 and then changed it back a short time later. Troubleshooting the cause of that mysterious reset was hampered because the engineers didn’t discover it until after event logs had been purged. The newer jump of 55 days, on a machine running Windows Server 2016, prompted him to once again search for a cause, and this time, he found it.

The culprit was a little-known feature in Windows known as Secure Time Seeding. Microsoft introduced the time-keeping feature in 2016 as a way to ensure that system clocks were accurate. Windows systems with clocks set to the wrong time can cause disastrous errors when they can’t properly parse timestamps in digital certificates or they execute jobs too early, too late, or out of the prescribed order. Secure Time Seeding, Microsoft said, was a hedge against failures in the battery-powered onboard devices designed to keep accurate time even when the machine is powered down.

“You may ask—why doesn’t the device ask the nearest time server for the current time over the network?” Microsoft engineers wrote. “Since the device is not in a state to communicate securely over the network, it cannot obtain time securely over the network as well, unless you choose to ignore network security or at least punch some holes into it by making exceptions.”

To avoid making security exceptions, Secure Time Seeding sets the time based on data inside an SSL handshake the machine makes with remote servers. These handshakes occur whenever two devices connect using the Secure Sockets Layer protocol, the mechanism that provides encrypted HTTPS sessions (it is also known as Transport Layer Security). Because Secure Time Seeding (abbreviated as STS for the rest of this article) used SSL certificates Windows already stored locally, it could ensure that the machine was securely connected to the remote server. The mechanism, Microsoft engineers wrote, “helped us to break the cyclical dependency between client system time and security keys, including SSL certificates.”

Simen wasn’t the only person encountering wild and spontaneous fluctuations in Windows system clocks used in mission-critical environments. Sometime last year, a separate engineer named Ken began seeing similar time drifts. They were limited to two or three servers and occurred every few months. Sometimes, the clock times jumped by a matter of weeks. Other times, the times changed to as late as the year 2159.

“It has exponentially grown to be more and more servers that are affected by this,” Ken wrote in an email. “In total, we have around 20 servers (VMs) that have experienced this, out of 5,000. So it's not a huge amount, but it is considerable, especially considering the damage this does. It usually happens to database servers. When a database server jumps in time, it wreaks havoc, and the backup won’t run, either, as long as the server has such a huge offset in time. For our customers, this is crucial.”

Simen and Ken, who both asked to be identified only by their first names because they weren’t authorized by their employers to speak on the record, soon found that engineers and administrators had been reporting the same time resets since 2016.

In 2017, for instance, a Reddit user in a sysadmin forum reported that some Windows 10 machines the user administered for a university were reporting inaccurate times, in some cases by as many as 31 hours in the past. The Reddit user eventually discovered that the time changes were correlated to a Windows registry key in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\SecureTimeLimits. Additional investigation showed that the time changes were also linked to errors that reported valid SSL certificates used by the university website were invalid when some people tried to access it. The admin reached the following conclusion:

TLDR: Windows 10 has a feature called Secure Time which is on by default. It correlates time stamp metadata from SSL packets and matches them against time from the DCs. It processes these various times by means of black magic and sets the system clock accordingly. This feature has the potential to flip out and set the system time to a random time in the past. The flip out MIGHT be caused by issues with SSL traffic.

Other examples of people reporting the same behavior—for example, here and here—date back to 2016, shortly after the rollout of STS. More recent reports of harmful STS-induced time changes are here, here, and here.

“We've run into a show-stopping issue where time on a bunch of production systems jumped forward 17 hours,” one Reddit user wrote. “If you've been in the game more than a week, you know the havoc this can cause.”

STS primer
To determine the current time, STS pulls a set of metadata contained in the SSL handshake. Specifically, the data is:

ServerUnixTime, a date and time representation showing the number of seconds that have elapsed since 00:00:00 UTC on January 1, 1970
Cryptographically signed data obtained from the remote server’s SSL certificate showing whether it has been revoked under a mechanism knowns as the Online Certificate Status Protocol.
Microsoft engineers said they used the ServerUnixTime data “assuming it is somewhat accurate” but went on to acknowledge in the same sentence that it “can also be incorrect.” To prevent STS from resetting system clocks based on data provided by a single out-of-sync remote server, STS makes randomly interspersed SSL connections to multiple servers to arrive at a reliable range for the current time. The mechanism then merges the ServerUnixTime with the OCSP validity period to produce the smallest possible time range and assigns it a confidence score. When the score reaches a sufficiently high threshold, Windows classifies the data as an STSHC, short for Secure Time Seed of High Confidence. The STSHC is then used to monitor system clocks for “gross errors” and correct them.

Despite the checks and balances built into STS to ensure it provides accurate time estimates, the time jumps indicate the feature sometimes makes wild guesses that are off by days, weeks, months, or even years.

“At this point, we are not completely sure why secure time seeding is doing this,” Ken wrote in an email. “Being so seemingly random, it’s difficult to [understand]. Microsoft hasn't really been helpful in trying to track this, either. I've sent over logs and information, but they haven't really followed this up. They seem more interested in closing the case.”

The logs Ken sent looked like the ones shown in the two screenshots below. They captured the system events that occurred immediately before and after the STS changed the times. The selected line in the first image shows the bounds of what STS calculates as the correct time based on data from SSL handshakes and the heuristics used to corroborate it.
Click to expand...

Lots more at the link.

Log in or Sign up

It Appears That There’s a Mass Emborkening With Windows Server

Tuckerfan BMF

Log in or Sign up

It Appears That There’s a Mass Emborkening With Windows Server

Tuckerfan BMF

Useful Searches