NTP's Fate Hinges On 'Father Time'

Figure 1:

(Image: Geralt via Pixabay)

In April, one of the open source code movement's first and biggest success stories, the Network Time Protocol, will reach a decision point. At 30 years old, will NTP continue as the preeminent time synchronization system for Macs, Windows, and Linux computers and most servers on networks?

Or will this protocol go into a decline marked by drastically slowed development, fewer bug fixes, and greater security risks for the computers that use it? The question hinges to a surprising degree on the personal finances of a 59-year-old technologist in Talent, Ore., named Harlan Stenn.

The Network Time Protocol is important enough that the likes of Google and Apple speak up if they find a bug in the protocol that needs fixing, or a modification they think is needed. But NTP has worked so well for so long that few people think there's any problem.

Not all is well within the NTP open source project. The number of volunteer contributors -- those who submit code for periodic updates, examine bug reports, and write fixes -- has shrunk over its long lifespan, even as its importance has increased. Its ongoing development and maintenance now rest mostly on the shoulders of Stenn, and that's why NTP faces a turning point. Stenn, who also works sporadically on his own consulting business, has given himself a deadline: Garner more financial support by April, "or look for regular work.”

Figure 6:

(Image: Margaret Clark)

Stenn's shaky personal finances illustrate one very real risk to the future of the Internet. A number of widely used foundations of the Internet -- such as OpenSSL, the Domain Name System, and NTP -- are based on open source code. Open source means no one owns the software, anyone can use it, and it's maintained through a collaborative process of people submitting changes to a central governing group. Some open source projects, such as the Android mobile OS, have a rich uncle like Google that pays people who maintain the code as a side job. Or, the project is trendy enough that working on it helps to spur consulting work. But a project like NTP, which is buried deep in the infrastructure, doesn't have a clear-cut financial backer. That leaves support up to people like Stenn.

For the last three-and-a-half years, Stenn said he's worked 100-plus hours a week answering emails, accepting patches, rewriting patches to work across multiple operating systems, piecing together new releases, and administering the NTP mailing list. If NTP should get hacked or for some reason stop functioning, hundreds of thousands of systems would feel the consequences. "If that happened, all the critics would say, 'See, you can't trust open source code,'" said Stenn.

Sam Ramji, CEO of the Cloud Foundry Foundation, cited Stenn’s work in an address at the Open Compute Summit 2015 in San Jose Mar. 11. He dubbed him "Father Time," and said he was "scraping by" as he continued to work on NTP.

Stenn is hardly the only open source coder living in such straits. Ramji also mentioned Werner Koch in Germany, the author and maintainer of Gnu Privacy Guard, which is used in three popular email encryption programs. In a Feb. 5 article, Koch told ProPublica that he was "going broke" on $25,000 a year since 2001. Chet Ramey, part of the networking infrastructure team at Case Western Reserve, has been the primary maintainer of the Bash shell for Unix since 1990 with minimal support.

Ramji noted that OpenSSL developers had been receiving less than $2,000 a year in donations when the Heartbleed exploit of OpenSSL broke out last April. "Secure code is hard to write and maintain," Ramji noted. Users have to decide whether they want to leave these projects to survive as best they can.

Next Page: Watching the timekeeper

Figure 2:

(Image: Christong via Pixabay)

NTP: Google And Apple Are Watching

The original NTP code was pulled together by a small group of academic and commercial developers led by now-retired Professor David Mills at the University of Delaware, who Stenn called "a super-genius," and with whom he still consults on a regular basis. But NTP was conceived "in friendlier times," Stenn observed wryly.

The need to develop NTP further coincides with new devices and new applications coming all the time onto the network, while the stakes behind reliable releases have gone up.

The most recent release of NTP 4.2.8 was "hurried out the door," said Heiko Gerstung, a managing director of Meinberg, a German producer of time servers based on NTP. The company expressed concern to Stenn on that point. But Meinberg executives know the condition of the overall project. "Considering the fact that this guy manages the releases of NTP all by himself, he is doing a heck of a job," Gerstung said in an email interview with InformationWeek. Meinberg is one of the few direct financial backers of the NTP project.

As the NTP project lead, Stenn gets calls from the biggest Internet and industrial companies about problems or suggested additions for NTP. He's happy to help. Occasionally, he pitches them to sign up as supporters of the Network Time Foundation, a nonprofit corporation he set up to receive donations for NTP. According to Stenn, they seldom do. In fact, just six companies support the foundation, with VMware the only household name among them.

The importance of NTP to the daily functioning of businesses can't be overstated. The NTP time stamp is part of how equities firms show that trades took place when they say they did, an element that helps them stay in regulatory compliance. Air traffic control relies on NTP for synchronized clocks. Robotic manufacturing uses it to carry out closely timed operations requiring coordinated time. Google search operations rely on it, which is why the Google security team scrutinizes NTP for bugs.

Apple Macintosh computers and servers running OSX use NTP, and Stenn said Apple developers have called him for help on several NTP issues. In the last such incident, he said he delayed a patch to give Apple more time to prepare OS X for it. When they were ready, he applied the patch and asked "whether Apple could send a donation to the Network Time Foundation," Stenn recalled. "They said they would do their best to see that Apple throws some money our way." But it hasn't happened yet.

"Everybody loves us," Stenn said. "But people with money say, 'We don't give to open source projects.'"

Asked whether running through his personal savings to support NTP was a sustainable position, he acknowledged he gets credit for creating well-crafted NTP releases, "but I never said I was smart."

NTP is nevertheless the protocol that everyone depends on. Other candidates exist, and Stenn himself said there are good ideas included in the young Precision Time Protocol project. But nothing else is in the running to take over synchronizing time on the network.

Linux Foundation Executive Director Jim Zemlin raised the problem of NTP's continued development in a keynote address at the Linux Collaboration Summit in Santa Rosa, Calif., on Feb. 18. (That's what led InformationWeek to seek out Stenn; he didn't come to us with his story.) Zemlin said poorly supported open source projects pose a risk to all the systems that depend on them. The OpenSSL project, an encryption project widely used to secure websites, had been receiving less than $2,000 a year in donations until the Heartbleed exploit compromised OpenSSL code.

"There are certain projects that have not received support commensurate with their importance," Zemlin said. "Too many critical open source software projects are underfunded and under-resourced."

OpenSSL, the Domain Name System, NTP and a handful of other open source projects on which the Internet depends have a broad following, but few people understand that, for aging projects with little glam, financial backers and code contributors alike have moved on to more stimulating challenges. Companies including IBM, Salesforce.com, HP, Adobe, Amazon, Bloomberg, and Google do support the Linux Foundation's Core Infrastructure Initiative, started after Heartbleed in May 2014. CII has raised $6 million, which Zemlin said is "not nearly enough." Stenn gets $7,000 a month from the fund, or $84,000 a year, to cover all the expenses of maintaining NTP, renting the data center space, and running the infrastructure required for support.

Next Page: Why synchronizing time matters

Figure 3:

(Image: 237607 via Pixabay)

Why Synchronizing Time Matters

Every computer has a clock, but in general computers "are known to have bad clocks," said John Engates, CTO of Rackspace, in talking about NTP. Getting two computers to agree on the time can be difficult. The NTP protocol has, for 30 years, consulted the best clocks available and derived a consensus time, which it then imposes across a mapped hierarchy of servers and their client PCs.

Greenwich Mean Time is a known source of reliable time, as is the US Naval Observatory. Their time is based on the solar day -- the time it takes for the earth to complete one 24-hour, 360-degree rotation while in orbit around the sun. NTP consults UTC or Universal Coordinated Time, which is Greenwich Mean Time expressed in the military's 24:00:00 hours terms.

On a daily basis, NTP also consults atomic clocks, which tick off precise seconds based on radioactive Cesium-133 decomposition. A GPS receiver can be tied into an NTP server, and use the transmission of a GPS satellite to get the correct atomic time. A GPS satellite has three atomic clocks, so if one falls out of synch, the other two can overrule it and keep the system on track. For GPS time to be off by a billionth of a second means its answer to a location query will be off by a foot. So GPS relies on precisely counted time, not the solar day.

NTP's job is, in some ways, simple: Consult UTC and atomic clocks and come up with the correct time. But, as Engates said, "Time gets complicated fast."

The solar day varies slightly from year to year. The earth wobbles. Tidal friction slows the earth's rotation by a tiny fraction of a second each year. Geophysical events, like the huge earthquake in 2004 in the Indian Ocean, cost the earth's rotation another fraction of a second. GMT and UTC account for these changes with leap seconds; atomic clocks do not. Currently, there's a 35-second gap between the two.

NTP can referee those differences, allowing it to synchronize operations as computer systems grow larger and more distributed. The Internet wouldn't function as well without it. Network Time Protocol doesn't just determine the correct time, but implements synchronized time between two systems. NTP software on a client or remote server asks an NTP reference server for a time check. The NTP software on the requestor captures how long it took for the query to reach its destination, and adds that amount of time to the time stamp that comes back.

Despite variances in traversing a network due to congestion and other causes, this NTP process will usually leave two systems coordinated to within 10 milliseconds (10 thousandths of a second) of each other. If the two are on the same campus network, the adjustment is likely to be within one thousandth of a second or less.

That's not as precise as what can be achieved with Precision Time Protocol, an IEEE standard released in 2002. But NTP is already in place, with proven reliability, and it's easy to use. "Basic configurations [of an NTP server] involve no more than a few statements," wrote Peter Rybaczyk in his book, Expert Network Time Protocol. Even PTP starts with NTP, then tries to make it more precise.

NTP has another point in its favor: A strong record on security (so far). It's a protocol whose misuse could corrupt and cause the failure of manufacturing systems, chemical processing, financial markets, and satellite communications. Its reliability is tied to billions of dollars of transactions a day; the NTP time stamp is one of the few ways equities firms have of proving to regulators they were in compliance of making a trade when they said they did. So far, it has withstood the danger of being hacked.

The Heartbleed vulnerability in OpenSSL opened the open source community's eyes to the threat from benign neglect of these foundational elements of the computing world and the Internet. As with Secure Sockets Layer, the Linux Foundation views NTP as critical to the continued reliability of both Linux and the Internet.

Next Page: NTP on shaky financial ground

Figure 4:

(Image: Pixelman via Pixabay)

Shaky Finances

Stenn called the Linux Foundation's $7,000-a-month contribution to NTP "wonderful and awesome." But he said that he hasn't been told by the Linux Foundation yet whether the payments will continue after the end of April, their current end date. Even if they do, that amount doesn't come close to sustaining the effort needed for NTP, he said.

Asked to describe a proper NTP support organization, Stenn listed a project research scientist, project manager, several full-time developers, two technical writers, a system and network administrator, and two standards "wranglers" to represent NTP to the IETF, IEEE, and ITU. As he toted it up in his head, he came out at a minimum of $3 million a year.

If he gets more support, he'd prefer to obtain it from a broad base of NTP users. "I need everyone to help a little bit, not one or two bigs," Stenn said. Here's his reasoning: Suppose one big technical company comes in and doubles the financing behind his effort with $100,000 a year. When they call with a suggested change to NTP, what's he supposed to say?

For companies looking to make a big donation, therefore, the best approach might be to fund the Linux Foundation, which can support efforts such as NTP through the recommendations of its industry advisory board. That foundation includes security expert Bruce Schneier, and Columbia law professor Eben Moglen, chairman of the Software Freedom Law Center, among other industry experts.

With the Linux Foundation's $7,000 in monthly cash flow, Stenn finances his movement between his home lab, in Talent, Ore., and the NTP servers located in San Jose, Calif. In Oregon, Stenn lives with his wife and does most of his patch inspection, code writing, and release building three weeks a month. The fourth week, he stays in San Jose, close to two colocation data center providers that host NTP computers. He rents a room there to work on server and network administration, maintain the email list, and check on server backups.

Much of the travel, room, replacement hardware such as disk drives, or needed commercial software such as the Intuit QuickBooks for NTP and NTF accounting, must come out of the $7,000 monthly stipend or be charged to his consulting business.

Most of his 17 to 20 servers came out of a one-time, $10,000 grant in 2010 from the Internet Society, a policy and technology infrastructure advisory body for the Internet founded in 1991. Those servers are running at ISC.org in Redwood City, Calif., which hosts BIND and several other open source pieces of Internet infrastructure. For 15 years, it has provided space, electricity and some management "smart hands" to host NTP operations, without charging, said Stenn. "They would love for us to pay them," he said, and he once totaled the monthly bill at $1,400. But ISC.org also knows the NTP project can't pay and continues to host it, Stenn added.

Stenn also uses five to six servers at a Hurricane Electric colocation in Fremont, Calif., as a disaster recovery site. The cost of those servers is charged to his consulting business. According to Stenn, those charges against what little consulting he still does has made his business a barely break-even proposition for three of the last four years.

In addition to his consulting business, Stenn founded the non-profit Network Time Foundation in 2010 in hopes of having an umbrella organization that could support multiple network time projects and accept donations.

For most of that period, he said he has collected membership fees from only two companies, Meinberg and VMware, the marketshare leader in virtualization software. The latter also contributes code. More recently, four other firms signed up: Microsemi, ixSystems, Deer Run Associates, and Sol.net Network Services. According to Stenn, their fees support the foundation's part-time business development consultant, Sue Graves, and continued efforts to build membership.

VMware became a first-year contributor at $12,000 and has upped its donation since then. Accurate network time is crucial to VMware's products as it tries to coordinate virtual machine activity in data centers and to live-migrate running virtual machines between hosts. "NTP synchronizes the time of a physical or virtual host … in a unique and mathematically elegant way," said Mike Adams, director of vSphere product marketing.

NTF's nonprofit model is good, "but it needs more companies to make a contribution," said Heiko Gerstung from Meinberg. "The companies currently supporting NTP on behalf of the rest of the planet are not enough."

Next Page: The nightmare before Christmas

Figure 5:

(Image: Geralt via Pixabay)

The Release Before Christmas

Stenn told us his workload got a little heavier in October 2014, when Google security team member Chris Ries notified him that he had discovered a security risk in NTP. It was a buffer overflow in NTP autokey, the public key/private key authentication system used to verify downloaded code. Although no one was known to have used it yet, the vulnerability had the potential to let a hacker launch malicious code remotely through an NTP server.

Stenn said Google previously had made clear to him that it will publish vulnerabilities 90 days after notifying the party responsible for the code. Stenn felt the clock had started ticking, and he didn't ask for a waiver. He set to work, putting in 16 to 18 hours a day for 10 weeks to correct the defect and get a new release out before the 90 days were up. It would be upsetting to all NTP users to have a vulnerability aired with no fix in hand.

On Dec. 18, he posted news of the vulnerability on the support Web site, sent notices out on the NTP email list, and posted a fixed version of the code. For this effort, Stenn said he got a lot of feedback -- and not in a good way.

As best he can estimate, "I pissed off over a hundred thousand folks by announcing this fix" seven days before Christmas, he recalled. "Yow." People wanted more warning, and they accused him of favoritism and letting some people know about it sooner. It was tough, but also offered a deeper realization of the true position he was in.

One of Stenn's main pillars of support is the originator of NTP, Professor David Mills, "who knows more about NTP code than any other human being," said Stenn. In many cases, he checks with Mills before making changes to the code, in part because Mills has embedded comments in the code that should be checked with before the code is altered.

The core functionality of NTP is described as simple and straightforward. But Mills, in an interview with InformationWeek, said that other parts having to do with monitoring and control "are so complex that the whole thing falls apart if you change something."

Mills, 76, is long retired from teaching computer and electrical engineering at the University of Delaware, where he originated the first version of NTP. At this point, he is also blind and can't help Stenn review code. To Mills, NTP "was kind of a hobby" for many years, and Stenn got in early with good patches as he worked with NTP in his contract jobs, and did some of the thankless tasks like release manager. Asked if Stenn should get more support, Mills responded, "I didn't realize he was working on it full time."

"Dave never saw the need for the type of end-user support that we offer," said Stenn. "He has no patience to deal with people who need that sort of handholding."

Independent, outside contributors do still submit code to NTP, though they tend to focus on the single operating system version they like to work with. One expert, Poul-Henning Kamp, is working in Denmark "with great plans for a future implementation," said Stenn.

When it comes to fixing existing bugs and vulnerabilities, there's Stenn as the sole full-time code committer and a few volunteers he can coax into looking at specific problems.

Stenn clearly likes the work, though. He described himself as an introvert who loves resolving issues of time. At his home lab in Talent, he has four GPS receivers on the roof collecting the combined wisdom of 12 atomic clocks. When the question of taking vacations came up in our discussion, his wife Margaret, who's listening in in the background, issued a hearty laugh. Stenn said vacations are a trip to the movies a few times a year. "My wife thinks I'm insane," he said as an aside in a later email.

As Stenn looks to the future, he sees NTP undergoing further development, including possible coordination with PTP, so that NTP "could speak PTP" for those who need more precise time than NTP can deliver. Such a move will take lots of work, though, and Stenn says he'll need to cut back his hours drastically, and start consulting full time, unless the Linux Foundation and other donors support NTP's work.

"There is a need for support for the free public infrastructure," Stenn said. "But there's just no revenue stream around time right now. People scream if their clocks are off by a second. They say, "Yes, we need you, but we can't give you any money.'"

Figure 7:

(Image: Geralt via Pixabay)

Attend Interop Las Vegas, the leading independent technology conference and expo series designed to inspire, inform, and connect the world's IT community. In 2015, look for all new programs, networking opportunities, and classes that will help you set your organization’s IT action plan. It happens April 27 to May 1. Register with Discount Code MPOIWK for $200 off Total Access & Conference Passes.