As Lennart Poettering talked about systemd (pronounced system D), I was struck not so much by his reputation as the enfant terrible of the Linux Kernel development process and more by the fact that he's a very bright programmer, commanding attention in an auditorium full of bright programmers.
Poettering took the podium at CoreOS Fest in San Francisco this week to talk about one of his favorite subjects, systemd, which he co-authored, and how it will help run Linux containers. I'm not going to pretend that I absorbed every nuance. There was a lot of nspawn and description=foobar stuff in it, and how all the stdout+stderr goes into the System Log and will never be lost. I'm still working on some of those details.
But mainly systemd is the controversial replacement for an old, familiar Linux initialization program, init. Systemd is about starting Linux faster, launching services, and when necessary, killing services off. It does so many things that init hadn't done before that it brought down the wrath of established Linux contributors and community as being contrary to the first rule of Unix development: Basic functions should be done as simply as possible. Systemd is a lot more complicated than init and tries to do a lot more. You can see where Poettering, perhaps in his naïveté and enthusiasm, ran afoul of this open source community.
Except that he and co-developer Kay Sievers have won. As a senior software engineer at Red Hat, Poettering received some validation when Red Hat adopted systemd into Fedora in place of init, and then into Red Hat Enterprise Linux itself. The Debian Linux distribution adopted it as well, which meant that CEO Mark Shuttleworth at Canonical, against his better judgment, was forced to follow. Ubuntu is based on Debian. Shuttleworth already had commissioned his own init alternative in Ubuntu, Mir display server, and he had no enthusiasm for systemd, but Debian's move forced his hand.
Poettering got a hearing at CoreOS Fest this week because CoreOS, a version of Linux optimized for running containers, uses systemd. Red Hat's Atomic Host, another Linux version optimized for running Linux containers, also uses it. It turns out systemd is good for fast startup and for killing off of containers, an operational point heavily in its favor now that containers are on everyone's to-do list.
But getting this far hasn't been easy. The Linux development process is known for its brutally direct comments on the code that contributors submit. That's a good thing because so many people want to make additions to Linux, which is already on the cusp of 20 million lines of code. A lot of discrimination rightly should be applied to what code gets in.
Over the course of 20 years, some of the discriminators have gotten rather good at their job, and Poettering, a German citizen, at first was greeted with a withering storm of criticism. Systemd contained 69 packages or code modules, compared to the much more compact init. It relied on declarative language commands, an approved coding concept in many settings, but an affront to init's existing use of shell scripts.
Prior to systemd, Poettering was the developer of PulseAudio, a sound server, and Avahi, a zeroconf protocol implementation for network device discovery. He had been active in 36 other projects. He started systemd in 2010, and, as it was picked up by more Linux users, a group of Linux traditionalists started a BoycottSystemd website and refused to use any distribution that picked it up.
[Want to learn more about Red Hat's commitment to containers? See Red Hat Launches Atomic Host To Run Containers.]
A sample from the BoycottSystemd site:
Systemd flies in the face of Unix philosophy: "do one thing and do it well." … Its responsibilities grossly exceed that of an init systems, as it [Systemd] goes on to handle power management, device management, mount points, cron, disk encryption, socket API/inetd, syslog, network configuration, login/session. ... Keep it simple, stupid.
But Poettering was busy mounting his response. In a January 2013 post, "The Biggest Myths," Poettering responded to his critics. By going beyond what init could do, systemd initiated socket-activated containers, which allowed a user "to drastically increase the density of your cloud system" -- meaning run more containers per host, a widely sought attribute.
Systemd was not incompatible with shell scripts, "We just don't use them," he wrote. On the contrary it could work with 99% of them, if the user chose to employ them. He set out to refute "a few myths." When he was done, he had rebutted 30 specific criticisms of systemd in detail.
Last fall, in an Oct. 6 post to Google+, Poettering, 33 at the time, went further and said the Linux community had become too intolerant of newcomers with new ideas, a complaint that has been voiced before. He wrote:
The Open Source community is full of a**holes, and I probably more than most others am one of their favorite targets. I get hate mail for hacking on Open Source. People have started multiple "petitions" on petition websites, asking me to stop working. ... Recently, people start collecting Bitcoins to hire a hitman for me (this really happened!). Just the other day some idiot posted a "song" on YouTube, a creepy work filled with expletives about me and suggestions of violence. People post websites about boycotting my projects, containing pretty personal attacks.
Some of the criticism appears to be driven, not only because Poettering and Sievers weren’t doing the unglamorous follow-up work of fixing bugs, but also because they were weren't conforming to both spoken and unspoken rules about how the kernel could be modified.
Last year, the tension between systemd developers and the kernel process came to a head as a bug
in systemd disrupted kernel operations. Linux's kernel project leader Linus Torvalds singled out both Sievers and Poettering as violators of the process in a message reported on the online Linux news site, Phoronix:
Kay, I'm f*cking tired of the fact that you don't fix problems in the code *you* write, so that the kernel then has to work around the problems you cause… This has been going on for *years*, and doesn't seem to be getting any better… I'm not willing to merge something where the maintainer is known to not care about bugs and regressions and then forces people in other projects to fix their project. Because I am *not* willing to take patches from people who don't clean up after their problems, and don't admit that it's their problem to fix. Kay - one more time: you caused the problem, you need to fix it. None of this "I can do whatever I want, others have to clean up after me" crap. Linus
In a follow-up email, he appeared to single out Poettering's lack of teamwork. Systemd was providing service parsing, Torvalds acknowledged, but that "does become a problem when you have a system service developer who thinks the universe revolves around him, and nobody else matters, and people sending him bug-reports are annoyances that should be ignored rather than acknowledged and fixed."
Poettering and Sievers may have been wrong on a bug fix or multiple fixes, but clearly the upstarts had offended the gatekeepers on more than just the issue at hand.
Poettering in his Google post warned would-be Linux contributors: The open source community around Linux is viewed as a collegial body where contributors are valued by the technical quality of their code. "Well, it's not like that. It's quite a sick place to be in." After his post, Poettering was criticized for not having participated in the Linux Kernel Mailing List, and defended himself in that forum.
I don't think that Torvalds should be held responsible for the more extreme views expressed by the fringe of the Linux community, and surely some of the feedback Poettering collected was coming from the fringe. But when it comes to the value of the code, Poettering's ideas and approach have been more in step with the times than many other developers.
With systemd, Linux can launch and run containers as a service and run a set of containers on a host, maintaining log files that were sometimes lost under the init daemon. Not only Red Hat Fedora and Enterprise Linux, but OpenSUSE, CentOS, Debian, and Ubuntu have adopted systemd, even if some of their customers held their noses as they did so.
I think Poettering's real offense has been to advocate faster development of Linux at the expense of remaining POSIX compliant and to discard other tenets of the faith long adhered to. As he spoke at CoreOS Fest, however, he was in programmer mode, not protest mode. It was all just down-to-earth programmer talk, without a hint of bombast or hyperbole. Whatever rejection he may have felt from the kernel process must be somewhat allayed by the attention he commands in forums full of audience members in the thick of today's container and Linux action.
Systemd is good for running containers and mobile devices, among many other things. Poettering and Sievers have had the smarts to step outside the box and assemble a set of code modules that in all likelihood will last long into the future. And Poettering in his protest has perhaps gotten a concern for civility back into a community that might benefit from having a little more of it.
[Did you miss any of the InformationWeek Conference in Las Vegas last month? Don't worry: We have you covered. Check out what our speakers had to say and see tweets from the show. Let's keep the conversation going.]Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio