diff -u: What's New in Kernel Development


Kernel configuration has become more and more complex through the years with the proliferation of new drivers, new hardware and specific behaviors that might be needed for particular uses. It has reached about 3,000 config options, and that number will only increase.

Jean Delvare recently pointed out that a lot of those config options were relevant only to particular hardware, and yet the config system presented them to users who didn't have that hardware. This seemed like a bug to him, and he suggested that maintainers begin requiring proper hardware dependencies for all config options.

He acknowledged this would be a big task—especially for existing drivers. But creating the requirement for new drivers would at least get the ball rolling, and older drivers could follow along more gradually.

Josh Boyer agreed with all this. From his work on the Fedora project, he had to deal with the config system intensively, and he found it difficult. He said, "I've gotten to the point where I can somewhat guess based on the driver name which arch it's for (lately the majority are for ARM), but that isn't really a great way to handle things."

There was some resistance to the idea. Greg Kroah-Hartman, in particular, suggested that there were existing alternatives. For example, he said users simply could compile everything as modules. Then, they'd be loaded into the system only as needed.

But, neither Jean nor Josh liked that suggestion. Jean said that in the old days it was fine to build everything as a module, but nowadays there were just too many modules, and that "Saying 'm' to everything increases build times beyond reason. You also hit build failures you shouldn't have to care about, depmod takes forever, updates are slow as hell. This is the problem I am trying to solve."

Greg didn't see how build times could be a problem. Building the kernel on his laptop, he said, took about 20 minutes—with all 3,000 modules compiled in. If that wasn't good enough, he suggested upgrading the hardware to get a faster build time.

But, Jean said this wasn't a practical solution for some cases. He said, "We have 34 kernel flavors for openSUSE 13.1, for example. And, every commit potentially triggers a rebuild of the whole set, to catch regressions as fast as possible. So every module we build, for no good reason, gets built a hundred times each day." He added that it would cost a lot of money to upgrade the hardware underneath that build system.

Greg said he understood the issue, but that fixing the config system was just a hard problem to solve. It boiled down to enforcing better habits on everyone producing patches. He said, "If you see new drivers show up that you don't know where they work on, ask the developers and make up patches." He added, "Perhaps a few developers could be auditing the new Kconfig items of every kernel around -rc3 time frame to ensure that they don't do stuff like this."

Jean said that -rc3 would be too late, because "all kernel developers and distributions have already moved to the new kernel so they have already answered the n/m/y question for all new entries." He added, "It's the reviewer's job to refuse new drivers with bad Kconfig descriptions in the first place. This must happen as early as possible in the chain."

No clear decision came out of the discussion, but it does seem as though there's a vast mountain of configuration options that are becoming more and more difficult to deal with. Eventually, I think some form of clean hardware dependencies will end up being implemented, along the lines of Jean's suggestion.

With all the new devices coming out on the market, there's a big desire to get Linux running properly on all of them. Things like Intel's Quark system use only a few MB of RAM and have other tight hardware requirements. Shrinking Linux down to the right size poses a challenge.

Andi Kleen recently pointed out, "One problem on these small systems is the size of the network stack. Currently enabling IPv4 costs about 400k." He wanted to give users the option to prune down the Linux networking stack to only the bare essentials and get it down to 100K per application—competitive with Adam Dunkel's LwIP (Lightweight IP) project.

Andi posted some patches to create three available options for the networking stack: a full system with all current features, a partial system that supported regular users but not servers or a minimal networking system for the special userland on deeply embedded systems. The minimal system, he said, would "remove rtnetlink (ioctl only), remove ethtool, raw sockets".

This seemed extreme to Richard Weinberger, who said that on such a minimal system, even the ps command wouldn't work. Tom Zanussi on the other hand, said that the microYocto Linux distribution ran okay with Andi's patches and had decent workarounds to keep ps working properly. But, he added that microYocto was "very much a work-in-progress with a lot of rough edges, but it is a fully functional system on real hardware".

Alexei Starovoitov felt that trying to support such a minimal system would only create config options that led to a hacky, work-in-progress, rough-edged system. He said that if the goal was to create a half-functional system that was just very very small, the same thing could be accomplished with linker hacks. Simply "compile kernel as-is. Most functions have a stub for mcount() already. Use it to track whether the kernel function was called or not. Collect this data in userspace (as perf already does), add a few more functions that had a 'notrace' attribute on them, and feed this into a special linker that unpacks existing vmlinux, throws away cold functions, relocates the rest and here you have tiny vmlinux without recompilation."

But, Andi pointed out that for networking code, this wouldn't really work. He objected, "How would you know you exercised all the corner cases in the TCP stack? And you wouldn't want a remotely exploitable system because some important error handler is missing."

The discussion ended inconclusively. But, it does seem as though real patches, and not linker hacks, will be used to support all new hardware—even the tiny hardware that's been coming out lately.