Kernel Planet

At the osmocom project, we recently discovered the most interesting NVS NV08C-CSM module. It not only is a superb GPS receiver, but it includes GALILEO and GLONASS receivers, too. However, it's only available as an industry module, or as an expensive (700 EUR or so) evaluation kit.

Given the cheap PCB prototyping service at seeedstudio, I thought I'd spend an afternoon creating the schematics and PCB layout for an evaluation board. It exports the two 3.3V UARTs on OsmocomBB-style 2.5mm jacks, so they can be used with the T191 cables. I have the feeling this 2.5mm jack is becoming a new standard for low-voltage RS232 links ;)

Furthermore, it exports the SPI, I/O and I2C on a 20pin 2.54mm pitch header, connects to an external antenna via a MCX socket and has an optional footprint for a CR2032 battery on the bottom side.

So far, the board seems to be working fine. If there is interest in the bare PCB itself (without components!), please send me an e-mail. Depending on the amount of interest we might add it to the sysmocom webshop.

Schematics and Gerber files will be available at http://openbsc.osmocom.org/trac/wiki/osmo-nvs-gps soon.

January 25, 2012 01:00 AM

Some days ago, I noticed that the famous OP25 project (a Free Software implementation of the APCO25 system, a digital trunked radio system) was no longer reachable on-line. It seems they were running this on a desktop PC in a university. As nobody in the project still seems to be at that university, a change in the network configuration had accidentally rendered the website unreachable.

After some quick e-mails, I offered to host them within the osmocom.org family of Free Software Projects for mobile communications. This is when op25.osmocom.org was created, and a full-site backup uploaded + installed.

I'm really happy that we were able to do a small part to help to make sure this valuable project remains accessible to interested parties in the signal processing and mobile communications field.

January 25, 2012 01:00 AM

I gave a presentation on UEFI at LCA 2012 - you can watch it here, with the bonus repeat (and different jokes) here. It's a gentle introduction to UEFI, followed by some discussion of the problems we've faced in dealing with implementation bugs.

The fundamental problem is that UEFI is a lot of code. And I really do mean a lot of code. Ignoring drivers, the x86 Linux kernel is around 30MB of code. A comparable subset of the UEFI tree is around 35MB. UEFI is of a comparable degree of complexity to the Linux kernel. There's no reason to assume that the people who've actually written this code are significantly more or less competent than an average Linux developer, so all else being equal we'd probably expect somewhere around the same number of bugs per line. Of course, not all else is equal.

Even today, basically all hardware is shipping with BIOS by default. The only people to enable UEFI are enthusiasts. Various machines will pop up all kinds of dire warnings if you try to turn it on. UEFI has had very little real world testing. And it really does show. In the few months I've been working on UEFI I've discovered machines where SetVirtualAddressMap() calls code that has already been (per spec) discarded. I've seen cases where it was possible to create variables, but not to delete them. I've seen a machine that would irreparably corrupt its firmware when you tried to set a variable. I've tripped over code that fails to parse invalid boot variables, bricking the hardware. Many vendors independently fail to report the correct framebuffer stride. And those are just the ones that have ended up on hardware which crosses my desk, which means I haven't even tested the majority of consumer-grade hardware with UEFI.

The problems with UEFI have very little to do with its design or the ability of the people implementing it. After a few years of iterative improvements it stands a good chance of being more reliable and useful than BIOS. Increased commonality of code between vendors is a blessing and a curse - in the long term it means that these bugs can be shaken out, but in the short term it means that at least one hardware-destroying bug has ended up carried by multiple vendors. Right now we're still in the short term and it's likely that we'll find yet more UEFI bugs that cause all sorts of problems. The next few years will probably be a bumpy ride.

comments

January 23, 2012 03:52 PM

(Update January 18th 2012 - you probably want to read this for details on why the technical details described below are not the difficult bit of the problem)

An obvious question is why Linux doesn't support UEFI secure booting. Let's ignore the issues of key distribution and the GPL and all of those things, and instead just focus on what would be required. There's two components - the signed binary and the authenticated variables.

The UEFI 2.3.1 spec describes the modification to the binary format required to produce a signed binary. It's not especially difficult - you add an extra entry to the image directory, generate a hash of the entire binary other than the checksum, the certificate directory entry and the signatures themselves, encrypt that hash with your key and embed the encrypted hash in the binary. The problem has been that there was a disagreement between Microsoft and Intel over whether this signature was supposed to include the PKCS header or not, and until earlier this week the only widely available developer firmware (Intel's) was incompatible with the only widely available signed OS (Microsoft's). There's further hilarity in that the specification lists six supported hash algorithms, but the implementations will only accept two. So pretty normal, really. Developing towards a poorly defined target is a pain. Now that there's more clarity we'll probably have a signing tool before too long.

Authenticated variables are the other part of the puzzle. If a variable requires authentication, the operating system's attempt to write it will fail unless the new data is appropriately signed. The key databases (white and blacklists) are examples of authenticated variables. The signing actually takes place in userspace, and the handoff between the kernel and firmware is identical for both this case and the unauthenticated case. The only problem in Linux's support here is that our EFI variable support was written to a pre-1.0 version of the EFI specification which stated that variables had a maximum size of 1024 bytes, and this limitation ended up exposed to userspace. So all we really need to do there is add a new interface to let arbitrary sized variables be written.

Summary: We don't really support secure boot right now, but that's ok because you can't buy any hardware that supports it yet. Adding support is probably about a week's worth of effort at most.

comments

January 18, 2012 01:05 AM

I wrote about the technical details of supporting the UEFI secure boot specification with Linux. Despite me pretty clearly saying that this was ignoring issues of licensing and key distribution and the like, people are now using it to claim that Linux could support secure boot with minimal effort. In a sense, they're right. The technical implementation details are fairly straightforward. But they're not the difficult bit.

Secure boot requires that all code that can touch hardware be trusted

Right now, if you can run unstrusted code before the OS then you can subvert the OS. Secure boot gives you a mechanism for making sure you only run trusted code, which protects against that. So your UEFI drivers have to be signed, your bootloader has to be signed, and your bootloader must only load a signed kernel. If you've only booted trusted code then you know that your OS is safe. But, unlike trusted boot, secure boot provides no way for you to know that only trusted code was executed. That has to be ensured by OS policy.

This doesn't sound like too much of a problem. But it is. Imagine we have a signed Linux bootloader and a signed Linux kernel, and that these signatures are made with a globally trusted key. These will boot on any hardware using secure boot. Now imagine that an attacker writes a kernel module that sets up a fake UEFI environment, stops the kernel from running code and then executes the Windows bootloader - kind of like kexec, but a little more fiddly. The Windows bootloader believes that it's performing a secure boot, but in fact everything that it believes is trustworthy is the attacker's fake UEFI code. The user's encryption passphrase is logged, the Windows kernel is modified to hide the Linux code and despite everything looking fine your credit card details are on their way to China. In this scenario, the signed Linux kernel is simply used as a malware loader. The only sign that anything is wrong is that boot will be slightly slowed down.

Signing the kernel isn't enough. Signed Linux kernels must refuse to load any unsigned kernel modules. Virtualbox on Linux? Dead. Nvidia binary driver on Linux? Dead. All out of tree kernel modules? Utterly, utterly dead. Building an updated driver locally? Not going to happen. That's going to make some people fairly unhappy.

(The same applies to Windows, of course. Windows 7 allows you to disable driver integrity checks. Windows 8 will have to forbid that when the system's using secure boot)

Licensing

GPLv3 has various requirements for signing keys to be available. Microsoft's new requirement that systems support the installation of user keys would let users boot their own modified bootloaders, so that may end up being sufficient to satisfy the license. But we're then beholden on Microsoft - if they remove that requirement then users lose that freedom, and suddenly we're in an awkward licensing situation. There are ongoing conversations about exactly what we're able to do here, but it's not a solved problem.

Key distribution

The UEFI spec doesn't describe or mandate a central certifying authority. Microsoft require that everyone carry their key. We could generate our own, but we have much less sway with vendors. There's no way to guarantee that all hardware vendors will generate our key. And, obviously, if we generate a key, we can't just hand the private half out to others. That means that it becomes impossible for people to produce derivative versions of Linux distributions without getting their own key. The kind of identity verification that would be required for getting such a key is likely to be expensive, and also fairly likely to require that the distribution have a legally registered company in order to facilitate the identity verification. Think Extended Validation certificates, not Startssl Free. Hobbyist Linux distributions will be a thing of the past.

Doesn't custom mode fix this?

Microsoft's certification requirements now state that all systems must support a custom mode, implying that it will be possible for a user to install their own keys. Linux vendors would then be able to ship with their own keys on the install media and impose their own policies. Everyone's happy. It's not really good enough, though. People have spent incredible amounts of time and effort making it easy to install Linux by doing little more than putting a CD in a drive. Asking them to go into the firmware and reconfigure things adds an extra barrier that restricts the ability to install Linux to more technically skilled users. And it's even worse than that. This is the full description of the requirement for custom mode:

1. It shall be possible for a physically present user to use the Custom Mode firmware setup option to modify the contents of the Secure Boot signature databases and the PK.
   
2. If the user ends up deleting the PK then, upon exiting the Custom Mode firmware setup, the system will be operating in Setup Mode with Secure Boot turned off.
   
3. The firmware setup shall indicate if Secure Boot is turned on, and if it is operated in Standard or Custom Mode. The firmware setup must provide an option to return from Custom to Standard Mode which restores the factory defaults.

There's a few things missing from this, namely:

• Any description of the UI. It's effectively impossible to document Linux installation when the first step becomes (a) complicated and (b) vendor specific. Vendors are using the UEFI transition to differentiate themselves by coming up with their own unique firmware interfaces. Custom mode is going to look different everywhere.
• Any description of the key format. A raw binary representation of the key? An EFI_SIGNATURE_DATA struct? A base64 encoding of one, further protected with ROT13? We just don't know.
• Any way to use custom mode for unattended installs. It's a firmware interface that requires a physically present user. Want to install a few thousand machines over the network? This isn't a scalable approach
• …and this one's nitpicking, but there's not actually any requirement that the user be able to add keys - a vendor could conform to this language by only letting users delete keys. This is actually ok as long as the user deletes Pk, because then we'll effectively be back in setup mode and can install our own keys from the installer, but it still results in some practical problems

So no, custom mode doesn't make everything ok. Custom mode with a mandated UI and a documented key format would be much closer, but it wouldn't solve the problem of unattended automated installs.

Summary

We can write the code required to support secure boot on Linux in a minimal amount of time - in fact, most of it's now done. But significant practical problems remain, and so far we have no workable solutions for any of them.

comments

January 18, 2012 12:55 AM

This is a pre-announcement so people can start planning travel for the year.

The Linux Security Summit for 2012 will be held on the 30th and 31st of August in San Diego, CA, USA.  It will be co-located with LinuxCon North America, plumbers and the kernel summit.

More details to follow.

January 18, 2012 12:43 AM

This is what I got when tracking one of my inbound shipments:

It seems DHL is having fun bouncing the package back and forward between Hong Kong and Leipzig(Germany). So far, it started in HK, then arrived in Leipzig on January 8, went back to HK, back to Leipzig, back to HK, back to Leipzig and is currently allegedly again in Hong Kong _after_ succesfully passing German customs clearance on January 15.

For the TCP/IP nerds among the readers: I wonder when the TTL expires.

January 17, 2012 01:00 AM

I’ve set up a new git repository for the Linux kernel security subsystem on the new kernel.org server.

The URLs are:

git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security.git
http://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security.git
https://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security.git

Developers should work against the “next” branch.

A web-browsable interface via gitweb may be found at:

http://git.kernel.org/?p=linux/kernel/git/jmorris/linux-security.git;a=summary

The temporary repo on selinuxproject.org will go away soon, so please update your repositories.

January 16, 2012 05:02 AM

I shot videos of several of the presentations at the Fedora User and Developer Conference yesterday.  For your viewing pleasure:

• “State of Fedora” from the Fedora Project Leader, Jared Smith [ogg]
• Mike McGrath, team lead for OpenShift, demoing OpenShift [ogg]
• Jon Masters and Chris Tyler, on the ARM architecture in Fedora [ogg]. ARM is a secondary architecture today.  By Fedora 18, with your help, it needs to become a primary architecture.
• David Nalley presented on CloudStack, which is aiming for Fedora 17 inclusion. [ogg]
• Dan Prince and Russell Bryant giving an introduction to OpenStack [ogg]
• Mo Morsi presenting the Aeolus cloud management project [ogg]

[Update 1/18/2012] I was able to upload all the videos to YouTube.  http://www.youtube.com/playlist?list=PL2BAA7FF83E6482C2
is a playlist with all 6.

January 15, 2012 05:50 PM

I mentioned it briefly before: I've designed a small E1/T1/J1 transceiver board, which is going to be used for experimentally interfacing such a TDM line with microcontroller and/or FPGA. The name of this board is osmo-e1-xcvr.

The first prototype PCBs have arrived yesterday, and despite lots of other more important work I couldn't resist but to actually solder some of the units. The result can be seen here:

I don't have time to do anything beyond very basic testing right now, but so far the boards seem to be doing fine. Now we need a driver for the transceiver chip, and connect its control interface over SPI to some microcontroller (likely sam7s/sam3s/sam3u in my case). The actual serial bitstream will end up at the SSC peripheral of the controller.

January 14, 2012 01:00 AM

I thought it would be easier to do a round of stable kernel releases in the middle of the larger kernel merge window, to prevent the next round from being so big (given that there are a lot of patches usually applying during the -rc1 merge window cycle).

So, I've now done:

• Linux 2.6.32.54-rc1 release
• Linux 3.0.17-rc1 release
• Linux 3.1.9-rc1 release
• Linux 3.2.1-rc1 release

Please go test and let me know if there are any problems with any of these kernels. If I've missed any patches that you feel should be in them, also please let me know.

Note, this is most likely going to be the LAST 3.1.y kernel release, so please move off to the 3.2 kernel at this point in time. Maintaining so many different kernel branches all at once is not trivial, and I want to minimize it if at all possible.

January 10, 2012 10:54 PM

It's already two weeks since my last post mentioning OsmoSDR only very briefly. By now, OsmoSDR is fully public and you can read all about it on the http://sdr.osmocom.org/ website.

Specifically, the website includes Schematics, and one of my colorful block diagrams. I am a text guy and really hate to work with graphics, but the block diagram of the Calypso has helped a lot in the context of making people understand OsmocomBB, so I tried again for OsmoSDR:

So as you can see, it is a very simple, straight-forward design with a chip tuner, direct I/Q down-conversion, ADC for differential I and Q signals as well as a small FPGA for basic filtering and serial format conversion, followed by a Atmel SAM3U microcontroller (Cortex-M3, high speed USB).

And yes, it's receive only. However, it has a stacking connector for later addition of a transmit daughter-board which may eventually follow later in 2012.

So what is this OsmoSDR going to be used for? Well, for receiving any kind of signals between 64 MHz and 1700 MHz (possibly up to 1800 MHz, untested). We don't build it for a specific application, but we simply thought that for many applications a member of the USRP family is expensive overkill, and the FCDP has this arbitrary restriction at 96 kHz sampling frequency.

Please note that while I may be the only OsmoSDR developer that blogs about this project, it is very much a team effort and I'm only a minor part of that team. Apart from selecting the SAM3U, writing firmware and drivers for it as well as some discussions early on in the project, I didn't have any involvement in the Hardware design. Those credits go to Christian Daniel and Stefan Reimann.

So where do we stand? There are 5 prototypes of the first generation, they look like this:

There are some smaller hardware issues that were easy to work around, but one bigger problem related to the fact that the Si570 programmable oscillator output levels didn't match quite with what the FPGA clock input requires.

There will be a second generation board fixing this and other problems, and hopefully we'll see some progress in the weeks to come.

Firmware-wise, the code is currently scattered over a couple of different repositories, but I'm going to consolidate that soon. If you've worked with OpenPCD or SIMtrace, you will find some similarities: We split the flash of the controller into two partitions: One for the DFU bootloader, and one for the application code. You can then use the USB DFU (Device Firmware Upgrade) standard to quickly update the actual firmware of the device, without having to resort to jumpers or un-plugging and re-plugging of the hardware.

This also meant that I had to re-implement the old sam7dfu code for the SAM3U, which has considerable differences in the USB peripheral, cpu core, board startup code, etc. So it's really a reimplementation than a port. The good news is that I tried to make it as general as possible and integrate it with at91lib, Atmels reference code. So it should be easier to use it with other Atmel SoCs like the sam3s which I want to use e.g. in SIMtrace v2.

January 10, 2012 01:00 AM

As 3.2 is now out, here's a note as to the current status of the different stable/longterm kernel trees.

First off, please everyone remember to mark any patch that you want to have applied to the stable kernel trees with a simple:

Cc: stable <stable@vger.kernel.org>

marking in the Signed-off-by: area. Once the patch hits Linus's tree, I will automatically be notified of it and it will be applied if possible. If it does not applied, you will be notified of that.

Note that the address is stable@vger.kernel.org, not the older address that used to be used before October of 2011.

At this time, all stable and longterm kernel trees are being maintained in one big git tree, located at:

git.kernel.org:/pub/scm/linux/kernel/git/stable/linux-stable.git

There are different branches for every different major kernel version.

Here's the different active kernel versions that I am maintaining at the moment:

• 3.2.y - this will be maintained until 3.3 comes out
• 3.1.y - there will be only one, maybe two, more releases of this tree
• 3.0.y - this is the new "longterm" kernel release, it will be maintained for 2 years at the minimum by me.
• 2.6.32.y - this is the previous "longterm" kernel release. It is approaching it's end-of-life, and I think I only have another month or so doing releases of this. After I am finished with it, it might be picked up by someone else, but I'm not going to promise anything.

All other longterm kernels are being maintained in various forms (usually quite sporadically, if at all), by other people, and I can not speak for their lifetime at all, that is up to those individuals.

If anyone has any questions about any of this, please let me know.

January 10, 2012 12:43 AM

Unwittingly, I sat myself up for a yearly excersize, but let's look for a silver lining, shall we?

Blog	Posts	Delta
Ani-nouto	149	-220
Meenuvia	93	-28
LiveJournal	27	-5
Tracy Today	1	-3
Mixi	0
Duke City Fix	0
Advogato	0

Looks like I'm taking my life back from the Internet.

January 09, 2012 03:10 AM

Part of our work to make it possible to use UEFI Secure Boot on Linux has been to improve our EFI variable support code. Right now this has a hardcoded assumption that variables are 1024 bytes or smaller, which was true in pre-1.0 versions of the EFI spec. Modern implementations allow the maximum variable size to be determined by the hardware, and with implementations using large key sizes and hashes 1024 bytes isn't really going to cut it. My first attempt at this was a little ugly but also fell foul of the fact that sysfs only allows writes of up to the size of a page - so 4KB on most of the platforms we're caring about. So I've now reimplemented it as a filesystem[1], which is trickier but avoids this problem nicely.

Things were almost working fine - I could read variables of arbitrary sizes, and I could write to existing variables. I was just finishing hooking up new variable creation, but in the process accidentally set the contents of the Boot0002 variable to 0xffffffff 0xffffffff 0x00000000. Boot* variables provide the UEFI firmware with the different configured boot devices on the system - they can point either at a raw device or at a bootloader on a device, and they can do so using various different namespaces. They have a defined format, as documented in chapter 9 of the UEFI spec. At boot time the boot manager reads the variables and attempts to boot from them in a configured order as found in the BootOrder variable.

Now, obviously, 0xffffffff 0x00000000 is unlikely to conform to the specification. And when I rebooted the machine, it gave me a flashing cursor and did nothing. Fair enough - I should be able to choose another boot path from the boot manager. Except the boot manager behaves identically, and I get a flashing cursor and nothing else.

I reported this to the EDK2 development list, and Andrew Fish (who invented EFI back in the 90s) pointed me at the code that's probably responsible. It's in the BDS (Boot Device Selection) library that's part of the UEFI reference implementation from Intel, and you can find it here. The relevant function is BdsLibVariableToOption, which is as follows (with irrelevant bits elided):

BdsLibVariableToOption (
  IN OUT LIST_ENTRY                   *BdsCommonOptionList,
  IN  CHAR16                          *VariableName
  )
{
  UINT16                    FilePathSize;
  UINT8                     *Variable;
  UINT8                     *TempPtr;
  UINTN                     VariableSize;
  VOID                      *LoadOptions;
  UINT32                    LoadOptionsSize;
  CHAR16                    *Description;

  //
  // Read the variable. We will never free this data.
  //
  Variable = BdsLibGetVariableAndSize (
              VariableName,
              &gEfiGlobalVariableGuid,
              &VariableSize
              );
  if (Variable == NULL) {
    return NULL;
  }

So so far so good - we read the variable from flash and put it in Variable, Variable is now 0xffffffff 0xffffffff 0x00000000. If it hadn't existed we'd have skipped over and continued. VariableSize is 12.

  //
  // Get the option attribute
  //
  TempPtr   =  Variable;
  Attribute =  *(UINT32 *) Variable;
  TempPtr   += sizeof (UINT32);

Attribute is now 0xffffffff and TempPtr points to Variable + 4.

  //
  // Get the option's device path size
  //
  FilePathSize =  *(UINT16 *) TempPtr;
  TempPtr      += sizeof (UINT16);

FilePathSize is 0xffff, TempPtr points to Variable + 6.

  //
  // Get the option's description string size
  //
  TempPtr     += StrSize ((CHAR16 *) TempPtr);

TempPtr points to 0xffff 0x0000, so StrSize (which is basically strlen) will be 4. TempPtr now points to Variable + 10.

  //
  // Get the option's device path
  //
  DevicePath =  (EFI_DEVICE_PATH_PROTOCOL *) TempPtr;
  TempPtr    += FilePathSize;

TempPtr now points to Variable + 65545 (FilePathSize is 0xffff).

  LoadOptions     = TempPtr;
  LoadOptionsSize = (UINT32) (VariableSize - (UINTN) (TempPtr - Variable));

LoadOptionsSize is now 12 - (Variable + 65545 - Variable), or 12 - 65545, or -65533. But it's cast to an unsigned 32 bit integer, so it's actually 4294901763.

  Option->LoadOptions = AllocateZeroPool (LoadOptionsSize);
  ASSERT(Option->LoadOptions != NULL);

We attempt to allocate just under 4GB of RAM. This probably fails - if it does the boot manager exits. This probably means game over. But if it somehow succeeds:

CopyMem (Option->LoadOptions, LoadOptions, LoadOptionsSize);

we then proceed to read almost 4GB of content from uninitialised addresses, and since Variable was probably allocated below 4GB that almost certainly includes all of your PCI space (which is typically still below 4GB) and bits of your hardware probably generate very unhappy signals on the bus and you lose anyway.

So now I have a machine that won't boot, and the clear CMOS jumper doesn't clear the flash contents so I have no idea how to recover from it. And because this code is present in the Intel reference implementation, doing the same thing on most other UEFI machines would probably have the same outcome. Thankfully, it's not something people are likely to do by accident - using any of the standard interfaces will always generate a valid entry, so you could only trigger this when modifying variables by hand. But now I need to set up another test machine.

[1] All code in Linux will evolve until the point where it's implemented as a filesystem.

comments

January 06, 2012 08:17 PM

Perhaps even a flatworm can learn to turn left in T-maze when he's hit with an electric shock, but every time I install new Fedora, I try to leave SElinux enabled. In Fedora 15 it lasted for almost a week. This time, things went sour in 10 minutes.

Thought 1: The troubleshooter is quite nice, they really are getting civilized these days. Poor Polkit is doing isatty(), let's see if we can help.

Thought 2: Well, this is a bit ugly, but hey, it's copy-pasteable! Hurray for supporting common UNIX tools...

Thought 3: I am dumber than a flatworm. How could I believe that this time they get it right.

Actually, I lied. I disabled the sorry thing the moment I saw this:

The reason I continue to suffer through this excercise twice a year is that the security issue is important, and the shifing focus of attacks at non-daemon, non-system applications (read: browsers) is a matter of great concern for me. Every time Dan Walsh posts something nice to his LJ it looks like progress is being made, but then I install new Fedora and feel despair. It is as if some fundamental architecture is broken or something.

January 06, 2012 05:11 AM

Hey, kmod 3 is out. Really nice to finish this release. I was hoping to have it between the holidays, but there were some major bugs pending. It’s nice to see udev from git already using it instead of calling modprobe for each module. Kay reported a hundred less forks on bootup after start using [...]

January 06, 2012 04:50 AM

A bit over a year ago, I wrote about how an incredible number of Android tablets on the market were in violation of the terms of the GPL. I've had rather a lot else to do since then so it's now awfully out of date - but taking a quick browse through the current stack of cheaper devices indicates that things aren't all that much better. We've got source code for some chipsets that were missing it before, but to compensate we've got a whole bunch of new hardware that's entirely lacking. It's all pretty poor, really.

At the time, I wrote the following:

"(Side note: People sometimes ask why Google aren't doing more to prevent infringing devices. For the vast majority of these cases, Google's sole contribution has been to put Android source code on a public website. Red Hat own more of the infringing code than Google do. There's no real reason why Google should be the ones taking the lead role here, and there's fairly sound business reasons why it's not in their interest to do so)"

Factually speaking, nothing's changed. Each of these devices contains code owned by Google, and Google could absolutely take legal action against the vendors. Equally, so could Red Hat, Intel, Nokia and dozens of other companies who hold copyright on portions of the code carried on these devices, and so could thousands of individuals around the world. Nobody's obliged to enforce their copyrights, and in the absence of anyone else doing so it's unreasonable to insist that Google should do it.

However.

Google gives Android away. This seems like an odd thing for them to do, given that it's a significant engineering effort and costs a lot of money to produce. But remember what Android brings to Google - it's a platform with a well-integrated mechanism for distributing advertising to users. Scanning the market shows a huge number of ad-supported apps, and Google's getting money for every single one of those that gets shown. The more Android devices, the bigger the market for apps - and the wider their advertising reach.

In other words: unscrupulous hardware vendors save money by ignoring their GPL obligations. This lets them appeal on price, increasing the number of Android devices in use and increasing Google's profits. Google makes money off other people's violation of the GPL.

Could Google do anything to stop this? Yes. They could sue for copyright infringement, but that kind of thing's time consuming and awkward and any argument about the GPL always seems to end up as a big argument involving conspiracy theories. Instead, Google could attach some extra conditions to the Android trademark. Requiring that the trademark only be attached to GPL-compliant products ought to allow Google to take advantage of the existing well-tested mechanisms for seizing counterfeit goods, providing a direct economic incentive for companies to come into compliance. For added marks, they could restrict the adwords code to devices that use the trademark - if the vendor removes the trademark, applications depending on the adwords functionality would refuse to run and Google wouldn't make money off the infringing hardware.

Or, of course, they could just carry on making extra money as a result of vendors denying users the freedoms granted by the copyright holders. Although that sounds kind of evil to me.

Edit: You probably also want to read the followup post here.

comments

January 04, 2012 05:08 PM

My post yesterday on how Google gains financial benefit from vendor GPL violations contained an assertion that some people have questioned - namely, "unscrupulous hardware vendors save money by ignoring their GPL obligations". And, to be fair, as written it's true but not entirely convincing. So instead, let's consider "unscrupulous hardware vendors have economic incentives to ignore their GPL obligations".

The direct act of compliance costs money

Complying with the GPL means having the source code that built the binaries you ship. This is easy if your workflow involves putting source in at one end and getting binaries out at the other, but getting to that workflow means having a certain degree of engineering rigour. If your current build process involves mixing a bunch of known good binaries you got from somewhere but you can't remember where with a hacked up source tree that exists on someone's hard drive and then pushing all of these into a tool that only runs on Windows ME, before taking the resulting image and replacing chunks of it by hand, compliance is effectively impossible.

We all know that this is against all kinds of best practices and probably causes so many problems that it's more expensive in the long term, but retooling and hiring someone to oversee all of this takes time and money, and given the margins on many of these devices that's probably enough to make you uncompetitive for a couple of product cycles. Maybe you'll be in a better position afterwards, but you don't know that there'll be an afterwards.

Suppliers who don't provide you with the source code may be cheaper than those who do

You can't be in compliance if you don't have the source code in the first place. The same arguments that apply to the hardware vendors also apply to the people selling you your chips, so there's also an economic incentive for them to avoid complying. And there's an obvious incentive for you to choose the cheaper chipset, even if they don't comply.

Getting the source may cost money

Buying a chipset doesn't necessarily get you the software that makes it work - several silicon vendors will charge you for the SDK. But many of these devices are effectively reference platforms, so are basically identical from a hardware perspective. So if one of your competitors paid for the SDK, you can just dump the binaries off their machine, flash them onto your own boards and save yourself a decent amount of money. You obviously don't get the source, and nor do you have the standing to insist that the vendor whose binaries you misappropriated give you the source.

In the absence of enforcement, GPL compliance only works if it's the norm

Let's imagine two companies, A and B. Both build a tablet device, and buy the full SDK including source code. Both find a bunch of bugs in the vendor SDK and fix a different subset of them. They ship. A provides source code. B doesn't. B can now take A's bugfixes and incorporate them, resulting in a more compelling product without any significant extra cost. You now have two products that can sell for the same price, but B's is better. A would need to prove that B copied their bugfixes rather than simply fixing them themselves , which probably isn't going to happen.

In a larger market, if B is the only vendor who does this then their advantage isn't large - some of A's work is misappropriated by B, but A does benefit from the engineering work contributed by C, D, E, F and G. A combination of social pressure and legal threats may bring B into compliance. But if infringement is the norm, A has no incentive at all to release the source - by doing so they'll be helping not only B, but also C, D, E, F and G. Everyone undercuts A and they go out of business quite quickly.

Moral: In the absence of enforcement, if everyone else is infringing, a single company who complies is at a disadvantage.

If compliance cost nothing then everyone would do it

You can argue that cheap tablets from China are infringing simply because nobody knows better. But what's HTC's excuse? They've clearly decided that there's a benefit in holding back their source code releases[1], balancing this against the risk of being sued. They know full well what they're doing. If compliance was free they'd ship the source at the same time as they shipped the binaries. Other significant vendors are also fully aware of their obligations but choose to ignore them anyway.

Summary

There are economic incentives to infringe the GPL, and therefore (all else being equal) an infringing device can be sold for less money. All else being equal, a cheaper device will sell more units. More sales means more devices selling adverts for Google. Google makes more money because Android vendors infringe the GPL.

Edited to add:

But don't just take my word for it - Jean-Baptiste Queru says the same here (search for "scrubbing" - is there any way to link directly to a Google plus comment?)

[1] The usual argument is "We will release the source code within 120 days", implying that it's a process that takes time and we should just be patient. Every single time I've started making threatening noises, the source has appeared within a week.

comments

January 04, 2012 04:48 PM

A discussion a couple of days ago about DPI detection (which is best summarised by this and this and I am not having this discussion again) made me remember a chain of other awful things about consumer displays and EDID and there not being enough gin in the world, and reading various bits of the internet and wikipedia seemed to indicate that almost everybody who's written about this has issues with either (a) technology or (b) English, so I might as well write something.

The first problem is unique (I hope) to 720p LCD TVs. 720p is an HD broadcast standard that's defined as having a resolution of 1280x720. A 720p TV is able to display that image without any downscaling. So, naively, you'd expect them to have 1280x720 displays. Now obviously I wouldn't bother mentioning this unless there was some kind of hilarious insanity involved, so you'll be entirely unsurprised when I tell you that most actually have 1366x768 displays. So your 720p content has to be upscaled to fill the screen anyway, but given that you'd have to do the same for displaying 720p content on a 1920x1080 device this isn't the worst thing ever in the world. No, it's more subtle than that.

EDID is a standard for a blob of data that allows a display device to express its capabilities to a video source in order to ensure that an appropriate mode is negotiated. It allows resolutions to be expressed in a bunch of ways - you can set a bunch of bits to indicate which standard modes you support (1366x768 is not one of these standard modes), you can express the standard timing resolution (the horizontal resolution divided by 8, followed by an aspect ratio) and you can express a detailed timing block (a full description of a supported resolution).

1366/8 = 170.75. Hm.

Ok, so 1366x768 can't be expressed in the standard timing resolution block. The closest you can provide for the horizontal resolution is either 1360 or 1368. You also can't supply a vertical resolution - all you can do is say that it's a 16:9 mode. For 1360, that ends up being 765. For 1368, that ends up being 769.

It's ok, though, because you can just put this in the detailed timing block, except it turns out that basically no TVs do, probably because the people making them are the ones who've taken all the gin.

So what we end up with is a bunch of hardware that people assume is 1280x720, but is actually 1366x768, except they're telling your computer that they're either 1360x765 or 1368x769. And you're probably running an OS that's doing sub-pixel anti-aliasing, which requires that the hardware be able to address the pixels directly which is obviously difficult if you think the screen is one size and actually it's another. Thankfully Linux takes care of you here, and this code makes everything ok. Phew, eh?

But ha ha, no, it's worse than that. And the rest applies to 1080p ones as well.

Back in the old days when TV signals were analogue and got turned into a picture by a bunch of magnets waving a beam of electrons about all over the place, it was impossible to guarantee that all TV sets were adjusted correctly and so you couldn't assume that the edges of a picture would actually be visible to the viewer. In order to put text on screen without risking bits of it being lost, you had to steer clear of the edges. Over time this became roughly standardised and the areas of the signal that weren't expected to be displayed were called overscan. Now, of course, we're in a mostly digital world and such things can be ignored, except that when digital TVs first appeared they were mostly used to watch analogue signals so still needed to overscan because otherwise you'd have the titles floating weirdly in the middle of the screen rather than towards the edges, and so because it's never possible to kill technology that's escaped into the wild we're stuck with it.

tl;dr - Your 1920x1080 TV takes a 1920x1080 signal, chops the edges off it and then stretches the rest to fit the screen because of decisions made in the 1930s.

So you plug your computer into a TV and even though you know what the resolution really is you still don't get to address the individual pixels. Even worse, the edges of your screen are missing.

The best thing about overscan is that it's not rigorously standardised - different broadcast bodies have different recommendations, but you're then still at the mercy of what your TV vendor decided to implement. So what usually happens is that graphics vendors have some way in their drivers to compensate for overscan, which involves you manually setting the degree of overscan that your TV provides. This works very simply - you take your 1920x1080 framebuffer and draw different sized black borders until the edge of your desktop lines up with the edge of your TV. The best bit about this is that while you're still scanning out a 1920x1080 mode, your desktop has now shrunk to something more like 1728x972 and your TV is then scaling it back up to 1920x1080. Once again, you lose.

The HDMI spec actually defines an extension block for EDID that indicates whether the display will overscan or not, but doesn't provide any way to work out how much it'll overscan. We haven't seen many of those in the wild. It's also possible to send an HDMI information frame that indicates whether or not the video source is expecting to be overscanned or not, but (a) we don't do that and (b) it'll probably be ignored even if we did, because who ever tests this stuff. The HDMI spec also says that the default behaviour for 1920x1080 (but not 1366x768) should be to assume overscan. Charming.

The best thing about all of this is that the same TV will often have different behaviour depending on whether you connect via DVI or HDMI, but some TVs will still overscan DVI. Some TVs have options in the menu to disable overscan and others don't. Some monitors will overscan if you feed them an HD resolution over HDMI, so if you have HD content and don't want to lose the edges then your hardware needs to scale it down and let the display scale it back up again. It's all awful. I recommend you drink until everything's already blurry and then none of this will matter.

comments

January 03, 2012 05:46 PM

This is how the year begins for the Fedora kernel.

Open bugs
F15	F16	Rawhide
394	339	143	(876)

To be continued..

New year, new bugs. is a post from: codemonkey.org.uk

No related posts.

January 03, 2012 02:52 PM

Oh, man, this is just too cool.  And sadly I just upgraded to an iPhone 4, not 4s (I am both a sucker for a deal, and normally filled with buyer’s regret).  But anyway, this guy used the Radio Thermostat API (which I mentioned in my last post) together with a Siri hack to add voice control to his thermostat.  How awesome is that?  (The thermostat he used is the 3M-50 available at Home Depot, a rebranded version of the CT-30 available at Amazon [amzn]).

Looks like his code is here.

December 30, 2011 07:58 PM

Woo-hoo!

User-Level Implementations of Read-Copy Update has appeared in the February 2012 issue of IEEE Transactions on Parallel and Distributed Systems. Many thanks to everyone involved, especially co-authors Jon Walpole, Michel R. Dagenais, Alan Stern (who did the symbolic-logic heavy lifting), and Mathieu Desnoyers (who is the lead author). Mathieu also managed to convince me to go once more into the breach, which was not an easy task given that I received my license to collect RCU rejection notices all the way back in 1995. ;-)

So it does feel very good to see this finally hit print!

December 29, 2011 09:34 PM

My presentation at the Real Time Linux Workshop this past October, titled “On migrate_disable() and Lantencies” (presentation), was a bit of a departure for me. This presentation used a Markov model to analyze the behavior of some recent changes to the scheduler for the 3.0 series of -rt kernels.

Although this approach produced some interesting results, one difficulty is that a number of the corresponding scheduler measurement simply do not fit the exponential distribution very well. This question of course came up during my talk, where an audience member suggested instead using the Erlang distribution. Unfortunately, my only memory of Erlang distributions was of a 1980s operations-research class, where I learned how to use an Erlang distribution, but not why anyone would want to, at least not beyond the professor's vague assurances that it is helpful when modeling telephony networks.

So I answered that I might consider an Erlang distribution, but that I figured that I could match the data quite well by using cascaded Markov-model stages to represent a single state in the higher-level model. The big advantage of this cascading approach is that the math and software remains the same: You simply map additional states into the model.

However, my work reducing RCU's need for scheduler-clock ticks took precedence, so it was only recently that I got time to work out the math for cascaded Markov-model stages. I got the results I expected, but I figured that I should also do a quick review of the Erlang distribution. So I got out my old copy of “Introduction to Operations Research” by Hillier and Lieberman. Imagine my surprise to learn that the Erlang distribution is exactly cascaded Markov-model stages!

So my response to the question during my talk was essentially: “I might try the Erlang distribution, but I am going to try deriving it from scratch first.” Oh well, I would much rather look foolish with the correct answer than to look smart with the wrong answer!

On the other hand, this approach did give me a very good understanding of not only how to use the Erlang distribution, but also how to derive it and why you might want to use it. :-)

December 29, 2011 04:13 AM

I love it when device manufacturers realize that they can do well by selling a device with documented interfaces, in the hopes that a nebulous community of hackers will invent cool new things for it. So I was pretty happy when the Radio Thermostat Company of America announced that they had an API available for their wifi thermostats (such as the 3M-50 at Home Depot or the CT-30 [amzn] at Amazon). The API docs are currently available under a link on their “latest news” page.   The API describes a JSON interface to the thermostat, so you can retrieve or send information using http.  For example:

$ curl http://thermostat1/tstat
{"temp":67.50,"tmode":1,"fmode":0,"override":0,"hold":0,
"t_heat":62.00,"time""{"day":1,"hour":9,"minute":47},
"t_type_post":0}

We can get current temperature, set point, mode (heating, cooling, or off), humidity (on some models) etc. And it’s not just a read-only interface; you can set parameters like target temp, schedules, etc, as well as display some information on the screen (4 digits on the CT-30 [amzn] , alphanumeric on the CT-80 [amzn])

There’s a Windows tool out there called “Set Your Thermostat” which seems to have gotten the most press, and there’s a 3rd party iPhone app called Thermostapp which likely uses this API as well. So far all I’ve done with it at home is to query the devices every 5 minutes for setpoint and current temperature, so I can graph stuff out, like in the graph above.  At the end if this post is a basic & ugly perl script to gather that information from the thermostat; I process this a little with a bash script and use MRTG to graph it out.

I think there’s a lot more that could be done, such as:

• Display of household electricity data, or thermostat run time, on the display
• Use of indoor & outdoor temps to start heating/cooling in time to hit setpoint at desired time
• Better graphing than MRTG can provide, maybe with Cacti or PCP.
• Addition of X10 or similar occupancy sensors for automatic setback
• Automatic coordination of two zones to minimize boiler firing cycles

 

I’ll leave you with my simple perl script to gather some basic data from a thermostat:

#!/usr/bin/perl

use LWP::UserAgent;
use MIME::Lite;
use MIME::Base64;
use Getopt::Std;
use JSON;
use strict;

use vars qw/ %opt /;

my $debug = 0;
my $host = "thermostat2";
my $tempurl = "http://$host/tstat";

my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => $tempurl);
my $res = $ua->request($req);

if (! $res->is_success) {
    exit;
}

my $scalar = from_json($res->content);

# target isn't there if not in heat mode
my $t_heat = 0;
$t_heat = $scalar->{ 't_heat' } if $scalar->{ 't_heat' };

print "Temp is " . $scalar->{ 'temp' } . "\n";
print "Target temp is " . $t_heat . "\n";
print "Mode is " . $scalar->{ 'tstate' } . "\n";

 

December 27, 2011 04:31 PM

Noam Chomsky once wrote “Colorless green ideas sleep furiously” as an example of a sentence that is grammatically correct but nonsensical. However, my daughter Sarah and I were discussing this earlier today, and each of us managed to generate meanings for this phrase.

Sarah:

• Colorless: Uninspiring to the lumpen proletariat.
  
• Green: Pertaining to improving the environment.
  
• Sleep: Ignored by mainstream words and actions.
  
• Furiously: Angering the idea's proponents.
  

In other words, uninspiring ideas for improving the environment are ignored by the mainstream, to the fury of their proponents.

Paul:

• Colorless: Pertaining to transparency, as in window glass and also as in transparent transistors.
  
• Sleep: Residing in an idle state, as in idle CPUs.
  
• Furiously: Consuming excessive quantities of electrical power.
  

In other words, if you fail to specify CONFIG_NO_HZ=n when building the Linux kernel for a computer constructed of transparent transistors fabbed onto a pane of glass that controls that pane of glass, this colorless green idea will sleep furiously.

In both cases, the key element is the recently added meaning for the word “green”: As the word “green” has accreted a meaning, so has the sentence “Colorless green ideas sleep furiously.”

However, it turns out that Sarah's and my line of thought has been anticipated here, here, and here, and most probably elsewhere as well.

But that is OK. It turns out that Chomsky himself was also anticipated by Lucien Tesnière, the game cadavre exquis, and possibly also Bertrand Russell. Furthermore, a poet managed to accrete meaning onto Russell's “Quadruplicity drinks procrastination,” as can be seen here. The poet had originally intended to accrete meaning onto Chomsky's sentence, but, finding that others had beat him to it, turned to Russell's sentence instead.

The fact is that thinking up a new-to-you idea can be just as much fun as thinking up a new-to-the-human-race idea. Perhaps there really is nothing new under the sun. Even if there is something new under the sun, the human race is not the only thing under the sun! ;-)

December 26, 2011 10:22 PM

About a month ago I’ve been to the Automotive Linux Summit in Yokohama, Japan to present about the BlueZ stack and how it would fit in a CarKit system. You can find the slides here. The conference was very nice, about 200 attendees and many interesting talks.

Also on Dec, 1st I  attended the GENIVI Members Summit, where they created the Japanese working group of GENIVI.

Finally, I would like to thank Linux Foundation for sponsoring my travel to Yokohama.

December 26, 2011 02:51 PM

The Taiwanese smart phone maker HTC is widely known to be delaying its Linux kernel source code releases of their Android products. Initially, this has been described to to the requirement for source code review, and making sure that no proprietary portions are ending up in the release.

While the point is sort-of moot from the beginning (there should be no proprietary portions inside the Linux kernel for a product that wants to avoid entering any legal grey zone in the first place), I was willing to accept/tolerate it for some time.

At one point more than one year ago, gpl-violations.org actually had the opportunity to speak in person to senior HTC staff about this. I made it very clear that this delay is not acceptable, and that they should quickly fix their processes in order to make sure they reduce that delay, eventually down to zero.

Recently, I received news that the opposite is happening. HTC still has the same delays, and they are now actually claiming that even a 120 days delay is in compliance with the license.

I do think neither the paying HTC customers, nor tha Free Software community as a whole have to tolerate those delays. It is true that the GPLv2 doesn't list a deadline until when the source code has to be provided, but it is at the same also very clear what the license wants: To enable people to study the program source code. Especially in todays rapid smart phone product cycles, 120 days is a very long time.

So I hereby declare my patience has ended here. I am determined to bring those outrageous delays to an end. This will be one of my new year resolutions for 2012: Use whatever means possible to make HTC understand that this is not how you can treat Free Software, the community, its customers, the GPL and in the end, copyright itself.

December 24, 2011 01:00 AM

I'm currently quite excited to be doing more bare iron programming as well as actual electrical engineering work again.

There's actually not just one project I'm working on, but a variety of them:

• OsmoSDR - an upcoming small form-factor, inexpensive USB SDR. Not big and expensive like USRP or real "professional" solutions, but also not as weak as the ultra-narrow-band funcube dongle. I wasn't involved in the hardware design, but have volunteered to take care of the firmware development for the Atmel SAM3U micro-controller inside.
• osmo-e1-xcvr - a relatively simple circuit board containing the magnetics, LIU, clock generation and transceiver for E1/T1/J1 lines. The idea here is to have a solution for the analog part, as well as HDB3/B8ZS/AMI encoding, which can then be attached to any FPGA, CPLD or micro-controller development board. It exposes SPI for controlling the transceiver, and a synchronous serial bit-stream for Rx and Tx.
• unnamed sim-bank project - here the goal is to find a cheap solution to attach a large number of SIM cards to either a PC or directly to Ethernet. This can be very useful for testing, where you host your sim cards in a centralized location and can borrow them to remote users/devices over TCP/IP. There are commercial devices available for this, but they are quite expensive (like 1700 USD for 32 card device) and intended to be used with some proprietary windows software (who wants that?!?).

In the latter two projects I'm also doing the component selection, schematics design and PCB layout. One project with KiCAD, the other in EAGLE, as I really want to get first-hand experience of the usability of Free vs. proprietary EDA tools. I'd love to also evaluate Altium Designer, but they are still windows-only, and that would just make life way too difficult for me.

The projects will be duly announced soon, and they are all intended to be Open Hardware designs with Free Software. We'll probably also make all of them available at shop.sysmocom.de, too.

December 23, 2011 01:00 AM

No new videos yet, need to fix some more rendering bugs so it looks nicer :)

So I've been working towards 3 setups:

a) intel rendering + nouveau offload
b) nouveau rendering + DVI output + intel LVDS output
c) hotplug USB with either intel or nvidia rendering.

Categorisation of devices roles:
I've identified 4 devices roles so far:
preferred master: the device is happy to be master
reluctant master: the device can be a master but would rather not be
offload slave: device can be used as an additional DRI2 renderer for a master
output slave: device can be used an additional output for a master

For the 3 setups above:
a) intel would be preferred master, nvidia would be offload slave
b) nvidia would be preferred master, intel would be output slave
c) usb devices would be output slaves, however if no master exists, usb device would be reluctant master.

I've rebased the prime work[1] on top of the dma-buf upstream work, and worked through most of the lifetime problems. Some locking issues still exist, and I'll have to get back to them. But the code works and doesn't oops randomly which is good.

prime is the kernel magic needed for this work, as it allows sharing of a buffer between two drm drivers, so for (a) it shares the dri2 front pixmap between devices, for (b/c) it shares a pixmap that the rendering gpu copies dirty updates to and the output slaves use as their scanout pixmap.

So I've done nearly all the work to share between intel and nouveau and I've done the kernel driver work for udl, but I haven't done the last piece in userspace for (c), which is to use the shared pixmap as usb scanout via the modesetting driver.

Today I hacked in a switch on the first randr command, so I can start the X server with intel as master and nouveau in offload mode. I can run gears on intel or nouveau, then after the randr command and another randr command to set a mode, the X server migrates everything to the nouveau driver, puts it in master mode, and places the intel driver into output slave mode. It seems to render my xterm + metacity content fine.

So the current short-term TODO is:
fix some issues with my nouveau/exa port rendering
fix some issues with xcompmgr
add usb output slave support.

Medium-term TODO:
worked out how to control this stuff, via randr protocol. How much information do we need to expose to clients about GPUs, and how do we control them. Open issues with atomicity of updates to avoid major uglys. Switching from intel master to nvidia master + intel outputs, means we have to reconfigure the Intel output to point at the new pixmap, but the more steps we put in there for clients to do, the more ugly and flashing we'll see on screen, however we probably want a lot of this to be client driven (i.e. gnome-settings-daemon).

Longer term TODO:
Get GLX_ARB_robustness done, now that Ian has done the context creation stuff, this should be a lot more trivial. (so trivial someone else could do it :)

[1] http://cgit.freedesktop.org/~airlied/linux/log/?h=drm-prime-dmabuf

December 22, 2011 05:28 PM

In an earlier post I had tried out a few ways to figure out if our energy-saving efforts with respect to natural gas use were paying off; I did a few bar graphs of therms per day, per heating degree day, rolling yearly averages, etc.  I knew that I needed to normalize for the weather using Heating Degree Days, since natural gas is our primary heating fuel, and I probably needed to find a way to separate space heating from water heating, which have different conservation methods, and which may or may not be weather dependent.

I came across this article, Degree Days – Handle with Care! (run by the same folks who provide www.degreedays.net, a great source of free  HDD data) which listed a lot of the issues involved, some of which I had thought of, and some of which I had not, including:

• Separating out non-space-heating use (water, cooking, etc)
• Getting the right base temperature for the building in question
• Using proper degree day data for the actual monitoring period
• Differences in occupant behavior over the time period

But the other thing that caught my eye was the idea to do a scatter plot of HDD vs. heating therms used, and doing a linear regression; the slope should give you an idea of your average therms per heating degree day.

The graph above is my attempt at this; I found some old data from 2004-2005, which was before we did any insulation work, and I have more recent data for the past 4 or 5 years as well, so I have a decent data set.  The graph above does not attempt to factor out water heater use, or to perfectly match HDD (which I actually obtained from the Minnesota Climatology Working Group website, as monthly data) to the precise billing period (which can vary from 27 to 35 days).  It simply graphs therms vs. heating degree days, by heating season, for any months which had over 200 HDD.  But even with that basic, raw data, I still think it shows some interesting trends.

(edit: if you click above, the live graph now has all datapoints for all months; this more accurately shows the base gas load at the Y intercept.  It is also more accurately matched to the actual HDD for the billing period.)

If you click the graphic above, you will be taken to an interactive javascript plot of the data (made with my new favorite toy, jqplot).  By clicking on the legend you can turn data sets on or off; most interesting is to compare 2004-2005 (before any work) with 2010-2011 (after all insulation and air sealing, but before the extra heating zone was added in the basement).  All the years in between are pretty clumped together, with roughly the same slope (or, therm per hdd efficiency).  It’s a pretty messy data set so far; I’ll try to come up with a good way to factor out water heating etc, but I still found it to be an interesting way to visualize the data.  What do you think?

Edit: After a brief discussion with the nice folks at degreedays.net / energylens.com, I have a few more ideas on doing a better analysis.  Hopefully will have some spare time over the holidays to try out some other things, and do another post.

December 22, 2011 05:21 AM

I’m glad to announce the second version of kmod.  I’m sorry for not sending the first version to the mailing lists. Now I’m both writing it here and sending to the mailing list. I thank very much the feedback received for the first version and that now Jon Masters, the maintainer of module-init-tools, is helping us with kmod [...]

December 21, 2011 02:40 AM

So, in last post I said kmod 2 could be released sooner than expected if there were major bugs. Not as much as a surprise, there was 1: depending on the alias passed to the lookup function we were blocked iterating a list. It’s now fixed in git tree. Thanks to Ulisses Furquim for fixing [...]

December 16, 2011 06:59 AM

Was at a meeting today and at some point a whole side of the table was lined with half-closed laptops - except one that was closed fully. You could play "spot the GNOME 2.x user" (hello, Richard). That is because GNOME 3 on Fedora 16 has a reliable suspend, so everyone has it enabled by default, because that's what one normally wants. At a meeting, people used to close the tops to direct their attention to the presentation, or to indicate that they do. But that was then. Now, doing so triggers suspend, and that bumps you off VPN or weird WiFi providers like BitBuzz. Result is funny-looking and awkward. My neighbour eventually resorted to locking the screen (which blanks) instead of closing the top. Although blocking the distraction effectively, this has the disadvantage of not sending the right signal to the presenter, but tough cookies.

I think ideally I might like some kind of hotkey-suspend or whatnot, and disable suspend-on-close, but I dunno. Seems kind of bothersome to RTFM for GNOME. Worse, getting everyone in a meeting to agree on upon a non-default configuration seems like a non-starter.

December 16, 2011 05:48 AM

My work on RCU does require a pioneering spirit. For example, there are no classes to guide my efforts other than those I teach, and there are no publications to draw from other than those I write. One saving grace is that I work in the Linux community, which means that people can (and often do!) contribute their thoughts and ideas, many of which are now reflected throughout the Linux kernel's RCU implementation. (Thank you all! You know who you are, and there are too many of you to name you all. If you want the list, the git log command is your friend.) Nevertheless, being too afraid to stray from the beaten path implies being too afraid to work on RCU.

But there are times when the RCU implementation needs a more sane approach. During those times, I must find some other outlet for my insanity: To do otherwise is to break RCU. Fortunately, this time around, an appropriate outlet was readily available in the guise of Ubuntu's new Unity window manager.

I decided to emulate the environment of a typical Linux hacker. This meant installing, configuring, and using Unity without the advice and counsel of my acquaintances within Ubuntu. To those acquaintances who might feel some irritation at the contents of the remainder of this blog entry, I do apologize in advance. However, experiments are invalid unless the environment is properly controlled, and part of the control for this experiment was to isolate myself from such help. I did search the web, including of course ubuntuforums.org.

So here are the issues I encountered, more or less in the order I encountered them:

1. Right-clicking doesn't give you a way to configure the desktop, aside from setting the background image (pure black for me!). I found the answer in ubuntuforums.org: install and run “ccsm” to make major changes desktop configuration.
   
2. I work with large numbers of xterms, so that each desktop has nine 24x80 xterms in a three-by-three pattern. (And yes, when screens were smaller, I used a four-by-four pattern.) I use a script creatively named “xterms” to create them, and I expect them to be automatically placed in a non-overlapping three-by-three pattern, which they did under Gnome. But under Unity, many of them will be placed directly on top of each other. The solution is to add a “sleep 0.250“ in my script's loop. So Unity appears to be trying to do the right thing, but is a bit slow on the uptake. I learned about this experimentally, mostly because when you query for “tiled” you get advice on how to make your windows never overlap. In contrast, I want my windows to be non-overlapping by default, but if I am (say) debugging rcu_start_gp(), I want to be able to expand the window from 24x80 vertically to use the full screen size, and in that case, I am happy with the resulting overlapping windows.
   
3. Unity uses “workspaces” rather than “desktops”. Unfortunately, there doesn't seem to be to identify the windows of a given type that have been minimized from a given workspace. So if I have nine xterms in one workspace for my RCU work and another nine in another workspace for working on a paper, clicking the xterm button gets me all 18, shuffled around so that I cannot easily tell which goes with which workspace. Perhaps shift-click should show only those xterms associated with the current workspace!
   
4. I tried to work around the above problem by creating multiple desktops via “ccsm”. However, although the additional desktops exist, Unity cannot show you any windows assigned to them. The only way to see them again is to reduce the number of desktops back to 1, which will force them back onto the sole remaining desktop. (This might actually be a useful feature, but...) Worse yet, when you have more than one desktop, you lose the ability to move a given window to some other workspace by right-clicking on it: Instead, you can only move it to other desktops. Longer term, Unity should more gracefully handle multiple desktops. Until then, “ccsm” should not offer to create more than one desktop. And until then, just say “no” to the temptation to create multiple desktops. Use workspaces instead.
   
5. My habit of clicking on icons at the lower right corner of my screen to move among desktops died hard, but the window-s hotkey does turn out to be very nice. When you hit window-s, you get a screen that shows you all your workspaces, and you can use the arrow keys to move among them. When you get to your destination workspace, just hit the enter key.
   
6. Unfortunately, focus does not always follow you from one workspace to another. A quick window-s;left-arrow;enter;"cd /foo";enter sequence is quite likely to cause the xterm on the previous workspace to change to directory /foo, which can be a bit disconcerting. This really badly needs fixing, as the mental effort to verify that focus has indeed followed me sometimes causes me to forget why I wanted to be in the new workspace in the first place. This can be frustrating. (And yes, I am not as young as I used to be. Then again, neither are you!)
   
7. In Gnome, deleting a window (for example, typing exit to a bash prompt) automatically transfers focus to some other window on that same desktop. In contrast, in Unity, deleting a window sometimes transfers focus. Always would be far preferable!
   
8. I searched for synaptic in vain, finally learning about the new Ubuntu Software Center icon, which is the shopping-bag icon on the left-hand toolbar. Ubuntu Software Center seems OK, though I am not sure whether or not I would be happy to do without apt-get on the command line. Fortunately, I have both.
   
9. The left-hand toolbar did grow on me due to the fact that it seems to track the most heavily used applications. Unfortunately, there is no way to use this toolbar to query for the existence of an application: if there is an instance, it moves to the workspace containing the instance and transfers focus to it, but if there is no instance it creates one. (If there are multiple instances, it displays them all and lets you choose — but without letting you know which instance goes with which workspace.) Again, perhaps a job for shift-click, though there needs to be a way to: (1) query the current workspace and (2) query all workspaces. And a way to see which instances are associated with which workspaces. And a way to see which instances have been minimized.
   
10. A natural reaction to this sort of behavior is to fire up “ccsm” and experiment with different settings. Bad move! Doing this has a high probability of fatally confusing Unity. “You too can power-cycle your laptop.”
    
11. Unity sometimes loses track of the keyboard. Moving back and forth among workspaces helps it find the keyboard again. Unless the screen is locked, in which case life appears to be quite hard, with power-cycling the only option I have found thus far. Fortunately, this seems to be quite rare, only two sightings in the several weeks that I have been using Unity. Oops, make that three sightings. Hmmm... Four sightings. OK, this problem appears to be triggered by switching to a workspace then immediately hitting shift-F1.
    
12. Under Gnome, resizing an xterm displays a handy popup showing how many rows and columns of text the xterm contains as it is being resized. Unity badly needs this feature.
    
13. Where did the application and system menus go??? Well, they are gone. Once I got used to it, the replacement works much better for me. Shift-F1 pops up a window that allows you to search for apps, so that shift-F1;chr;enter pops up an instance of the Chromium browser. Shift-F2 works as it does in Gnome, except that it displays the possible completions as icons. Unfortunately, in both cases, Unity doesn't always keep up with touch-typists. Sometimes it re-executes the previous command instead of the one you just typed, even though the display has already updated to show the new icon. This indicates some performance, coordination, and concurrency issues: Unity's right hand does not always know what my left hand is doing! It is therefore not obvious to me that the Unity development and test teams include any touch typists. One way or another, this sort of thing badly needs fixing.
    
14. Banshee is quite similar to Rhythmbox. One nice thing about Banshee is that it does not come with a pile of Internet radio stations preconfigured, so that you can easily find the ones you add. (My tastes in music are such that my favorites are never going to appear in any reasonable set of defaults.)
    
15. If you push a window beyond the bounds of the sides or bottom of the screen, it ends up spanning multiple workspaces, where the workspaces are connected in a toroidal fashion. This is sort of OK, except that windows that span multiple workspaces (up to four of them!) are not handled gracefully by the left-hand taskbar. Although this behavior was mildly entertaining for a while, I would prefer that the workspaces not be connected.
    
16. If you push a window up to the top of the screen, it maximizes. This is almost never what I want — the reason I pushed the window to the top of the screen was to have it appear at the top of the screen, not to have it monopolize the entire screen!!! On the rare occasions when I need to maximize a window, double-clicking the title bar works just fine, thank you!!!
    
17. Where did the per-window menus go? Well, they are mostly gone. You can get them back by right-clicking the title bar of a given menu, but I am growing to like the alternative, which is to move the mouse to the very top of the screen, causing the per-window menus to appear on the screen's upper bar. This leaves more screen real estate for the rest of the application. However, some applications keep the window-bar menus, and I have no idea why.
    
18. So I have a browser in one workspace, and I want one in another workspace. Clicking on the icon on the left-hand bar pops me back to the original workspace: Not what I wanted! Right-clicking on the icon pops up a menu that allows me to create a new browser instance in the current workspace. However, this works only for some applications.
    
19. The soffice command does not automatically background itself, in unhappy contrast to Maverick's ooffice command. OK, OK, I can easily type &, but it is still annoying.
    
20. If you start soffice from an xterm, it splats on that xterm every time you do “save as”. Yes, I know that you failed to open the file! I wouldn't have done “save as” if the file already existed!
    
21. The soffice application occasionally goes into contrarian mode when you resize it. For example, an attempted horizontal expansion of the window might instead be applied vertically. Sometimes soffice simply refuses to let you resize it, which can be especially frustrating if it has decided that it should be so small that there is no room to actually display the document in question. Repeatedly maximizing and unmaximizing the window seems to get it out of this passive-aggressive mode of operation.
    

So Unity does appear to have some potential, and I intend to keep using it for at least a little while longer. However, it does still need a fair amount of additional work.

December 15, 2011 09:54 PM

For some weeks now I and Gustavo Barbieri at ProFUSION have been working on a new library and a set of tools, libkmod and kmod respectively. This is the announcement of its first public release. Overview The goal of the new library libkmod is to offer to other programs the needed flexibility and fine grained [...]

December 15, 2011 02:05 AM

No words to describe free fall. If you never did, you should be rethinking your next vacations

December 12, 2011 07:56 PM

In my last two posts I talked a little about our utility usage for electricity and for water; last up is natural gas usage.  (I don’t think I’ll ever start weighing my garbage, but who knows).

Oddly, gas was a little tougher to get a handle on – there is a lot more variability month-to month, as well as year-to-year depending on the weather.  Our therms used per billing period has ranged from around 10 to almost 200 in the last 3 years.  I’ve also done a lot of work that should help things (note, some amazon affiliate links follow):

• Sealed the attic and added insulation
• Retrofitted blown-in cellulose insulation in exterior walls
• Sealed as many air leaks as I could find
• Added programmable [amzn] (and wifi [amzn]) thermostats to set back home temp at night (I bought a 3M-branded version of that wifi thermostat at Home Depot)
• Insulated hot water pipes as far as I could
• Added low-flow shower heads [amzn] & sink aerators [amzn]

And on the increased-use side, we turned the basement into a zoned, heated space last year.

So it’s a ton of variables all working together.   How does it look, are there any trends?  My best attempt to analyzing that question is to look at therms used per heating degree day during the heating months.

A heating degree day is”a measurement designed to reflect the demand for energy needed to heat a home or business. It is derived from measurements of outside air temperature” according to Wikipedia.  I gathered degree days per month from www.degreedays.net (which didn’t perfectly overlap with billing periods, but close enough), subtracted about ten therms from each period (a rough approximation of our water heating use, probably a little low), and graphed the remaining therms per degree day:

We were on a fairly steady decline until about March/April 2011… which is when we put the 2nd heating zone in the basement.  Oops!  This is what actually prompted these posts; I hadn’t looked for a few months, and when I did, I realized that something had apparently gone a bit wrong in our gas conservation efforts.  I’ve since decided to let the basement stay cold unless we are down there; the baseboard radiators heat up in just a couple minutes, there’s no need to keep it warm down there all day long.

Here is the graph of therms per HDD (without normalizing for water heating use) and therms per day as a 12 month rolling average.  Around March 2011 looks again like a reversal of the downward trend:I’m not sure if this is really enough data to see a trend – and I’m not quite sure how to properly normalize for weather & separate out hot water from space heat use.  This is the best effort so far; I’m open to suggestions!

See also:

• Part 1 – Electricity
• Part 2 – Water

December 06, 2011 04:08 PM

In my last post, I talked about how I keep track of our electricity usage, to be sure that our efforts at efficiency & conservation are staying on track.  But that’s just one of 3 utilities; water has been another focus in our house.  Although we live right next to the Mississippi, and it’s unlikely to go dry anytime soon (I hope!) there’s still decent reason to conserve; if nothing else, it takes a fair bit of energy to treat that water before it gets to the house, and the sewage after it leaves.  (According to Energy Star, we spend $4 billion and use 50 billion kWh annually to treat water.)  And then there’s the gas for the hot water…

Anyway, water really wasn’t that hard to reduce.  We did these things:
(note, some Amazon affiliate links follow; these are the items I used)

• Got a high efficiency (HE) clothes washer
• Put low-flow (1.5 gpm) aerators on several faucets
• Put dual flush adapters in both toilets [amzn]
• Got low-flow showerheads for the upstairs and the basement showers [amzn]
• Use a rain barrel, and don’t water the lawn too much

None of this is too tough, or especially impactful on our lives.  The showers are quite fine; they aren’t human car washes, but c’est la vie.  And the grass isn’t always a lush green, but that’s just natural consequences in a drought, right?  The dual flush adapters work surprisingly well, and the clothes washer is a great machine.  The dishwasher is about all that’s left; we have to use a roll-around one so our choices are limited.

Anyway, it’s paid off pretty well; here are my gallons per day over time (measured quarterly via our utility bill) with a yearly rolling average:

We’re down to about half our peak use.  And here is a graph comparing use in each quarter (you can see that some summers we did use quite a lot more):

It still amazes me that we use 100 gallons per day; that does seem like a lot, although maybe not so bad for 4 people.  But as the kids grow up and start to want long and/or daily showers, we’ll see how things go!

Part 3 will cover my gas usage, and there’s an admission to make there …

See also:

• Part 1 – Electricity
• Part 3 – Gas

December 03, 2011 10:34 PM

I’ve just installed a plugin in my blog to display the common social sites below each post. From now on, if you like a post you can share it with the world using your preferred social media. I’m using the Socialize plugin in WordPress. I’m not sure if it’s the best one: I just installed [...]

December 02, 2011 01:12 PM

Eternal Vigilance is the Price of Efficiency — power is ever stealing [into your home]. — Me, with apologies to Wendell Phillips, Thomas Jefferson, or whoever said it first.

Those who know me know that I’m a little obsessed with energy, specifically renewables, and conservation/efficiency (it all goes hand in hand; with more conservation & efficiency, renewables can be that much closer to powering our world).

To that end, I’ve done a lot of efficiency work on our house, and have tried to make some behavioral changes as well, to conserve. Some of this work has been chronicled in an answer at Quora.

But how do I know if it’s really paying off? Not from a financial payback sense, but from a “am I actually using less energy than before” sense?  We measure, of course… and we graph.

Let’s start with Electricity. It’s easy to measure from monthly bills, and since by and large we don’t use AC, our electricity use isn’t terribly seasonal (modulo more lighting in the winter, and a couple hot weeks in the summer). I’ve found two types of graphs to be informative. The first and most obvious is to graph usage over time. Because the number of days in a given utility bill can vary, I plot the usage from each month’s bill, normalized to kWh per day for that period:

The pure month-to-month is pretty noisy; a rolling 6 month average gets things reasonably smoothed out, and there I can see a trend that I like. It’s plateaued a bit, though; we seem to have reached an ongoing average of around 12 kWh/day, or around 365kWh/month. While that is less than half of the Minnesota state average, I do know of a family of four who can come in under 200kWh/month – so there is room for improvement (somewhere… finally getting this server down to 18W will help to the tune of about 10kWh/month).

(Note, above is actual household usage, independent of any solar production, which is another variable.  I have, however, plotted the ongoing percentage of our use covered by solar on the green line.)

But there is indeed some seasonal variability; for that, it’s neat to plot each month against the same month from previous years, a trick I learned from the soon-to-be-defunct Microsoft Hohm:

Apparently, around September 2009 is when I started paying attention. A few months after that, it got even better. Early on, it was obvious things like getting rid of the insane power-hogging Pentium 4 Mythbox that ran 24/7, and not leaving the electric heater running in the basement.  Many other improvements followed – incremental lighting efficiency improvements, diligent use of the clothesline, making sure all computers slept/suspended appropriately, etc.  What amazes me is that we’ve cut it almost in half since those early days.  Maybe I was just an unabashed power-pig before?

It looks like we’re hanging in there at 12kWh/day – I’m sticking to my diet. Yay!  I’ll talk about graphs for water and gas in subsequent posts.  I’ve slipped a bit on one of them.  Stay tuned…

See also:

• Part 2 – Water
• Part 3 – Gas

December 02, 2011 04:30 AM

The first incarnation of the KOSS Legal Conference was a big success. There were many participants from a variety of backgrounds, such as

• Independent Korean legal experts
• Legal scholars from Korean law schools
• International legal experts (e.g. Till Jaeger, Carlo Piana, etc.)
• Representatives from the major Korean IT industry
• Representatives of the community organizations like FSFE
• Independent technical experts like Armijn Hemel and myself

The discussions have been a big success, with significant participation from the floor. There are many events that I attended where it was hard to actually get any participation from the audience - but the KOSS Law conference was definitely not one of them. Some of the questions were easy to respond to, some other questions really tackled the difficult issues in Free Software License Compliance.

What was clear to see from the Industry participants: FOSS License Compliance has become an important topic in the last couple of years: One the one hand as a result of virtually no TV set / mobile phone / PMP or other device running without Linux or other FOSS. On the other hand, I'm sure that the enforcement efforts of gpl-violations.org and the SFLC also have had significant impact on that.

What I personally find important is that compliance is only considered as part of the overall FOSS picture. Complying with the license text is the minimum that companies involved with FOSS should do. Rather, they should look beyond mere compliance and consider the benefit of engaging more actively with the community, contribute code back upstream/mainline and really becoming a first-class citizen of the Free Software world.

As a big surprise to everyone, Jim Zemlin of the Linux Foundation made a surprise visit towards the end of the second day of the conference.

Many thanks to the KOSS Law center for bringing this together and organizing such an event. Thanks also to the Korean NIPA (National IT Industry Promotion Agency) and the FSFE for their support of the event.

November 28, 2011 01:00 AM

Hoje dei uma palestra na AndroidConf sobre “Modificando a API do Android”. Referências ao projeto que falei sobre AVRCP podem ser encontradas em um post meu anterior. Estou disponibilizando abaixo os slides. EDIÇÃO 02/12/2011: coloquei uma nota no slide 7, relatando o que falei durante a apresentação sobre uso de IDE. Para aqueles que não [...]

November 26, 2011 07:17 PM

I've released man-pages-3.35. For the moment, the various pieces of the release remain at different locations than usual. The browsable online pages can be found on man7.org; the release tarball is available here.

A git repository for man-pages can be found on github. You can clone the repository with the command:

  $ git clone https://github.com/mkerrisk/man-pages.git

The most notable changes in man-pages-3.35 are as follows:

• A new recvmmsg(2) page (written by Andi Kleen and me) documents the recvmmsg() system call that was added to Linux back in version 2.6.32.
• A new setns(2) page (written by Eric Biederman) documents the setns() system call that was added in Linux 3.0.
• Various global edits to improve the consistency of terminology used in the man pages.

November 23, 2011 09:25 AM

I've released man-pages-3.34. Again, because of the problems at kernel.org, the various pieces of the release are at different locations than usual (and possibly some of these locations may become permanent). The browsable online pages can be found on man7.org; the release tarball is available here.

A git repository for man-pages can be found on github. You can clone the repository with the command:

$ git clone https://github.com/mkerrisk/man-pages.git

The most notable changes in man-pages-3.34 are as follows:

• A new rt_sigqueueinfo(2) documents the rt_sigqueueinfo() and rt_tgsigqueueinfo() system calls. At the same time, the sigqueue() man page was moved from Section 2 to Section 3, since the interface it documents really is a library function (layered on top of rt_sigqueueinfo()).
• New man pages for the cciss(4) and hpsa(4) drivers. Thanks to Stephen Cameron.
• The fstatat(2) man page adds documentation of the AT_NO_AUTOMOUNT flag added in Linux 2.6.38.
• The lseek(2) man page adds documentation of the SEEK_HOLE and SEEK_DATA commands scheduled for the upcoming Linux 3.1. (These commands allow a program to map file holes.)
• The madvise(2) man page adds documentation of the MADV_HUGEPAGE and MADV_NOHUGEPAGE commands added in Linux 2.6.38.
• The feature_test_macros(7) man page adds documentation of the _ISOC95_SOURCE macro added in glibc 2.12

November 23, 2011 09:25 AM

After a long gap, I've released man-pages-3.33. Given the ongoing problems at kernel.org, the various pieces of the release are at different locations than usual (and possibly some of these locations may become permanent). The browsable online pages can be found on man7.org; the release tarball is available here.

A git repository for man-pages can be found on github. You can clone the repository with the command:

$ git clone git@github.com:mkerrisk/man-pages.git

The most notable changes in man-pages-3.33 are as follows:

• The sync(2) page adds documentation of the syncfs() system call that was added in Linux 2.6.39.
• The syscalls(2) has been updated to be current as at Linux 3.1.
• The syslog(2) page adds documentation of the CAP_SYSLOG capability added in Linux 2.6.37.

In addition, various fixes were made in many other pages.

November 23, 2011 09:24 AM

Almost exactly a year ago I posted about switching my kitchen to some PAR20 LED bulbs, talking about the economics of them, given their very long (50,000 hour) expected lifetime, and 5 year warranty.

The bad news – last night one quit on me. The good news is, Home Depot treated me very well; the guy from the electrical section just said “well, 5 year warranty and we haven’t even carried them that long. Here’s a new one.” I had no receipt, and they didn’t even bother looking it up.

I had to manhandle it into the PAR20 gimbal a bit; maybe that is why it died an early death? I hope this isn’t common. I paid $25 each (now $20) and expected to never have to change those bulbs again. Check back next year to see how the other 5 are faring.

November 23, 2011 04:28 AM

Tools shape the way we work, because they change where we perceive risk when we write code.  If common compilers warn about something, I’ll code in a way that will trigger it in case of mistakes.  eg: instead of:

    int err = -EINVAL;
    if (something())
         goto out;
    err = -ENOSPC;
    if (something_else())
         goto cleanup_something;
...
cleanup_something:
    undo_something();
out:
    return err;

I would now set err in every branch:

    int err;
    if (something()) {
        err = -EINVAL;
        goto out;
    }
    if (something_else()) {
        err = -ENOSPC;
        goto out;
    }

Because when I add another clause to the initialization and forget to set err, gcc will warn me about it being uninitialized.  This bit me once, and it can be hard to spot the problem when you’re only reviewing a patch, not the code as a whole.

These days, we have valgrind, and despite its fame as a use-after-free debugger, it really shines at telling you when you rely on the results of an uninitialized field.  So, I’ve adapted to lean on it.  I explicitly don’t initialize structure members I don’t use in a certain path.  I avoid calloc(): while 0 is often less harmful than any other value, I’d much rather know that I’ve thought about and set up every field I actually use.  When changing code this is particularly important, and I spend a lot of my time changing code.  I have even changed to doing malloc() in some cases where I previously used on-stack or file-scope variables.  Valgrind doesn’t track on-stack usage very well, and static variables are defined to be zeroed, so valgrind can’t tell when I wander into the weeds.  I think these days, that’s a misfeature.

So, if I were designing a C-like language today, I’d bake in the concept of undefined values, knowing that the tools to leverage it are widely available.  10 years ago, I’d have said 0-by-default is safest, but times change.  I think Go chose wrong here, but it may not be as bad as C for other reasons.  I’d have to code in it for a few years to really tell.

November 21, 2011 06:04 AM

I’m back from LinuxCon Brazil, that was held in Sao Paulo on 17 and 18 November. Before the first keynote, ProFUSION was announced as becoming member of Linux Foundation ! Our logo is already in their members page. It was also a great time to talk again to some developers I met in LinuxCon Europe last month [...]

November 19, 2011 07:54 PM

Yesterday I wrote about an alleged attack on the Windows 8 secure boot implementation. As I later clarified, it turns out that the story was, to put it charitably, entirely wrong. The attack is a boot kit targeted towards BIOS-based boots. It lives in the MBR. It'll never be executed on any UEFI systems, let alone secure boot ones. In fact, this is precisely the kind of attack that secure boot is intended to protect against. So, context.

The MBR contains code that's executed by the BIOS at boot time. This code is unverifiable - it's permitted to have arbitrary functionality. There's only 440 bytes, but that's enough to jump to somewhere else and read code from elsewhere. There's no way for the BIOS to know that this code is malicious. And one thing this code can obviously do is load the normal boot code and modify it to behave differently. Any self-validation code in the loader can be patched out at this point. The modified loader will then load the kernel, and potentially also modify it. At this point, you've lost. Any attempts to validate the code can be redirected to the original code and so everything will look fine, up until the point where the user runs a specific application and suddenly your kernel is sending all your keystrokes over UDP to someone in Nigeria.

These attacks exist now. They're in the wild. In a normal UEFI world you'd do the same thing by just replacing the UEFI bootloader. But with secure boot you'll be able to validate that the bootloader is appropriately signed and if someone's modified it you'll drop into some remediation mode that recovers your files, from install media if necessary.

Obviously, this protection is based on all the components of secure boot (ie, everything that runs before ExitBootServices() is called) being perfect. As I said, if any of them accept untrusted input and misinterpret it in such a way that they can be tricked into running arbitrary code, you'll still have problems. But when discussing the pros and cons of secure boot, it's important to make sure that we're talking about reality rather than making provably false assertions.

comments

November 18, 2011 09:03 PM

This is interesting. It's obviously lacking in details yet, but it does highlight one weakness of secure boot. The security for secure boot is all rooted in the firmware - there's no external measurement to validate that everything functioned as expected. That means that if you can cause any trusted component to execute arbitrary code then you've won. So, what reads arbitrary user data? The most obvious components are any driver that binds to user-controlled hardware, any filesystem driver that reads user-provided filesystems and any signed bootloader that reads user-configured data. A USB drive could potentially trigger a bug in the USB stack and run arbitrary code. A malformed FAT filesystem could potentially trigger a bug in the FAT driver and run arbitrary code. A malformed bootloader configuration file or kernel could potentially trigger a bug in the bootloader and run arbitrary code. It may even be possible to find bugs in the PE-COFF binary loader. And once you have the ability to run arbitrary code, you can replace all the EFI entry points and convince the OS that everything is fine anyway.

None of this should be surprising. Secure boot is predicated upon the firmware only executing trusted material until the OS handoff. If you can sidestep that restriction then the entire chain of trust falls down. We're talking about a large body of code that was written without the assumption that it would have to be resistant to sustained attack, and which has now been put in a context where people are actively trying to break it. Bugs are pretty inevitable. I'd expect a lot of work to be done on firmware implementations between now and Windows 8 ship date.

(Update: It's probably worth pointing out that although Ars talk about it defeating secure boot, the primary sources don't seem to mention secure boot at all. Windows 8 will boot without secure boot, so it's possible that this attack merely handles the non-secure boot case)

(Further update: It turns out to be a purely BIOS based attack. In other words, it's an example of the kind of thing that secure boot is intended to prevent, not any kind of compromise of a secure boot implementation)

comments

November 18, 2011 03:08 AM

Starting with Fedora 16 we're installing using GPT disklabels by default, even on BIOS-based systems. This is worth noting because most BIOSes have absolutely no idea what GPT is, which you'd think would create some problems. And, unsurprisingly, it does. Shock. But let's have an overview.

GPT, or GUID Partition Table, is part of the UEFI specification. It defines a partition table format that allows up to 128 partitions per disk, with 64 bit start and end values allowing partitions up to 9.4ZB (assuming 512 byte blocks). This is great, because the existing MBR partitioning format only allows up to 2.2TB when using 512 byte blocks. But most BIOSes (and most older operating systems) don't understand GPT, so plugging in a GPT-partitioned disk would result in the system believing that the drive was uninitialised. This is avoided by specifying a protective MBR. This is a valid MBR partition table with a single partition covering the entire disk (or the first 2.2TB of the disk if it's larger than that) and the partition type set to 0xee ("GPT Protective"). GPT-unaware BIOSes and operating systems will see a partition they don't understand and simply ignore it.

But how do we boot a GPT-labelled disk with a protective MBR on a system that doesn't understand GPT? The key here is that BIOS is pretty dumb. Typically a BIOS will see a disk and just attempt to execute the code in the first sector. This MBR code knows how to do the rest of the boot, including parsing the partition table if necessary. The BIOS doesn't need to care at all.

Of course, some BIOSes choose to care. We've seen a small number of machines that, when exposed to a GPT disk, refuse to boot because they parse the MBR partition map and don't like what they see. This is typically accompanied by a message along the lines of "No operating system found". What we've found is that they're looking for a partition marked with the bootable flag, and if no partitions are marked bootable they assume that there's no OS. This is in contrast to the traditional use of the flag, which is merely a hint to the MBR as to which partition boot code it should execute.

So, should we set that flag? The UEFI specification specifically forbids it - table 15 states that the BootIndicator byte must be set to 0. Once again we're left in an unfortunate position where the specification and reality collide in an awkward way.

If this happens to you after a Fedora 16 install, you have two choices. The first is to reinstall with the "nogpt" boot argument. The installer will then set up a traditional MBR partition table. The second is to boot off a live CD and run fdisk against the boot disk. It'll give a bunch of scary warnings. Ignore them. Hit "a", then "1", then "w" to write it to disk. Things ought to work then. We'll figure out something better for F17.

comments

November 17, 2011 03:54 PM

A reasonably common design for applications that want to run code at a specific time is something like:

time_t wakeup_time = get_next_event_time();
time_t now = time(NULL);
sleep(wakeup_time-now);

This works absolutely fine, except that sleep() ignores time spent with a suspended system. If you sleep(3600) and then immediately suspend for 45 minutes you'll wake up after 105 minutes, not 60. Which probably isn't what you want. If you want a timer that'll expire at a specific time (or immediately after resume if that time passed during suspend), use the POSIX timer family (timer_create, timer_settime and friends) with CLOCK_REALTIME. It's a signal-driven interface rather than a blocking one, so implementation may be a little more complicated, but it has the advantage of actually working.

comments

November 17, 2011 01:51 PM

I was at the Open Source booth at this year’s Grace Hopper Celebration of Women in Computing, last Thursday. We had Fedora media and swag also available Wed and Fri as well. The booth was shared with Mozilla, Kids on Computers. Wikimedia, OSU OSL, the Ada Initiative, and assorted other FOSS projects. While I talked about Fedora and handed out media and stickers, many attendees were students and new to tech as a whole, so “what is open source” was the first, most important, question.

I also attended the Codeathon for Humanity on Saturday, with about 200 attendees working on various open source philanthropic projects: Sahana, Systers, Mailman, peer-to-peer Haitian Women’s network, to name a few. The Mayor of Portland, Sam Adams, stopped by and spoke. I wasn’t expecting this, but was pretty nice to hear about the importance of open source to the city government, as well as the Portland tech scene as a whole!

November 17, 2011 01:40 AM

It is something I have done many times under many different operating systems. You write something into the startup scripts, take the system down, then regain control at startup. So I planned to take this approach for my KVM-based rcutorture setup: Mount the image file, deposit a test script in the startup scripts, unmount the image file, start qemu, then analyze the results (either deposited in the filesystem or dumped to the console). What could be easier?

So I manually mounted the file I use as the qemu disk image, my plan being to manually create simple tests as a first step towards automation.

Hmmm... Everything looks very strange and unfamiliar. Ah, yes, this is upstart. No problem, the Internet is but a mouse-click away, right? But wait, the documentation says that upstart and the more-familiar (to old guys like myself, anyway) sysvinit can co-exist on the same system. And what about this new systemd thing that I keep hearing about?

So my tests are going to have to inspect the filesystem and figure out which of these three things is installed. And maybe someone installed one of them, then switched to the other, leaving traces of their earlier choice lying around. How is my script supposed to figure all that out? Of course, if it was just me running the tests, I could simply make sure that a pre-selected boot-up mechanism was installed. But that is silly — anyone anywhere should be able to test RCU. So I need to be able to deal with any of the three boot-up mechanisms. But wait, it is far worse than that: What is my script supposed to do when someone else comes up with a fourth, fifth, or sixth boot-up mechanism? Ouch!!!

It is clearly time to just say “no”! After all, my main test (rcutorture) is coded in the kernel, so it is clearly going to be far easier to translate my driver scripts to Linux kernel code than to code up a script that can outwit all present and future perpetrators of bootup mechanisms. In-kernel coding will allow me to fully control the test using kernel boot parameters, and thus happily ignore which of sysvinit, upstart, systemd, or whatever is installed. Some times you just have to get a horse!

November 11, 2011 06:12 PM

Chris Siebenmann mentioned in e-mail to me that a new terminal window opens if one Ctrl-clicks on it in Activities in GNOME 3 (there are more tips in comments of his blog post). Previously, I found that right-click there brings up a menu that can open new windows, but this takes it to the new level. Owen Taylor, you are so devious!

Back in 1985 or so, a company where I worked obtained a very expensive and unique piece of equipment: an IBM PC with a hard drive. Someone installed a game on it, called "Leisure Suit Larry" one or the other, and the office took turns playing it. The game was a quest with "quasi-natural" command input. At some point in the game, the player has a dull knife that does not cut. Someone heard from someone that it needs to get sharpened, but nothing we tried worked. Nobody in the firm knew English, so I remember trying "make knife sharp", "knife sharp", "sharp knife", and any other permutation including "knife sharp do". The right answer was "sharpen knife".

GNOME 3 reminds me of that quest. Random inputs that make complete sense to Owen, same as the conjugation into imperative made complete sense to an English speaker, but entirely opaque and mysterious to people who are not in on the secret.

November 10, 2011 07:16 PM

This is mostly for my own reference, but since it might be useful to others:

By "Properly booting" I mean "Integrating into the boot system as well as Mac OS X does". The device should be visible from the boot picker menu and should be selectable as a startup disk. For this to happen the boot should be in HFS+ format and have the following files:

• /mach_kernel (can be empty)
• /System/Library/CoreServices/boot.efi (may be booted, if so should be a symlink to the actual bootloader)
• /System/Library/CoreServices/SystemVersion.plist which should look something like

  <xml version="1.0" encoding="UTF-8"?>
  <plist version="1.0">
  <dict>
          <key>ProductBuildVersion</key>
          <string></string>
          <key>ProductName</key>
          <string>Linux</string>
          <key>ProductVersion</key>
          <string>Fedora 16</string>
  </dict>
  </plist>

That's enough to get it to appear in the startup disk boot pane. Getting it in the boot picker requires it to be blessed. You probably also want a .VolumeIcon.icns in / in order to get an appropriate icon.

Now all I need is an aesthetically appealing boot loader.

comments

November 09, 2011 10:06 PM

At the end of October I’ve been to Prague, Czech Republic to take part in this year Bluetooth Summit to discuss the issues and the future developments of the BlueZ Project. It was the biggest BlueZ meeting we ever had, in total there was 21 people attending our summit.

The discussions were dominated by the next steps in the Low Energy, Bluetooth 3.0 + High Speed and BlueZ 5.0 (our next major release). On the LE energy side the discussion were in all parts of its stacks, there are pending patches for the interleaved discovery, Management commands for LE, RSSI and Tx power handling. and the development on GATT was one of the most discussed things.

On the BlueZ 5.0 side we are going to drop a lot of old stuff like pand and dund daemons, IPC mechanism for audio as well some of our DBus API we already marked as deprecated. In the management API side the idea is to finish it as soon as possible and enable it by default in the kernel, then some day in the future we can get rid of hciops and raw access to HCI sockets in bluetoothd.

My summit tasks are finish the L2CAP separation in sock and core parts and move the Bluetooth subsystem to use workqueues instead of tasklets. ;-)

After the Bluetooth Summit I attended the open day of the kernel summit, you can find nice reports of it in LWN here and here.

On Wednesday both LinuxCon Europe and Embedded Linux Conference Europe started. It was certainly a good idea to put both conferences together in the same place at the same time, I had the opportunity to meet people and see talks from both conferences. Some of the talks I took part were “Demystifying HTML5″ by Sulamita Garcia & Gustavo Barbieri, “The Linux NFC Subsystem” by Lauro Ramos Venancio & Samuel Ortiz. “systemd Administration in the Enterprise” by Lennart Poettering & Kay Sievers. “Linux for In-Car Infotainment” by Matt Jones.

Finally, I would like to thank ProFUSION to sponsor my travel to Prague.

November 09, 2011 07:27 PM

Recently I had been invited by the Korean Open Source Software (KOSS) Law Center to attend their 2011 KOSS conference scheduled for November 17 and 18 in Seoul, Korea.

This conference is organized by the KOSS Law Center with support by the Korean Government (National IT Industry Promotion Agency). Its primary purpose is to share best practises in terms of FOSS licensing, license compliance but also FOSS community interaction within the Korean IT industry and the public sector.

I'm happy to present on Beyond Legal Compliance - Embracing the FOSS community, where I will outline that the primary focus should not be on to-the-letter legal compliance, but to a proactive way of interacting with the FOSS community. After all, collaborative development is what FOSS is all about...

However, due to a schedule conflict with the DeepSec 2011 conference in Vienna (where I'm giving a two-day GSM security workshop), I'm only able to attend the second day of the KOSS conference.

The speaker line-up for the KOSS conference is quite impressive, and it includes Karsten Gerloff (FSFE), Till Jaeger (JBB), Carlo Piana (FSFE), Keith Bergelt (OIN), Armijn Hemel (gpl-violations.org/Tjaldur) and others.

Unfortunately there seems to be no homepage, at least none with an English language title that Google would be able to find. Carlo Piana has mentioned the event in his blog four days ago.

UPDATE: There now is a conference page, although in Korean language only ;)

November 08, 2011 01:00 AM

It seems I'm really getting too lazy to update this blog more frequently, which is a pity. Last week I was in Stockholm attending the Erlang User Conference 2011. This was the first Erlang conference I ever went to, and it was the first conference in many, many years where I was not speaking but merely a normal attendee.

Some of the readers of this blog will already have noticed my microblogging updates on identi.ca and Twitter that I made during the conference. They were not overly excited about the conference. Let me write some more details here. I have no idea how many technical conferences I have attended, but I am typically speaking at something like 10 to 14 every year, which I believe qualifies me as a "professional conference participant" ;)

Let me start with some positive feedback: There have been excellent and technical presentations, particularly by Kostis Sagonas (PropEr), Melinda Toth (Change impact analysis) and also the talk on Hashes/Frames/Structs as new built-in Erlang data types by Kenneth Lundin.

However, apart from those, i have quite a bit of criticism:

• Some presentations ended way ahead of their schedule.
  This is a pity, as it means that some hundred-odd highly paid software developers are then sitting in a room and wasting time. If you hold a presentation at a conference, you should make sure that this time is used in the most efficient way. If you have been allocated a 45 minute slot, please don't make a 15 minute presentation + 5 minute questions session. That's not what the audience expects!
• Keynote presentation by Ulf Wiger contained lots of hot air
  If I go to a technical conference aimed at Erlang users (i.e. software developers who write programs using the Erlang language, libraries and runtime system), then I expect it to be loaded with brilliant, technical content. I want to get excited about new developments, Erlang software projects, etc. The last thing that I'd want is having a real Erlang guru on stage talking about superficial, trivial aspects of embedded computing. Of course I respect the commercial decision of Ulf and/or Erlang Solutions to try to create a market for Erlang in the embedded sphere. But what is the technical relevance of this to the Erlang community? Ulf did not talk about great new schemes of optimizing the Erlang VM for battery-powered CPUs, or how he has extended powertop to give function or line-level accuracy on which of your Erlang code lines burn most CPU cycles or cause the highest number of CPU wake-ups from low power mode. That would have been exciting.
• Erlang/OTP Road-map presentation without much technical details
  When I see a slide with "Some SCTP improvements" then I want to see what exactly are those improvements. I think there was more than enough time to go into more details, if Kenneth would have spoken faster and put more content into the available time. Once again, the audience is a room full of intelligent, highly-paid professional software engineers. If you get their attention for whatever amount of time, I believe you should pack it as full with information as possible, rather than bore them with slowly and carefully reading each line from a slide...
• No Internet available at the Tutorials Can you believe it? In 2011, a technical conference aimed at software developers hosts tutorials inside a facility owned by one of the largest communications equipment suppliers (Ericsson) and then there is no provision for Internet access. It's really ironic, especially since at least some of the tutorial trainers expected the attendees would be able to clone git repositories on their laptops during the workshops.

In my hallway conversations with other attendees (who also have a background outside of Erlang and are more familiar with other conferences in the FOSS community), they independently observed those very same issues and agreed with my assessment.

All in all, the conference was a good trigger for me to finally sit down and start to use dialyzer on the various Osmcoom Erlang-language projects such as osmo_ss7, osmo_sscp and signerl. I'm already adding type specifications all over the code and am looking forward to soon starting with some PropEr test cases in the next couple of days.

November 08, 2011 01:00 AM