Kernel Planet

Dave Airlie: whats in drm-radeon-testing?

Mon, 08 Feb 2010 23:58:05 +0000

I'll try and post these regularly when I make major additions/removals.

drm-radeon-testing is the cutting edge KMS radeon branch, it is going to be rebased and things will be added/removed as they are worked on by developers. So you can base patches on it but you should talk to the developer who owns the area first.

git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6.git drm-radeon-testing

I've just pushed a rebased tree now with the following:

latest i2c algo + hw i2c engine code + all fixes squashed: This adds support for hw i2c engines found on radeons and
exposes them + sw i2c buses to userspace so i2c tools can use them. (agd5f).

pll algorithm reworking + quirks: cleans up the code to allow for the selection of the old pll algorithm on some hardware. (agd5f)

pm support so far: Adds all the current PM patches - just does engine reclocking so far using the power tables from the BIOS. (Zajec/agd5f)

Evergreen (Radeon HD 5xxx) support: basic KMS support for the evergreen range of devices - no irqs or accel yet. (agd5f)

radeon unlocked ioctl support (airlied)

bad CS recording (glisse)

misc cleanups/fixes - Dell/Sun server support ported from userspace hopefully.

The tree did contain Jerome's r600 CS checker but I've dropped it for now at his request as he has newer patches
in testing.

Valerie Aurora: Linux Storage and Filesystems Workshop

Mon, 08 Feb 2010 20:59:31 +0000

The 2010 Linux Storage and Filesystems Workshop has been announced: http://lkml.org/lkml/2010/2/8/221 One of the things I like most about the file systems workshop is the avoidance of canned presentations: Presentations are allowed to guide discussion, but are strongly discouraged. There will be no recording or audio bridge, however written minutes will be published as in previous years [...]

Pete Zaitcev: ncld is here, and what it is for

Sun, 07 Feb 2010 19:46:52 +0000

I sent out the "ncld" (a pun on ncurses) that I mentioned before. It is tested, and I already have tabled switched over to it. All is left is to get Jeff to apply it.

Savings in the code size are pretty good, but more importantly it is not impenetrable spagetti from 1968 anymore. And this is important if we want anyone ever hack on CLD voluntarily. People actually pay attention to shit code. To quote:

Fedora uses yum, which originally was developed by Yellow Dog. I forgot who told me this (I think it was either Jeremy [Katz] or Notting [Bill Nottingham]), but the story was that they (e.g. MSW [Matthew Wilson], Notting, and Jeremy) looked at things like apt-rpm, urpm, and yum as a base for [the] next up2date. Only yum passed the test "not to puke while looking at the source". That's how it came to be.

If you tilt your head just right, CLD is somewhat similar to Zookeeper in function (or so I heard), and one time someone asked why not just use Zookeeper. Jeff answered, "Zookeeper's API is too complex". I was concerned that someone would look at the code and think we were NIH hypocrites, because CLD's API before ncld was complex too (for no good reason - it was assembly-level complexity). Well, not anymore. What Jeff actually meant, I think, was that Zookeeper intrinsic architecture was too complex for what we want in Hail.

Pete Zaitcev: Why 1e100.net?

Sun, 07 Feb 2010 16:31:05 +0000

Jon Masters posted a somewhat ambiguous twit: "Why does Google use 1e100.net?" Obviously, 1e100 is the exponent syntax for gugol, but aside from that, why is it necessary to use a separate domain? It seems like a trick that is done often. Here's a short list:

Youtube uses ytimg.com. They started doing it before the acquisition by Google, and apparently it was used to host the static content. Youtube used 3-rd party CDNs for hot videos back then, but always with youtube.com for a domain. They continue that practice, except that ytimg.com now serves other random stuff now.

Google uses 1e100.net, which seems to pop randomly.

Facebook uses fbcdn.net. Obviously it means "Facebook CDN".

So, using a second SLD is clearly a common practice of some kind, that everyone in the business agrees is valuable. But how exactly does it work? Is it about the performance or security? Why only 2 domains and not 10 or 1000?

UPDATE: I have two friends called "Jon M." and other of them says:

It's for security. If you don't fully trust the security of your CDN, then you put it on a separate domain, so that content served from it can't access your users sessions.

Presumably it's because cookies are matched by domain. And Peter Jones adds:

It cuts down on http headers - especially cookies. If you put images on a second domain, it means /far/ fewer headers transferred, and fewer db lookups for the things /in/ them.

Those cookies!

Dave Miller: STT_GNU_IFUNC

Sun, 07 Feb 2010 15:46:00 +0000

I've always wanted to work on support for STT_GNU_IFUNC symbols on sparc. This is going to solve a real problem distribution makers have faced on sparc64 for quite some time.

What is STT_GNU_IFUNC?

Well, normally a symbol is resolved by the dynamic linker based upon information in the symbol table of the objects involved. This is after taking into consideration things like symbol visibility, where it is defined, etc.

The difference with STT_GNU_IFUNC is that the resolution of the reference can be made based upon other criteria. For example, based upon the capabilities of the cpu we are executing on. The most obvious place this would be very useful is in libc, where you can pick the most optimized memcpy() implementation.

Normally the symbol table entry points to the actual symbol location, but STT_GNU_IFUNC symbols point to the location of a "resolver" function. This function returns the symbol location that should be used for references to this symbol.

So when the dynamic linker resolves a reference to a STT_GNU_IFUNC type symbol "foo". It calls the resolver function recorded in the symbol table entry, and uses the return value as the resolved address.

Simple example:

void * memcpy_ifunc (void) __asm__ ("memcpy");
__asm__(".type foo, %gnu_indirect_function");

void *
memcpy_ifunc (void)
{
  switch (cpu_type)
    {
  case cpu_A:
    return memcpy_A;
  case cpu_B:
    return memcpy_B;
  default:
    return memcpy_default;
    }
}

So, references to 'memcpy' will be resolved as determined by the logic in memcpy_ifunc().

These magic ifunc things even work in static executables. How is that possible?

First, even though the final image is static, the linker arranges to still create PLT entries and dynamic sections for the STT_GNU_IFUNC relocations.

Next, the CRT files for static executables walk through the relocations in the static binary and resolve the STT_GNU_IFUNC symbols.

There are some thorny issues wrt. function pointer equality. To make that work static references to STT_GNU_IFUNC symbols use the PLT address whereas non-static references do not (they get fully resolved).

Back to the reason I was so eager to implement this. On sparc we have four different sets of optimized memcpy/memset implementations in glibc (UltraSPARC-I/II, UltraSPARC-III, Niagara-T1, Niagara-T2). Right now the distributions have to thus build glibc four times each for 32-bit and 64-bit (for a total of 8 times).

With STT_GNU_IFUNC they will only need to build it once for 32-bit and once for 64-bit.

I've just recently posted patches for full support of STT_GNU_IFUNC symbols to the binutils and glibc lists.

Linus Torvalds: Happy camper

Sat, 06 Feb 2010 13:22:03 +0000

I broke down and bought a Nexus One last week.

I got the original G1 phone from google when it came out, and I hardly ever used it. Why? I generally hate phones - they are irritating and disturb you as you work or read or whatever - and a cellphone to me is just an opportunity to be irritated wherever you are. Which is not a good thing.

At the same time I love the concept of having a phone that runs Linux, and I've had a number of them over the years (in addition to the G1, I had one of the early China-only Motorola Linux phones) etc. But my hatred of phones ends up resulting in me not really ever using them. The G1, for example, ended up being mostly used for playing Galaga and Solitaire on long flights, since I had almost no reason to carry it with me except when traveling.

But I have to admit, the Nexus One is a winner. I wasn't enthusiastic about buying a phone on the internet sight unseen, but the day it was reported that it finally had the pinch-to-zoom thing enabled, I decided to take the plunge. I've wanted to have a GPS unit for my car anyway, and I thought that google navigation might finally make a phone useful.

And it does. What a difference! I no longer feel like I'm dragging a phone with me "just in case" I would need to get in touch with somebody - now I'm having a useful (and admittedly pretty good-looking) gadget instead. The fact that you can use it as a phone too is kind of secondary.

Pavel Machek: Welcome to ugly world of windows

Sat, 06 Feb 2010 07:06:35 +0000

So you want example sources, cca 100K total. You have to download 15MB
slac341.zip, that extracts into 15MB Chronos-Setup.exe. That in turn
blackmails you into accepting about 20 pages of ugly legaleese
presented in tiny window. You get past that (in wine), select directory, and then InstallJammer tells you that you dont have permissions to the directory, oops.

Evgeniy Polyakov: Elliptics network got new on-disk format

Fri, 05 Feb 2010 23:45:38 +0000

Eventually any storage should go into production mode, which implies not only data storage itself but also access restrictions. Distributed hash table systems, like elliptics network, do not have dedicated servers which could store that information and manage access permissions, so each object should have its own set of rules. Although without proper security framework on top of network media this will not guarantee required data access granularity, but even in this model it is still possible to implement IO permissions to some degree.

Until now elliptics network did not have even a slight mechanism for doing this. And even that rudimentary supported metadata was stored in the transaction log and did not allow any kind of extensions or proper updates.

And although I did not yet write any line of code to deal with metadata, I already broke old-style transaction logs, which now contain only and only transaction information. There are no metadata objects at all, but I will update appropriate parts of the library to generate them and store in the separate entities.

It is possible to store metadata in the different objects like the ones being indexed by the hash of the original object's name plus some extension, but this will force system to perform two lookups to find out needed object and its metadata.
Another way is to add new object type to existing transaction and history log objects - all metadata will be stored close to the object itself and could be fetched using only object ID. In the filesystem backend where each object is stored as separate file, metadata will be indexed by the '.metadata' extension or similar - just like we have $ID.history for transaction logs. In the database backend (BDB and Tokyo Cabinet, although I seriously consider to drop the former, since it is unacceptibly slow compared to TC) it will be a separate table, indexed by the object ID.

Metadata will have flexible format (maybe even human-readable one based on strings?) to allow extensions without breaking backwards compatibility.

But first I should fix background log checker, which although syncs all kinds of objects currently (i.e. when there is no some object in the storage, but there is its copy with different ID, it will upload missing data from that copy), it does it slightly wrong way, namely messing with hashes and producing unneded additional transaction references. When checker is ready, whole storage fsck process will just combine a log based on metadata objects, and start check process for it.

Stay tuned, we are very close to the next major release, which will draw the line of the serious features and changes!

Matthew Garrett: Shaping young minds

Fri, 05 Feb 2010 18:54:34 +0000

I'm off to CMU at the weekend, in order to do a couple of talks on Monday (the 8th). I'll be giving an introduction to ACPI to the operating systems class in the morning, and an open presentation on Fedora, some of the challenges we face and how to get involved in Linux in the afternoon. This is as a result of our cooperation with CMU, which has led to things like the request on the right. How could we refuse?

Evgeniy Polyakov: Opened skiing season

Fri, 05 Feb 2010 03:13:47 +0000

Of course its downhill skiing, I used to hate runnng skiing wasted 3 years in the section when was in the university.

And actually I not only opened a season, but tried it first time. Some years ago I made a downhill run on the board, but I weared 'grinders' shoes instead of special shoes :)

Anyway, I do not know how to downhill, so I took an hour ski lesson, found myself can not being able to perform even the simplest things like V-deceleration (I do not know how it is called in english, but it is supposed that feet form kind of V figure).
Apparently I found a way to learn this quickly - when you flight into the wall on the bone-breaking speed it is better to find out a way to decelerate and stop. One of them is to fall, but it has own and rather serious problem - pain, haematomas and ego drop.

First trainig was rather painful and without interesting results except that I found myself very liking this stuff. So I went there in a day and enjoyed the hell skiing from the top. Even multiple times down to the bottom without falls. And EVEN once (or at least half-once) I moved the way and speed I wanted.
I'm sure, my carving and V-turns are likely ugly as hell, I like how things go.

A good news is that whole-year hill center (well, I was lazy to move to the real hills :) is very conveniently located between my home and office, so I can spent 1-2 morning hours there. I always wanted to be able to ski and its quite possible now with car.

Not me :)

Harald Welte: Symbian is Open Soruce - Really?

Fri, 05 Feb 2010 01:00:00 +0000

In recent news, the Symbian Foundation announced that "All 108 packages containing the source code of the Symbian platform can now be downloaded from Symbian's developer web site". This is great news!

This morning I tried to look at the parts most interesting to me: phonesrv (implementing call engine, cell broadcast and SIM toolkit APIs) and poc (implementing push-to-talk). Their pages don't have the usual "source code" tab at the bottom right which links to mercurial and tarball download pages!

Either I'm too stupid, or I am unable to find any source code for those two components. I'm quite sure something essential like the API's for making phone calls are considered part of the Symbian platform. So how does that match with the statement that all packages containing the Symbian platform can now be downloaded?

Pavel Machek: turning $3000, 4GFLOPS Unix workstation into... way too heavy wristwatch

Thu, 04 Feb 2010 23:38:22 +0000

I started mychronos project on sourceforge. Laurent Arditi asked me what it is about, but I could not write him back, so... let me explain here.

For now I do not have the watch, so I set up
simple emulator... that will also make development easier later. It is extremely hacky, but works well enough to show time and allow you to set it. Just run it with input redirected from /dev/input/eventX -- from your keyboard. And you may want to comment out first few lines of main().

I'd like to make the firmware compilable under Linux, then extend it
with new features ... like time of sunset/sunrise, etc.

BTW...does someone have official chronos firmware in .tar or .zip? All I could find was .exe, and I'd prefer not to install wine.

Pavel Machek: Insult to the injury

Thu, 04 Feb 2010 23:24:28 +0000

So the battery firmware not only killed my battery, it also claims it
is very much okay:

root@amd:~# cat /proc/acpi/battery/BAT0/info
present:                 yes
design capacity:         74880 mWh
last full capacity:      74880 mWh

Dave Airlie: video of GPU switching

Thu, 04 Feb 2010 02:16:12 +0000

Here's a really badly shot video of GPU switching in action ;-0 - whiteouts are mostly be logging out and in ;-)

Dave Airlie: hybrid graphics : the story continues (part 3)

Wed, 03 Feb 2010 07:07:43 +0000

v6 of the patch + another patch which needs some work before I can merge it are available now.

This mainly cleans up the patch architecture a lot and allow for Matthew to put his nvidia code in easier hopefully. Its moves the ATPX specific code to the radeon driver.

The second patch is from an experiment that I videod on a webcam but am now failing to upload, I'll probably get a better video tomorrow, the lighting was fairly bad for it today.

It basically allows for a delayed gpu switch ( it changes the debugfs API ), and allows gpu drivers to block the switch.

The switch file now takes ON/OFF like always, but the PCI IDs input is gone. There are 4 commands

IGD - try and switch now to the integrated device - can fail if drm drivers block it (mainly if X has the device open)
DIS - try and switch now to the discrete device - can fail if drm drivers block it (X again).
DIGD - try a delayed switch to integrated device
DDIS - try a delayed switch to discrete device.

So with X running you can echo DDIS to the file and log off X, it'll then switch as soon as X closes the drm device, and when
gdm restarts X it'll be running on the discrete GPU. If we had a shiny GUI on top of this it'd be as close as MacOSX can do it.
When you select to do a delayed switch we power up the other GPU straight away so the switch is quicker.

It needs more debugging, some open issues include:

after a few switches it can die on its ass
powering up the Intel glitches the display even when running the AMD
there may be race conditions in the patch, probably need a mutex around device open + this stuff
suspend/resume - since we D3 the card, if you do an s/r cycle it'll resume it, we need a flag in the
driver to say its powered off by the mux and to ignore s/r cycles - I've started adding this to radeon.

mjg59 has access to an nvidia laptop and is looking closely at how to make that all work.

Matthew Garrett: Engaged

Wed, 03 Feb 2010 02:36:18 +0000

Greg Kroah-Hartman: Android and the Linux kernel community

Wed, 03 Feb 2010 00:18:00 +0000

As the Android kernel code is now gone from the Linux kernel, as of the 2.6.33 kernel release, I'm starting to get a lot of questions about what happened, and what to do next with regards to Android. So here's my opinion on the whole matter...

Dave Airlie: hybrid graphics on Linux (Part 2)

Tue, 02 Feb 2010 05:49:55 +0000

Okay v4 of the patch is available at

http://people.freedesktop.org/~airlied/vgaswitcheroo/
http://people.freedesktop.org/~airlied/vgaswitcheroo/0001-vga_switcheroo-initial-implementation-v4.patch

First thing I added was power up/down methods. This calls the DRM suspend/resume methods + along with the cut power for discrete GPU. I'm not sure dynamic Windows ever does this as it seems to take a bit of work, I suspect they just run the second GPU in really low power modes. This is slow because we have to repost the ATI card after turning it back on.

I then talked to mjg59 and worked over the ATPX detection. I removed all the DMI code and it should detect *any* laptop that uses ATPX (i.e. ATI/ATI and Intel/ATI) from what I can see. I've tested on Lenovo W500 and T500 now and it appears to work on both. Would be nice if someone on a ATI/ATI and/or ASUS ATI/ATI or ATI/Intel machines could give it a whirl. I think the main problem with ATI/ATI is the poweroff methods have a hardcoded Intel PCI ID. I've no idea yet how to tell on an ATI/ATI which device is the IGD and which is discrete. Its probably more than likely the IGD is the one with the ATPX method on it.

It doesn't switch off at boot yet but I've added commands to let you do it.

echo "OFF" > switch - turns off the not in use card, so if Intel and ATI are on at boot, it will turn off ATI
echo "ON" > switch - turns back on not in use card
echo "PCIID" > switch - causes a switch with full off/on cycles.

nvidia combos appear to use a DSM method and in theory nouveau_acpi.c should be detecting that, so it might be possible for someone to hook that up.

I've also started looking at some desktop integration via gdm or logout - but its not my usual place to code so going is a bit slower ;-)

Dave Airlie: hybrid graphics on Linux

Mon, 01 Feb 2010 09:03:08 +0000

So someone thought it would be a good idea to make laptops with two graphics chips in them and switch betweem them to save power.

Now other OSes support this to varying degrees, I think XP + MacOSX require a logout cycle and Vista/Win7 can dynamically switch while running, while Linux basically falls over in a heap.

So I sat down today with a Lenovo W500 which has an Intel GM45 and AMD Radeon 3650 Mobility in it, and I wrote a patch to try and get closer to the XP/MacOSX level.

The result of one days straight hacking is at:
http://people.freedesktop.org/~airlied/vgaswitcheroo/

The patch is totally focused on the Lenovo W500, other switchers will need to add stuff to this codebase.

So what works?
Boot in switchable graphics - which boots with intel and radeon turned on
KMS drivers load for radeon and intel, radeon BIOS stored in start of VRAM (driver hacked to read it)
bind to both drivers + fbs for both.
mount debugfs - cat /sys/kernel/debug/vgaswitcheroo/switch
2
0 :0000:01:00.0
1+:0000:00:02.0
shows the 02.0 (intel) device is in charge of the MUX.
goto runlevel 5, play with X under the Intel driver, goto runlevel 3 kill X
at fbcon echo "0000:01:00.0" > /sys/kernel/debug/vgaswitcheroo/switch
barely glitches console and switches
goto runlevel 5, play with X under the ATI driver, goto runlevel 3 kill X
echo "0000:00:02.0" > /sys/kernel/debug/vgaswitcheroo/switch
goto runlevel 5, play with X under intel again.
wash and repeat.

What does it do?
So far its just switching the MUX using the ACPI method and remapping all the console to the other framebuffer device,
it also reset the bits that denotes which devices is the boot vga device which X uses to pick the primary GPU. This
means X doesn't need an xorg.conf to switch. (I think all those patches are in upstream X server).

What does it not do?
It doesn't powerdown the radeon when its not in use yet. I know the ACPI call to power it off/on, and since I have
the BIOS I should be able to repost it. So I'll try adding the callbacks into the KMS driver to do this soon.
It doesn't poewrdown the intel when its not in use yet. Not sure what I can do here, since there is no ACPI method to turn
it off. I think I can just D3 the GPU, and use the normal s/r paths to bring it back. Again requires more investigation.
The whole what ACPI + methods map to what device, how the mux ids match etc will probably all need to be stored in the DMI table.
Anything not a Lenovo W500 - probably not that hard to add other Intel/AMD variants to this, add DMI and mux switching method.
nouveau isn't hooked up - this could probably be done by some interested party - the driver hooks so far aren't very hard.
No idea about ATI/ATI or NV/NV ones either.

I'm really hoping interested community people can make this actually useful to them on other hw, I won't have permanent access to the W500 to keep this all tested in the future.

Can we do dynamic switch without restarting X?
No. X needs a lot of work, a lot more than the day it took to hack the kernel.

How do we go forward?
We probably need to add gdm support to move this forward. A logout button that is "Switch GPU", that gdm kills the X server,
then hits the switch port and starts a new X server. I'll try and talk to some gdm hackers over the next few days.
I'll try and push this into a git tree against Linus current, and we can add tested patches for other machines as they go in.
Also the DMI section is only imaginary of what I think others might need, we might have to rip it all out. Also I've no idea
if there are ACPI methods to query the switchable modes etc.

Evgeniy Polyakov: In a meantime elliptics network went production

Mon, 01 Feb 2010 00:20:44 +0000

We started the first production elliptics network distributed hash table cloud last week. Configuration is rather simple - there are 3 virtual datacenters each one contains two physical machines with several Tbs of space each.

I do not know precise number of HTTP proxies (fastcgi frontends for elliptics network) installed, but we talked about one or two of them in each datacenter. Each proxy has 5 uploading and 50 downloading processes. Uploading ones are hidden behind firewall, and downloading proxies are configured to read small objects through themself (like XML and image files) and big objects (data files) are downloaded directly from storage nodes, proxy only generates XML output with direct URL to some storage node.

This pilot project will host only about 2-4 Tb of data, each object will have 3 copies, each one stored in the appropriate datacenter. We will add geographically-spread datacenters soon, which will host only the most popular content, so that clients local to those storage nodes would not go the main datacenter.

Objects are not supposed to be updated during its lifetime, only once uploaded and removed when time requires. Data synchronization for failed and new nodes will be done using fsck application, which currently does not support advanced merge algorithms we had in the elliptics network, it only checks number of copies and optionally downloads/uploads if something is missing. Fsck application uses log file to get information about what and how objects should be checked, currently storage does not store this metadata with objects themself.

And actually background fsck daemon discussed previously will not be something very different from this application. Instead I suppose that having a script which will parse object's metadata and invoke application is a good approach. Fsck checker does not yet work with all possible types of uploaded objects, namely it was not yet tested with the transaction logs, only with objects themself, since it is what we use in the pilot project.

The more I think about transaction merge algorithms, the more I like the idea, when we only merge them using timestamps. Currently elliptics network has 5 merge algorithms, which may or may not be appropriate in some or other setup. Merge algorithm is invoked when node joins the storage to sync its content with what we have in the storage. Idea is to drop this join syncing and use postponed fsck which will sync data sometime in the future. Thus it is a prerequistic to have multiple object copies in such system, since it can take a while to sync joining node.

I understand that it is not possible to maintain in-sync clocks on multiple machines to have correct timestamps for each update transaction, but it should not be a problem if proper locking is used. Although there is no appropriate distributed locking system yet, it will be implemented anyway (plan is to write PAXOS locking daemon).

Fsck application will be extended to allow external libraries to merge data, namely I have requests to allow external entities to merge data based on its content and not meta informaion I might store, so fsck will just call external functions loaded from provided shared library if configured.

I did not make a new release yet, we will figure out possible bugs and complete fsck application testing first. Next version will not have syncing during join time, so this will be a major update.

Stay tuned!

Pavel Machek: trapping access to variables in C

Sun, 31 Jan 2010 21:11:43 +0000

I thought I'd try writing (simple!) emulator of wireless watch. Simple means -- compile it by gcc on host, then emulate registers, and do graphics by ascii art.

Unfortunately, registers are accessed in "P2DIR |= 0x1;" way, and I'd really really prefer not to have to emulate the cpu. Is there nice way to trap all accesses to variable?

I could put those variables on unmapped page, then eat all the SIGSEGVs that would generate; but that sounds complex & tricky. (But does anyone have an example I could copy from?)

I do not think it is possible to use macros.

Maybe C++ classes could be used to trap the accesses without too much difficulty? (Can they? Does anyone have examples?) I probably could do that if I was willing to implement all the different operators (+, -, |, +=, |=, ...)... or is there way to avoid that?

Oh, and today I had fun with 5kV -- physics in practice. Did you know that electric fences do not really work on snow? Apparently snow is good enough insulator. So I had great idea of repairing electric fence without turning it off.

Well, at the beggining it was ok. I could not even feel the electricity. But slowly my shoes/clothing got wet, and I got some mild shocks... snow is good insulator so nothing really bad. Then I tried to repair broken fence, grabbed one end in one hand and second end in second hand... and quickly let both go (after getting not-so-mild shock). Snow being insulator does not really help when you hold two pieces of wire...

Rik van Riel: Something smells on Wall St (and/or Washington)

Sun, 31 Jan 2010 18:03:18 +0000

Wait a moment... The bailout of AIG was so critically important to the stability of the financial system, that neither Paulson, Geithner or Bernanke were involved in the process?! If something is important, surely all the top people would be closely involved? What is really going on? Some interesting facts (and speculation) Bloomberg and Zero hedge on suspicious AIG transactions before the fall. Please pay attention to the facts in-between the speculation, the truth appears to be quite ugly.

Pavel Machek: grub2: welcome back in 1995

Sun, 31 Jan 2010 06:33:53 +0000

So I made a mistake of upgrading to grub2 on my main machine. Now I

a) have to rerun scripts to regenerate grub.cfg each time I change something -- which reminds me of 1995

b) grub menu runs in some kind of strange graphics mode, and very slowly. That's 1991 or something

c) grub still supports strange (hd0,2) syntax; unfortunately they made an off-by-one and partition number is now 1-based, not 0-based as it was in grub-1. Aaarggh.

Pavel Machek: Thinkpad x60: battery firmware killed my battery

Sat, 30 Jan 2010 20:12:47 +0000

Ok, so galeon eating 100% cpu 80% of the time, thus keeping the machine hot and thus the battery hot was probably part of the problem, too, but...

Three days ago, battery was good enough for half an hour of 100% CPU load... that's still pretty good (should be like 1.5hours of low load). Unfortunately, I tried using notebook on battery power after "only" 5 hours of charge, and battery went too low. Not the battery firmware declared the battery dead, and refuses to charge it -- so I can no longer even move the notebook from one wall socket to another.

Is there any way to reset the battery firmware? The cells are still capable of at least half an hour, probably hour or more, and stupid firmware does not even let me to use them for half a minute :-(.

Would tp_smapi help?

Jesse Barnes: progress

Fri, 29 Jan 2010 21:37:07 +0000

Progress

Wow it’s been awhile. Life in the land of Linux graphics has been exciting recently, and there have been a few interesting developments on the Linux PCI front as well.

Linux Graphics Maturing

The Linux graphics stack has really been maturing recently. The Intel and radeon KMS drivers are seeing a lot of bug fixing, and nouveau is getting into shape as well. I think the Intel driver is in better shape than the userland driver ever was at this point (though that’s not to say it’s without defects; our serious bug count is still way too high for my liking). It supports more hardware and features, including power saving, DisplayPort, new hardware, advanced rendering APIs, than ever, and has been shipping in Linux distros for quite awhile now.

We recently finished off the page flipping support, and landed it upstream (it’ll be part of 2.6.33). We also landed a new, core, buffer execution interface (creatively named execbuf2), that allows for more flexibility in the way we submit our command buffers. Specifically, it allows us to control whether a given buffer needs to be mapped with a fence register for operations performed by the commands in its parent execution buffer. This allows our command buffers to be larger, since we won’t exhaust our fence registers prematurely by mapping all objects unconditionally, and allows us to enable tiled texture rendering on pre-965 chips, which can improve performance significantly for some types of rendering.

To support the page flipping work, I had to extend the DRI2 protocol a bit to include support for a SwapBuffers request. While I was at it, I added support for the SGI_video_sync and OML_sync_control extensions, which meant adding support for a few more requests. The SGI_video_sync addition was an important one, since its absence was a regression relative to DRI1. All this new protocol meant new Mesa and X server code, new DRI2 interfaces between the server and DDX drivers, and a bunch of testing and reworking of the interfaces as I figured things out.

All these new features are landed now, and should be a part of Linux 2.6.33, Mesa 7.8, X server 1.9 and xf86-video-intel 2.11. See CompositeSwap for an overview of the features and how they’re implemented. With that out of the way I’ve been able to think more about how compositors and clients should interact, so I came up with CNP. It’s not implemented yet, since I’m still gathering feedback on it, but my hope is that it will help us reduce memory consumption and partial frames in composited environments, as well as address some of the undefined behavior of current GLX calls when drawables are redirected.

Finally, after some discussions with toolkit and compositor developers, I worked with Kristian and Ian to come up with the INTEL_swap_event GLX extension (note it’s definitely possible to implement this on non-Intel as well, but only Intel has support at the moment). This extension allows GLX clients to receive X events when previously queued buffer swaps complete. So rather than making another swap call before the previous one has completed, clients with mainloops can simply poll their X event queue and do other work if their last swap isn’t done yet, rather than wasting time blocked in the server or queuing another swap and getting too far ahead of the display.

Using it all

One side effect of the new DRI2 code is that glXSwapBuffers calls are now totally asynchronous. Previous versions of DRI1 and DRI2 would either block waiting for vblank, or only return after the blit to implement the swap had completed. With the new code, a DRI2SwapBuffers protocol request ends up in the X server, where it’s scheduled by the DDX driver to occur at some later time (though in some cases it will happen immediately, e.g. if the drawable is offscreen). This leaves more time for clients to do other work while their swap occurs; the INTEL_swap_event extension can help clients take advantage of this extra CPU time.

Some optimizations are present in the new code as well. For instance, if the drawable is the same size as the current root window pixmap and there’s no clipping to worry about, the DDX driver can queue a page flip instead. This saves a tremendous amount of memory bandwidth, and so can really increase performance, especially on high resolution and/or bandwidth starved configurations (e.g. most integrated and embedded graphics platforms). Similarly, if a simple back to front copy is requested for a window, if the back and front pixmaps are the same size (i.e. the window manager hasn’t reparented the front window to accommodate decorations and the like), the DDX can simply exchange the backing pixmap object pointers rather than blit. Again this is important on low memory bandwidth platforms (though note this code is currently disabled due to lack of testing; however it’s trivial to enable once I have some test cases).

New hardware

With our Core i7 parts launched, I can talk about some of the hardware feature work we’ve been doing. Zhenyu has been doing most of the bringup and hardware support work for this platform, but I’ve been busy with one of the more interesting hardware features in the Core i7-6xx series, called Intelligent Power Sharing (IPS). Core i7-6xx and 7xx chips are MCP (multi-chip packages); both the CPU and GPU/MCH are in the same physical processor package, but not on the same die. This means they share a thermal and power design domain. In many cases, only one of the components will be very busy, and thus generating much heat or drawing much power, and it would be a waste to let any extra thermal or power headroom go unused. IPS allows one component to use more than its share of power or thermal budget so long as the other component is idle enough to allow it. One of the key parts of this technology is so-called “graphics turbo", in other words the capability of the GPU to exceed its default frequency (and therefore thermal and power budget) when possible. I posted support for this at around launch time (latest patch here), and hope to be able to post the full IPS driver soon, since the potential graphics performance upside is fairly large (still collecting measurements but I’m hoping for something around 15% or maybe even a little higher). The code also allows the GPU to downclock when idle, saving power. The CPU already has its own opportunistic turbo mode which is very effective, but there may be cases where giving it extra power will be helpful (though I’ve yet to find a benchmark, again I’m still testing).

PCI

A recent thread highlighted an interesting design choice in Linux. All platforms supporting PCI (indeed pretty much every platform, PCI or no), splits its address space into multiple regions, allowing for memory mapped I/O (MMIO) from the CPU to different devices. Discovering which ranges belong to which devices is done in a number of different ways, from hard coded offsets (as is found on many embedded platforms), to firmware descriptor tables (as found in OpenFirmware or ACPI), to physically reading MMIO routing information from CPU host bridges down through the hierarchy.

There’s a drive in Linux to support the last option. After all, Linux is the operating system driving your hardware, it should do everything itself, right? Well, that’s where we get into trouble. Linux usually runs on platforms designed for Windows (either specifically for Windows or for Windows in addition to Linux). Windows generally uses the second option to make it easier to port to new platforms. For better or for worse (usually the latter) BIOS writers for new platforms generally consider their work done when Windows boots on their new platform and the Windows device manager doesn’t have any dreaded “yellow bang"s next to devices in the device tree. This usually means the ACPI tables used to describe MMIO layout need to be fairly accurate, or Windows may map a device into a location occupied by another or by a host bridge range with decode priority, causing hangs, corruption or the dreaded “yellow bang".

In October of last year, for arguably good reason, we tried to take Linux down the last path. Yinghai Lu added support for reading root bus resource ranges directly from the host bridge on Intel systems. The thought was that we’d be insulated from firmware bugs this way, and have a more accurate view of the system in general. Unfortunately, due to the above, bridge vendors like Intel have no reason to fully document all the decode windows of a given host bridge, which bits might enable or disable decode for a given region, or generally worry about providing the sort of info we’d need to make this approach tenable. So as of now, we’ve removed the supporting code, and are placing a bet that using the same information Windows does (and hopefully in the same way) will give us the same level of portability. We actually tried this back in 2.6.31 I believe, but had to disable it because our resource tracking code couldn’t handle all the resources handed us by some ACPI firmware implementations. We (well Bjorn hopefully) should fix that limitation for 2.6.34, and we’ll try again, and hopefully fix quite a few resource mapping related bugs in the process.

Pete Zaitcev: Andrey Savochkin on Rusty Russell

Fri, 29 Jan 2010 20:50:54 +0000

Just saw Jon Masters quipping on Facebook: "netfilter scares me", and someone commented that nftables look promising. Apropos those, some years ago (in 2000 or so), I touched bases with Andrey, who was probably more known as a maintainer of eepro100 driver than any number of bigger things he did. I think he invented the beancounter. These days he's gone into the depths of Parallels, never to be seen again. Anyway, Andrey said (approximately):

I have an enormous respect for Rusty. That guy worked on the problem of firewall in Linux for years. And he had the fortitude to throw away his own old crap and write it again from scratch. Twice! [e.g. rejecting ipfwadm and ipchains — zaitcev])

Is Rusty trying to do it again? Need to google nftables one day.

Jens Axboe: F5100 and IOPS

Fri, 29 Jan 2010 13:41:58 +0000

I finally got everything setup and wired up. The box now has 80 modules installed and I'm using 4 SAS controllers which each connects to two different expanders. The last part is the most problematic, as the 3801 LSI controllers max out at roughly 110K IOPS. That's pretty lame, and I hear the newer controllers do > 300K IOPS which is more reasonable. Not really a huge surprise, I've been calling HBA firmware piggy and slow for years, seems the SSD craze finally got them kicked into gear. But it's a shame, since that leaves 75% of the available IOPS on the floor. I have 2 more controllers coming in (thanks Eric!), so that'll up it a bit, at least.

To test the IOPS rate, I wrote a simple fio job file that does 4K random O_DIRECT reads at depth 32. Initial results were pretty shabby. I got ~200K with a number of devices, adding more made it drop down to ~150K. I was expecting 400K at least, so this was a worry. Some quick profiling didn't show much of interest: some locking overhead, but it is hard to quantify just how much. The test box has 32 cores / 64 threads, which is really nice for testing, but sometimes makes profiling a bit more difficult since the high CPU count has a tendency to mask some issues.

Booting with only 4 CPUs enabled was much better; I got 430K IOPS easily. Interestingly, the rq completion affinity knob (which I've blogged about before, merged in 2.6.29 and enabled by default in 2.6.32) makes a big difference. Disabling that and the IOPS rate drops to ~250K.

Now to find out why we suck at 64 CPUs...

Rusty Russell: Code Reviewing: popt

Thu, 28 Jan 2010 11:40:23 +0000

I decided that every day I would review some code in ctdb. I never spend enough time reading code, and yet it’s the only way to really get to know a project. And I decided to start easy: with the popt library in ctdb.

popt is the command-line parsing library included with the RPM source, and used by SAMBA. I vaguely recall Tridge telling me years ago how good it was. I started with the man page, and it is an excellent and useful library: it does enough to make the simple cases less code that getopt_long, yet allows flexibility for complex cases.

The idea is simple: similar to getopt_long, you create an array describing the possible options. Unlike getopt, however, simple integers and flags are handled automatically. So you set up the context with the commandline, then hand that context to poptGetNextOpt() to do the parsing. That keeps parsing args until it hits an error or you’ve marked an argument to return a value for special handling. The manual page is excellent and made me feel really comfortable with using the code.

Now, the code itself. Even before you notice the four-space tabs and bumpyCaps, you see the lint annotations and docbook-style comments cluttering the source. It does make the code annoying to read, breaking CCAN’s Golden Rule. Typedefs of structs, typedefs to pointers, and several #ifdef __cplusplus complete my minor gripes.

The code is old and cross-platform: the header test for features we’d simply assume in a modern Linux. It has a simple set of bitmap macros (taken from pbm, judging from the name), a helper routine to find an executable in $PATH (using alloca!) . These are the kinds of things which would be in separate modules, were this in CCAN.

The definition of _free() to be a (redundantly-NULL-taking) free() which always returns NULL is used to effect throughout the code:

    defs = _free(defs);

Here is a trick I hadn’t seen before to zero an onstack var, and really don’t recommend:

poptDone done = memset(alloca(sizeof(*done)), 0, sizeof(*done));

The help-printing code is in a separate file, popthelp.c. It’s actually about half 1/3 of the code! That’s mainly because it takes pains to pretty-print, and it’s done by manually tracking the column number through the routines (aka ‘cursor’). These days we’d asprintf into a temp buffer and strlen() to figure out if we should start a new line and indent before printing this. Or even better, create a struct with the FILE * and the column number, and create a fprintf variant which updated the column number and figured out wrapping for us. Routines like maxArgWidth() which iterates the table(s) to figure how much to indent will still suck however: probably simplest to build all the strings in the help table in memory and then dump at the end.

I thought I found a buffer overrun, but a test program disproved it: the singleOptionHelp() uses its maxLeftCol (plus some extra stuff) to size a buffer. This works because maxLeftCol is previously calculated as the worst-case size of the argument help. Now, the code is unclear (AFAICT maxLeftCol must always be sufficient, so the extra bytes are wasted), but not buggy.

In summary, this is excellent code. Talloc would help it, as would modern printf usage (%.*s for example), but generally the code is in really good shape. I know that the popt code in RPM is a little updated, but I can’t imagine that much has changed in such an obviously-stable codebase. The temptation to rewrite it is thus very low: who knows what the testsuite would miss?

Rusty Russell: linux.conf.au 2010

Thu, 28 Jan 2010 04:29:40 +0000

After slightly disastrous preparation (my left knee in a brace as detailed for the perversely-curious here) my week went well. I tried to get back to my hotel room early each evening to rest as per doctor’s orders (and managed it except Friday night), but polishing my Friday talk tended to take a few hours every day.

Sunday

The Newcomer’s Session was well attended, but Jacinta and I were slack with preparation so it was unbalanced for my tastes. I raced to the post-session pub assuming my crutches would ensure I’d be the trailer, to find that I was wrong. It would have been better to explicitly and immediately drag people towards the pub, because that’s (seriously) the most important part of the introduction to LCA.

Monday

Miniconf time, and I started in the Open Programming Languages miniconf. There was some interestingly random language stuff there: it’s one of my few opportunities to get exposure to higher level languages. The miniconf talks were enthusiastic and unpolished as such things are supposed to be. Haskell, and all the wonderful things it doesn’t let you do by Stephen Blackheath was interesting, but lacked solid examples. Introducing Gearman — Distributed server for all languages by Giuseppe Maxia was a great short intro into an unknown project. vcs-pkg.org by Martin F. Krafft was classic work-in-progress talk with insights into a mundane but critical infrastructure problem (standards and practices for coordinating upstream and across distributions using distributed revision control).

Die Flash Die – SVG has arrived by Andy Fitzsimon gave classic bling talk with a message about the animation potential for SVG. Useful content, too, for those dealing with this, and I was blown away to hear of Gordon, a FOSS Flash™ runtime written in JavaScript.

How to Use FOSS Graphics Tools to Pay for College by Elizabeth Garbee was an insight into the US education system and a chance to find out what my friend Edale (I know she hates that meme!) was doing. But her talk didn’t quite gel for this audience. Unfortunately using the words “did you spot the head-fake?” riles me. You are suddenly telling me that you’ve been using your intellect to compete with me rather than to inform and enrich me.

Then came my own Talloc: Pick Up Your Own Garbage! talk, which was delayed by my miscalculation of transit time on crutches. A mediocre rehash of my previous talloc talks, but I wanted to put it in front of this group because it really offers fresh view into a program’s data structures at runtime.

Writing Facebook Applications in Perl by Paul Fenwick was a nice little introduction to the FB API from a Perl point of view, but he kept his powder dry for his awesome lightening talk on Friday.

Tuesday

I peered in at the tail end of the keynote which was apparently excellent. I woke a little early then did some more work on my presentation, and by the time I had breakfast I was incurably late. One person admitted to me that they watched the live stream from their hotel room, but I wasn’t that clever.

This this day was all hallway track for me, catching up with many people I haven’t seen since last year. Then the Speaker’s Dinner at Museum of New Zealand: Te Papa Tongarewa. This is also a fun time to chat with everyone, but I was disappointed that my crutches forced me to forgo learning a traditional Haka. It was also the first chance to greet the two chief organizers, who had been sick the first two days of the conference. Edale and I also had fun playing with their very-past-bedtime hyper 2 yo Brooke (until we were told not to stir her up any more!)

Wednesday

The keynote by Benjamin Mako Hill was a little chaotic but made his point about antifeatures well: how such things are only viable when consumers don’t have freedom of choice (in particular, ability to examine, modify and share the software they’re using). I then headed to Introduction to game programming by Richard Jones, where I struggled with pyglet before giving up and paying half-attention. I did learn some things though, and everyone who was active seemed to get great satisfaction from the tutorial.

Open Sourcing the Accountants by Jethro Carr lacked density. It takes a great deal of work to give a thorough comparison of different accounting packages, and his insights into how accountants think were insufficient to make that the backbone of his talk either.

subunit: Testing across boundaries for fun and profit by Robert Collins was slightly familiar ground for me, but as libtap maintainer he wanted me to attend. It was a good bread-and-butter talk, which perhaps could have benefited from a few more snazzy real-life examples (making testing sexy is hard though!). He semi-seriously suggested I should take over the C output implementation for subunit; still thinking…

I caught the questions at Teaching FOSS at universities by Andrew Tridgell and Robert (Bob) Edwards, which I will watch seriously once the videos are uploaded.

Then was one compulsory-attendance presentation of the week: The World’s Worst Inventions by Paul Fenwick. I had made a comment to Paul earlier in the week that I was concerned that my talk lacked substance. His reply was “I won’t comment how much substance is in my talk”. And any conclusions were left to the minds of the audience as full-costumed Paul took us through a series of invention disasters. I teased him about it later, but let’s be honest: if I could present like that I wouldn’t have worried about content either!

That evening was the HackOff. I’ve never tried competitive programming, so when we came up with the plan of a SAMBA team, I heartily endorsed it :) Intimidation is important at these events, and the tweet from Adam Harvey was promising: At the #lca2010 HackOff. There’s a table with Rusty, Tridge, Anthony Towns and Andrew Bartlett. We’re fucked. However, despite having the largest team (with 6 of us), we only just squeaked in by 2 minutes. Subtract any one of the team and we wouldn’t have won, though with fewer we might not have tried to brute-force the final question.

Thursday

Glyn Moody’s keynote was excellent. Then I lost some more hallway time before emerging in The Elephant in the Room: Microsoft and Free Software by Jeremy Allison. I thought it was a worthwhile and balanced presentation; of course it had a few cheap laughs in it, but the examination of Microsoft’s actions wrt FOSS is a worthwhile exercise if we want to assess their potential impact.

I was a bit late to Building a Xapian index of Wikipedia in less time than this talk takes by Olly Betts, but it was too unprepared for my tastes and I went in not knowing what Xapian was (though I picked it up from context). Tux on the Moon: FOSS hardware and software in space by Jonathan Oxer was good, but another one I was late to (15 minutes between talks seems to give me enough time to start conversations, but not enough to finish them).

Simplicity Through Optimization by Paul McKenney was a good talk if you didn’t know your RCU. For me I would have liked to hear more what the various lines of code were doing (before they were excised by the optimized implementation). But being deeply familiar with the theory and somewhat familiar with the practice, I’m probably in a minority.

By this stage I was exhausted, and Using Functional Programming Techniques In Your Favourite Language by Malcolm Tredinnick was in the same room so I stayed. This talk was a disappointment to me (and, I think, Malcolm) because it didn’t quite contain the general insights he’d believed were there when he proposed the talk. Nice for me to get an refreshing exposure to functional programming though.

Dinner at an Indian restaurant with the SAMBA people, which meant I was right near the PDNS, so I dropped in briefly then returned to my hotel room for an early night.

Friday

Nat Torkington’s keynote contained the classic “heckle Rusty” factor and was delightfully punchy. He rolled over to a very very strong set of lightning talks; a format which works so well at these geek-rich events. Paul Fenwick’s “Unfriendly” Facebook app was an awesome way to close.

Patent defence for free software by Andrew Tridgell (late again!) was familiar ground for me, but I wanted to see how he presented such a dry area. Lots of text: I would have included some more diagrams (claim dependencies are well represented by a tree, for example). But the audience were rapt, so I’m clearly too picky!

Last minute prep for my talk: I decided the previous night that I would use Notes View, only to find that noone could get it to work. Both notes and presentation appeared on the projector screen, fortunately as I was about to give up and run without notes, someone suggested I simply drag the notes view back onto my laptop screen! Sometimes the low-tech approaches evade our over-complicated minds.

FOSS Fun With A Wiimote by Rusty Russell was well-received. I didn’t go as far with the project as I had intended, due to personal time contraints and time lost wrangling with actual hardware, but sometimes that’s instructive too.

The presentation itself was flawed in three places. Firstly, my intro slide appeared all at once rather than clicking in one point at a time, destroying my carefully practiced routine at that point. Secondly, noone knew what LED throwies were: (an open source graffiti technology developed at the Graffiti Research Lab) and I so that slide was completely lost. Finally, I should have set up my replacement two-year-old on the stage where the audience and the cameras could see her clearly.

The closing announced Brisbane for lca2011, and I handed the virtual speakers’ gift to the organisers. That done, I was ready to relax at the Penguin Dinner. Most years I don’t even drink, knowing that I’ll have to do the auction. But as there was no auction I sat next to Nat Torkington to guarantee great conversation and was ready to chill. I did some singing, didn’t try the Haga (again). I even got a cuddle with the organiser’s very well-behaved 5-month-old son Adam.

Unfortunately, events conspired against me and I was dragged into a pledging war for a prize I didn’t want to win (and at which my doctor would be aghast). I thought we could get more money from the Wenches For Winching, who were weasonably wealthy and weally wanted to win. Ted Ts’o had a similar idea. Unfortunately the prospect of crippled Rusty being “rescued” (after being dropped: that was the no way part) was too alluring for many, and I had to work hard to ensure I didn’t win.

A good time had by all, though exhausting after a long week.

Saturday

Briefly peered into the Open Day, which was buzzing with setup and opening, before heading home, spent. I did find out that wild weather had wuined the winching of wenches; but there is a standing offer when they find themselves in Wellington again.

Summary

Absolutely on par with previous awesome conferences; there were no organisational disappointments for me the entire week. I was particularly happy to see people digging in and fixing things when they were wrong instead of simply complaining.

A great achievement, everyone!

Harald Welte: Sorry for new blog updates

Wed, 27 Jan 2010 01:00:00 +0000

I've been busy day and night, hacking away on my latest pet project in the GSM field. In fact, it's been a long time since I've been able to dedicate so much time and energy into one particular project, without many distractions at all. The project is now finally looking quite promising and making nice progress throughout the last three weeks.

If progress continues, I hope in another week I'll be able to reveal what this is all about. I haven't felt this level of excitement since the early days of Openmoko :)

Vegard Nossum: Space invaders?

Tue, 26 Jan 2010 22:40:47 +0000

After my exams in December, I relaxed by writing a sort of space game using C++/SDL/OpenGL:

I ripped the sprites from Space Invaders and Galaga/Galaxian, please forgive me. If whoever wants to play/hack, the source code can be found here:

http://github.com/vegard/spaceinv

The video is a bit poor, mostly because the frame rate is supposed to be 60 FPS, but the Ogg/Theora capture was 25 FPS while the actual gameplay was probably closer to ~10 while doing the capture. Need to fix that. It looks better when playing.

Pavel Machek: no does not mean no, if you are an android

Tue, 26 Jan 2010 22:10:42 +0000

Android phones want to use data network, all the time, and
anywhere. It is quite bad in home
network -- there's no "allow data usage" toggle and andoid will
connect automatically upon boot. If you have plain o2 card, thats $1.2
for them. (Fortunately there is apndroid and friends, that allow easy
toggling, but you are still racing against gprs network attach. 'Boot
in airplane mode' option would be useful, and 'disallow data' mode
similar to airplane mode toggle should really be there.)

Bad enough in home network, data usage is really killer in
roaming. So... that's why android has 'data roaming' option. It is
default on, bad! What's worse -- the option does not really work;
apparently it is racy enough that you still get roaming charges when
crossing borders.

Andy Grover: Yup, Exadata uses OFED

Tue, 26 Jan 2010 02:41:52 +0000

I checked out this video today, with OF's Executive Director, Bill Boas, speaking to the Lustre Users Group 2009.

He mentions RDS favorably a number of times. One thing that he would probably also like me to mention is that yes, Oracle's Exadata product (you know, the one with all the ads in The Economist, The Week, and The WSJ) is based on InfiniBand and OpenFabrics' software, OFED. Perhaps OFED has succeeded in its mission so well, and become so ubiquitous, that it may have become one of those parts of your system that "just works", and gets no love.

I think it would be cool for Larry Ellison to name-check OFED, IB, and RDS as ingredients in the Exadata special sauce... but I don't know if he or the ads even mention it runs Linux? Or GNU software? Or OEL's lineage from code by a certain North American Enterprise Linux vendor? I'm pretty sure he does mention the database software, though. :)

So, what can you do? People working on HPC and high-speed networking already know who OpenFabrics is, or at least about OFED. Even as converged Ethernet and IBoE bring RDMA to a wider audience, I have my doubts whether OFED will become a well-known name. It may simply have to count being relied-upon by a growing, happy userbase as its success.

Evgeniy Polyakov: Fixed comments from anonymous users

Sun, 24 Jan 2010 20:32:43 +0000

Drupal updates did not sync up with module changes, so captcha was screwed a bit.

Comments work again now.

Jaya Kumar: Chocolate in India

Sun, 24 Jan 2010 16:45:35 +0000

The stuff sold by Cadbury, Hershey, Nestle and probably all those other big multinational corporations is not chocolate. It is confectionery: sugar, hydrogenated palm oil (which by the way is the most disgusting lipid one could ever use) and the minimum amount of cocoa solids to make it look like chocolate. It is disgusting that I paid for and ate that stuff for so long and thought it was chocolate. If you want real chocolate, you'll have to talk to actual cocoa farmers in your area. Shake the hand that feeds you.

Evgeniy Polyakov: Improving morphological analyzer and sentence generator

Sun, 24 Jan 2010 03:40:30 +0000

(сверху впечатлению заикалась дотация , вокруг видению она выныривала)

Since russian has noticebly more complex morphological structure than english, and I basically do not know english good enough (most of my blog readers already noticed that :), I work with my native language, although derived logic could be applied for other language analysis too.

First, automatic grammatics generation is not very complex task, but since every word can have multiple meanings (like the same word can be a verb and noun in multiple forms with different cases and so on), number of grammatics automatically derived from given sentence rarely equals to just one and tends to explode with large numebr of words in the input sentences.

So there should be a way to eliminate some of them from the initial set. One of the factors, which can allow to drop impossible combiations of the word forms is relations between some common word forms, like closest adjective and noun, which should have the same case, or some prepositions which are only applicable to noun in selected case. I'm pretty sure that number of such rules is quite big, but getting that my last russian language lesson was in school kind of 15 or so years ago, it is a bit hard to summarize how it should look like.

I loosely implemented just couple of rules, and result looks noticebly more correct now. From grammatical point of view of course - currently I do not aim at generating content with some correct meaning.

(мы начинаем спокойный видеоряд о кости)

Another issue I was stumbled upon is actually wrong morphological information stored in database and dictionaries. Like verb voice or noun 'animation' i.e. check whether it is related to alive subject or not.

System can derive active voice, but dictionary will suggest passive voice verb for the substitution, and phrase will explode reader's brain first by the fact, that it is a grammatical crap way before it will the head with its meaning. So I added couple of checks (applied to russian language only of course), and results improved noticebly.

Another big problem I have right now is preposition logic. Or actually its absence - in russian each preposition can be related to only limited set of noun forms, and to date I did not find such information structured into the form, suitable for the database. So I need to manually write such information for about several dozens of such 'small words', which I'm a bit lazy to do.

That's why automatically derived grammatics which contain plain prepositions without relation to the appropriate nouns usually produce rather ugly random sentences.

To play with the system I implemented ability to generate content from manually created grammatics. Usually it differs from automatically derived ones only a little bit.

(к решению вернулась правда , сзади сомнению она пожимала)

To date I consider this part of the AI task as ready. Next one is long-term memory and fact extraction.

Basic idea is to create a system which will be able to memorize not only document content, but also relations between separate word forms and phrases. Such memory will allow to extract knowledge from the input data according to already learned information.
Thus we will be able to select words not randomly like now, but according to system's memory, so that selected set of words will be intelligent and related to some of its internal previously learned facts.

And it has to be automatic. The same way system is able to automatically derive grammatics from the random sentences according to some rules, it should extract possibly hidden relations between terms in the input data.

Its time to have some serious thinking on this problem...

Matt Domsch: New uefivars project leverages 9-year-old efibootmgr work

Sun, 24 Jan 2010 02:24:31 +0000

Finnbarr P. Murphy (fpmurphy) posted on his blog yesterday about his new project, uefivars, to retrieve and display information about UEFI variables. UEFI is the new firmware standard, replacing legacy BIOS over time, which is present on Dell 11G PowerEdge servers today. fpmurphy’s work is based largely on my own efibootmgr project which I started back in 2001 when first working on EFI for the Itanium processor. I’m glad to see renewed interest in this work as more people get exposed to UEFI on new systems. Perhaps it’s time, 9 years later, for bits of efibootmgr to turn into a library for use by applications like uefivars.

Evgeniy Polyakov: Living outside of civilization

Sun, 24 Jan 2010 00:20:16 +0000

Tried to order Kindle DX from Amazon - fuck me, it is only available in US, although its description page clearly says it can be shipped outside (including Russia in menu):

Good news. Kindle DX can now be shipped to customers outside the U.S.

Apparently our dollars differ from ones in US - Kindle can not be ordered.

Amazon has a long negative story of relations with orders from Russia - it even did not want to allow me to order even a bunch of trumpet books, although could deliver them to much longer distances like New Zealand. Recently (well last year iirc) it changed delivery method and dropped air mail support completely. Fortunately UK Amazon still allows this.

And even eBay.com (there is only one Kindle DX seller) does not want to deliver outide of US.

I do not want to read books from notebook screen, although i do not have any negative feeling, but it is just unconvenient. And do not want smallish Sony PRS (even its 900 model) - 9.7" is a good fitting size for reading, close to usual book sizes :). And frequently waiting for paper copy is way too long, also frequently there are no needed books at all.

Crap.

Rik van Riel: How to build a slingshot antenna launcher

Sat, 23 Jan 2010 20:27:51 +0000

Slingshot antenna launcher Slingshot antenna launchers are used to launch a line over a tree branch, so antennas can be suspended from high up trees. They can be purchased from various sources. However, I like home brewing so I thought it would be fun to build one from scratch. In order to create a nice how-to document for hams on a budget, I built my slingshot antenna launcher with only the cheapest components - total price just under $20. The total assembly time was about half an hour. The end result is satisfactory, with the fishing weight (and line) nicely clearing even the tallest trees in my yard.

Dave Airlie: LCA 2010 talk

Fri, 22 Jan 2010 03:18:39 +0000

So I originally was going to attend LCA 2010 for the week, but real life interjected and I couldn't abandon family commitments for that long, so I ended up doing a crazy cross-Tasman dash. As well as the change in flights, Isabel came down with a virus and Gia also got it, I think I got a milder version of it, but they both seem alright by the time I left but I had little sleep the previous two nights.

So I flew on Wednesday morning, got in Wed afternoon, met up with ppl, had a couple of beers, wrote slides, slept, finished slides, went to Thur morning, drugged myself up on Nurofen Plus to combat viral effects, gave my talk, went to see ajax talk, went to professional network dinner, went for beers with ajax + benh (listening to an American and a Frenchman speaking about wine while listening to drum n bass in a Wellington pub was a bit wierd). Decided to push on through, so got back to room at 3am or so, checked out/left for airport at 4:20am, flew at 6:20am, into BNE at 7am, home, bed, sleep for a few hours and mind Isabel for afternoon.

So my talk was "So you've put kernel graphics drivers in the kernel... what next? can I haz ponies?". My slide deck is off the 0-content style + lots of pictures of various ponies, which I've found, they'll be on the LCA site soon and I'll upload them when I plug that laptop in again.

(a) stop people reading ahead of your bullet points so they don't doze off while you are catching up
(b) gives them something to look at apart from me while they actually have to listen to me :-P

It seemed well received, the room was pretty packed out (ppl standing/sitting - LCA schedulers you listening?) and I don't think the sickness or lack of decent preparation made a big difference. I'd like to apologise for not even mentioning SGX/poulsbo, I'm not sure how but it totally slipped my mind, but the situation hasn't really changed in terms of how screwed it is.

Evgeniy Polyakov: Suddenly: back to business

Fri, 22 Jan 2010 01:53:16 +0000

At took a bit of time to settle things down all over the place, so I can continue with the things I used to work with. Now huge post about things happened :)

First, recent development things. Altough a bit confused, but still.
Last several days I worked on morphological analysis and mainly on automatic grammatics extraction from texts and text generation based on them. I did not care much about performance, multithreading and the like, but instead concentrated on the idea. So I selected Lisp for prototyping, to date I do not know whether it will have more serious usage.

So, task of grammatics extraction is rather trivial when you have a database of morphological data. Namely I wanted to parse a sentence and get a knowledge about all words and their morphological forms, for example for nouns it could be at least cases and number and so on. Having a huge database for all possible word forms and all words in russian language (I selected it since I know it well and it has a quite reach morphological structure, but technique is applicable to any language of course) is not a feasible task.

So I developed a lemmer/stemmer, which has a limited database (I use about 10k words), obtained by aot.ru output parsing, although originally I wanted to DoS wikipedia with my requests. Getting that there are no on-line dictionaries (for russian at least) with tagged morphological data and structured output, I had to write regexp parsers to get that information from HTML pages. Wikipedia has noticebly worse and noisy output compared to aot.ru though.

To select unique words I downloaded one rather big text (Pelevin's "Generation P") and used Levenstein-Damerau distance to select words, which are rather far from each other (I used 0.25 normalized distance as a threshold, i.e. words are considered different in this metric, when roughly more than 1/4 of the letters are different). Then I received morphological data from aot.ru and stored it into local structures.

My lemmer allows not only to obtain information about words it has stored in the database, but also guess it for the words, which it does not know about based on matched word endings.
Grammatics generation becomes rather trivial - iterate over all words in the sentence and write realted morphological information without word form itself.

Second part - sentence generation, is rather simple, when morphological data is well structured. Namely I use derived grammatics and select random words which have to match morphological data for grammatics. So it looks like a grammatically valid sentence, but it contains nonsence of course, since words are not related to each other and do not follow some 'meaning' of the phrase. Also, there is no information for prepositions, which concatenate different forms, and some of them (in russian at least) can not be used with some forms and vice versa.
Another problematic part is multiple meaning of some words. It is even possible that the same word will not only have multiple noun cases with the same wordform, but simultaneously will be a verb in some form. And I did not yet develop a sentence analyzer, which will drop grammatics which do not match russian sentence rules, like must-have verb and/or noun forms presentd in the sentence and so on.

Below is several examples of how it works (in russian). First sentence is origianal text.

(какая-нибудь простая грамматика с текстом .)
(сдайся эротическая девушка меж мотоциклом .)
(жмурься рыжеволосая межа вне логиком .)
(вдумайся техническая папироса от борисовичем .)
(ответь тренажерная сидоровна изо симптомом .)
(просыпайся непосредственная харольдовна безо фактом .)
(тревожь забытая судьба между бомжом .)

It is possible to generate grammatics by hand and select words around some meaning, but such manual interference is not what I want. Next step is sentence processing rules described above. It is rather simple task, but it is a must-have feature for text generation.

Noticebly more complex is knowledge extraction problem and long-term memory, which in turn will allow to select words tied to each other based on previous experience. Using such technology system will be able to understand meaning of the data in terms of related words and generate reply based on its knowledge of their relations.

This is a task for some future though...
--------------------------------------------------------------------------------------------------------------

Another lexical problem I worked on is language detection. Common algorithms use N-M gramms, where N is number of letters and M is number of subsequent words, such NM-gramms are used to calculate conditional probability of the next characters based on probability of the previous ones in selected NM-gramm, so it is possible to detect languages, when system was trained and language-specific NM-gramms have been selected.

I think that brain works quite differently and does not calculate any kind of NM-gramms at all, but instead use highly parallel fuzzy words search in the memory. I did not yet develop a fast fuzzy searching except calculating Levenstein-Damerau distance against every word in the dictionary for every input word, which is rather costly task. So this is another interesting task to think about.

So I decided to switch to more simple matching - cut off endings and match against learned words. Thus I implemented in LISP a simple RADIX-based algorithm, where downloaded documents are parsed and reversed words (optionally without one or two last letters - kind of endings) are inserted into RADIX tree. Checked words are reversed and looked in this 'dictionary' optionally without one or two letters from its end - I kind of cut off the ending. When lookup returns a match system considers given word as being part of the language it refers to. Of course it is possible that the same word will be present in multiple languages (especially when training corpus contained words from different languages like what I used: raw wikipedia pages), so to determine document language we should check all words and calculate how many of them matched against every known to the system language. It is still possible to check single words of course.

This simple technique (less than 200 lines in LISP not counting RADIX tree implementation) behaves surprisingly good in the test case I ran. Namely I selected 3 big articles (several thousands of words) from wikipedia in english, turkey, ukrainian and russian languages, and then got wikipedia texts (not used in learning of course) and text matched its real language with probability (just a division of the matched words (in the above sence) to all words in the document) noticebly higher than for any other languages. All texts were downloaded automatically and CL-PPCRE based parser removed all tags, numbers and non-letter characters.

Here is an example output for english learning process:

$ ./get-page.lisp :radix-root-path=radix.obj.en :learn-url-path=/tmp/learn.en.txt \
  :output-dir=learn-data :check-url-path=/tmp/check.txt

url: http://en.wikipedia.org/wiki/Bahá'í_Faith, learned words (including dublicates): 9197
url: http://en.wikipedia.org/wiki/Carabane, learned words (including dublicates): 11072
url: http://en.wikipedia.org/wiki/Is_This_It, learned words (including dublicates): 6469

url statistics: http://tr.wikipedia.org/wiki/Uğultulu_Tepeler_(roman)
total match: 35 %

url statistics: http://en.wikipedia.org/wiki/Wuthering_Heights
total match: 60 %

url statistics: http://ru.wikipedia.org/wiki/Заглавная_страница
total match: 7 %

url statistics: http://uk.wikipedia.org/wiki/Головна_сторінка
total match: 6 %

As we see, it detected english language with 2 times higher probability. I skipped other tests (turkey, ukrainian and russian), but they show similar numbers.

Here is example for the most popular russian livejournal blogger Tema Lebedev LJ page and its profile:

TR: tema: 5%  profile: 21%
UA: tema: 30%  profile: 15%
EN: tema: 8%  profile: 27%
RU: tema: 47%  profile: 20%

Profile contains rather large number of english usernames words, so result is quite correct.
Percentage is far from 100% since small number of words were learned, I skipped prepositions and other small words and there are non-russian words there of course.

Not sure whether this is a very useful project, but I did not regret a day spent on thinking and development.

------------------------------------
That's it for noticeble development issues. Now lets more to life happenings.

I made an eye correction operation and can look at women without glasses now. I can also swim, play tennis and football and overall behave like a normal person. It is fucking cool feeling!

Operation itself is painless, but was rather complex from psycological point of view, at least for me. Especially things like vacuum cup on eye, can-opener-like part of the cornea cut and the like. But overall it was not something you should be afraid of.

I absolutely do not regret I did it.

I also filled an action at law against development company, which build a house I bought appartments in, to recognize me as a real owner of the appartments. It will take a while to settle though, I think a month or two.

Ugh, and I play trumpet. I do play it, and it sounds quite good, when I'm in a good mood and can play loudly on my Yamaha. I still can not improvise out of the head, but I usually have no troubles playing some melody after I learnt it. Learning can take a while if I did not hear melody before. Piano playing is rather stuck - I prefer to learn melody first on piano, but I have real troubles playing by both hands, even when left one part is really trivial like 2-3 notes. So I usually learn melody part only before trying it on trumpet :)

Heh, I'm back to business :)
Stay tuned!

Ted Tso: The Transitive Grace Period Public Licence: good ideas come around…

Wed, 20 Jan 2010 23:23:59 +0000

I recently came across the Transitive Grace Period Public License (alternate link) by Zooko Wilcox-O’Hearn. I fonud it interesting because it’s very similar — almost identical — to something I had first starting floating about ten years ago. I called it the Temporary Proprietary License (TPL). I’m sure this is a case of “great minds think alike”. One things that I like about my write up is that I gave some of the rationale behind why this approach is a fruitful one:

A while ago, I was talking to Jim Gettys at the IETF meeting in Orlando, and the subject of software licensing issues came up, and he had a very interesting perspective to share about the X Consortium License, and what he viewed as bugs in the GPL.

His basic observation was this: Many companies made various improvements to the X code, which they would keep as proprietary and give them a temporary edge in the marketplace. However, since the X code base was continually evolving, over time it became less attractive to maintain these code forks, since it would mean that they would have to be continually merging their enhancements into the evolving code base. Also, in general, the advantage in having the proprietary new feature or speed enhancement typically degraded over time. After all, most companies are quite happy if it takes 18-24 months for their competitor to match a feature in their release.

So sometime later, the companies would very often donate their previously proprietary enhancement to the X consortium, which would then fold it into the public release of X. Jim Gettys’ complaint about the GPL was that by removing this ability for companies to recoup the investment needed to make major developmental improvements to Open Source code bases, companies might not have the incentive do this type of infrastructural improvements to GPL’ed projects.

Upon reflection, I think this is a very valid model. When Open Vision distributed the Kerberos Administration daemon to MIT, they wanted an 18 month sunset clause in the license which would prevent commercial competitors from turning around and bidding their work against them. My contract with Powerquest for doing the ext2 partition resizer had a similar clause which kept the resizing code proprietary until a suitable timeout period had occurred, at which point it would be released under the GPL. (Note that by releasing under the GPL, it prevents any of Partition Magic’s commercial competitors from including it in their proprietary products!)

For more, read the full proposal.

No related posts.

Pavel Machek: thinkpad 560X still useful these days

Wed, 20 Jan 2010 05:34:30 +0000

My brother started using it. I was afraid about bricking it, but grub install went well, and now it is ticking on 2.6.33-rc4 kernel. Update may have broken audio, but I expected way bigger problems -- that machine is from 1995 or so. And yes, it has a better keyboard then X60...

Paul E. Mc Kenney: Parallel Programming: Terrible Tooling

Wed, 20 Jan 2010 02:00:13 +0000

In a previous life, I worked on a UNIX kernel on a team of a few tens of developers. This smallish team size resulted in a very smallish set of development tools. To see why, imagine a tool that required one developer one year to create, and that would provide a 1% improvement in everyone's productivity. Given 50 developers, it would take two years to recoup the time spent developing this tool. In contrast, consider the Linux kernel, which usually has thousands of developers. Assuming 1,000 developers, it would take less than six weeks to recoup the time spent developing this tool.

This example does much to explain the large quantity of tooling available for the Linux kernel. For example, the UNIX kernel I worked on did not have anything like lockdep, powertop, latencytop, or perf for in-kernel use, nor anything like valgrind or perf for user-mode use. Given the relatively small number of developers on that project, it almost always made more sense to find and fix problems manually.

It is important to note that this effect is due to the number of Linux-kernel developers, not to the FOSS nature of the Linux kernel — except insofar as the FOSS nature of the Linux kernel has enabled it to accumulate a large number of developers.

So what does this have to do with the perceived difficulty of parallel programming?

Matthew Garrett: Nook GPL update

Mon, 18 Jan 2010 23:00:00 +0000

"We are currently working on getting the code shipped out, we have your shipping address on file and will ship the code out as soon as it is ready, this will be by the end of this month."

2/10, must try harder. Even ignoring the fact that it's taking the best part of two months to get from shipping binaries to shipping source, it'll be interesting to see which version of the firmware the source release corresponds to given that the 1.1.0 update included a new kernel.

Speaking of firmware updates - as shipped, the nook was a disaster. The firmware update improved things greatly, but I'm still stuck with cases where books render with the last couple of words in a page left out, and the order of books in the local content library is now based on last modification time rather than, say, author name. The continuing lack of any way to implement catagorisation or provide worthwhile metadata means that it's frustrating having to remember which order the books in a series go in. The fact that there's even a distinction between content you've obtained from B&N and content you've obtained from elsewhere is mindboggling. You need to explicitly switch between the two, which means that collections are broken up if you've had the audacity to buy some books from B&N and some books from elsewhere.

Having now spent a while travelling with the nook, it's clear that it's still not an especially effective example of its type. If my Sony hadn't suffered an unfortunate accident, I'd be using it instead. The nook's hardware is significantly better and as a platform it's got much more potential, but it just doesn't do a terribly good job at its core role - reading books. It'll get better with time, but if you're in the market for something that actually works today then go elsewhere.

There's other little things that are upsetting at a conceptual level. The 1.0.0 firmware will flash unsigned firmware, which is useful[1] but also an obvious wtf. The filesystem is full of scripts that are world writable and executed as root. There's a libGLES, but the coverflow implementation appears to be handled entirely in software. There's a general feeling that the platform's been put together by people who either don't know what they're doing, don't care about doing it properly or were working under sufficient time constraints that getting it right wasn't an option. I'm really not impressed.

[1] Especially given that the 1.1.0 firmware will happily flash the 1.0.0 firmware...

Greg Kroah-Hartman: Stable kernel tree status, January 18, 2010

Mon, 18 Jan 2010 19:11:00 +0000

Here's the state of the -stable kernel trees, as of January 18, 2010.

2.6.27-stable

The 2.6.27-stable kernel tree is still living on, as a "long-term" stable release. But, I do have to warn users of this tree, the older it gets, the less viable it becomes. Not all bugfixes are being backported to this kernel version due to massive code changes in the over 2 years since this kernel has been released. I am doing my best to backport fixes that I become aware of, and I encourage anyone who does fix any types of bugs in the main kernel tree to let me know if the change should be applied to this older kernel version.

I'll probably keep maintaining it for at least 6-8 more months, but after that, I can not guarantee it's viability. Note, one other developer has volunteered to pick up the tree after I am finished with it, but I can not speak for him at this time.

2.6.31-stable

Today the last 2.6.31-stable kernel was released, all users of this kernel series are strongly encouraged to switch to the 2.6.32 kernel series, as there will not be any more updates for this branch in the future.

2.6.32-stable

I'd like to announce that the 2.6.32-stable tree is also going to be maintained as a "long-term" stable release, living for 2-3 years, like the 2.6.27 kernel is. This is because a number (i.e. more than 2) Linux distributions are basing their "enterprise" releases on this kernel version, and it will make their lives easier if I keep it alive.

Note, the viability of me keeping this tree alive for such a length of time relies on the developers working for those distros to keep me informed of patches that need to be backported and applied to it. Without their help, I will have no problem in stopping the maintenance of the tree.

Submitting patches for stable trees

Again, the easiest way to get your patch into a -stable tree is to merely add the line:

Cc: stable <stable@kernel.org>

to the Signed-off-by: area of your patch. When the patch goes into Linus's tree, it will be automatically sent to the stable address, and I will know to apply it to the trees. If I have any problem applying it at that time, I will email the author and reviewers of the patch about it.

If you forgot to add this line to the patch, or you have found a patch written by someone else that you wish to have applied to the stable trees, email the git commit id of the patch as it shows up in Linus's tree to the stable@kernel.org email address. Any stable correspondence sent to my personal accounts has the chance of being lost in the shuffle, so please try to not do that.

If a patch needs to be backported to one of the stable trees because it does not apply directly, please send the backported patch, along with the git commit id of the original patch, to the stable@kernel.org address, with a description of which kernel tree it should be applied to.

If anyone has any other questions about stable releases, please let me know.

Kernel Podcast: 2010/01/17 Linux Kernel Podcast

Mon, 18 Jan 2010 05:25:40 +0000

Audio: http://media.libsyn.com/media/jcm/linux_kernel_podcast_20100117.mp3

For the weekend of January 17th, 2010, I’m Jon Masters with a summary of the past week’s (and some holiday) LKML traffic.

Yep, we’re back, and ready for 2010.

In this weekend’s issue: A new format (version 3.0), async page faults, async suspend and resume, feature removal, mod_timer_msec, page allocator, raw_spinlocks, and Linus’ birthday.

A new format. Ever since I started doing these podcasts last May, it’s been a constant struggle to find the time each day to prepare and produce a show. It often takes upwards of an hour to prepare the material and produce the show. I have asked for volunteers, but clearly this will remain a one man effort for the moment. I’ve been sick for the past week (just working on day job stuff) and had a lot of hassles over the holidays that prevented me from pushing updates. It got me thinking about how I can make this show better – easier to produce, and more reliable. I have decided that in 2010 I will try to produce a daily show, but I will only “commit” to a weekly show covering the events of the past week (which will be longer format and produced on Sundays). Days that don’t have a show will be rolled into the next rather than producing multiple episodes that “catchup” the record (so I won’t bother covering a linux-next tree that’s already a week out of date). I would love to do a show every single day, but I need volunteers to help make that a reality (I need time to write on other projects and read a crazy backlog of books). What really matters is that there’s some kind of reference tracking what happened each day. So you’ll just have to make do with some episodes covering multiple days at a go. Unless you want to volunteer with production…

Async page fault. There was some discussion concerning asynchronous page faults. Hirouki Kamezawa originally posted a patch over the holidays that was now in its third iteration. “asynchronous page fault” support is intended to aid highly threaded applications deal with page faults without needing to contend for the per-process (containing multiple threads) mm->mmap_sem. This is achieved through a certain amount of speculative vma handling (knowing that it might be modified or unmapped without the protection of a lock) and falling back to taking the lock if the VMA RB tree is modified while walking it. There were some issues surrounding atomicity with the patches that Hiroyuki posted, while others (including benh) were concerned that some arches make various assumptions about mmap_sem being held. Peter Zijlstra followed up several days later with version 3 of the “speculative pagefault” patch series, making use of RCU for freeing vm_area_struct (VMAs). On yet another VM note, Gleb Natapov posted an updated round of “Add host swap event notifications for PV guest” patches (that allow guests to know about host page faults), and Mathieu Desnoyers posted version 2 of a patch series implementating a new system call that is named sys_membarrier (forces a process-wide barrier).

Async suspend and resume. Rafael J. Wysocki posted some updated benchmarks for his asynchrnous suspend and resume patches, for which he replaced the disk containing rotational media with a solid state device, and enabled KMS. His results suggest that asynchronous resume is 50% faster at suspending (300-350 ms vs. 600-700 ms) and much faster at resuming (1.1-1.2 seconds vs. 4 seconds). That might given Apple a run for their money.

Feature removal. Robert P. J. Day asked about Documentation/feature-removal.txt and whether it was going to be updated (since some items referred to 2005) soon.

mod_timer_msec. Arjan van de Ven posted a patch (intended for drivers) implementating mod_timer_msec, which can be used to set or change a timer for a relative amount of milliseconds. This allows Arjan (and also others too) to remove the need for certain drivers to work directly with jiffies and HZ.

Page allocator. Mel Gorman posted a patch from Corrado Zoccolo that divides freed pages into two classifications – those that have a high probability of being merged with their next-highest buddy, and those that do not. Those pages not likely to be merged in the near future are preferred on the freelist to those that might be mergeable in order to make higher order allocations available using the newly merged buddy pages, and reduce fragmentation.

raw_spinlocks. John Kacur, noting that Thomas Gleixner’s recent work had freed up the “raw_spinlock” name within the kernel for re-use, posted a number of patches converting existing spinlocks over to raw_spinlocks. This is required in the RT tree, wherein all spinlocks are be default converted to a sleeping variety, but for which certain locks must explicitly remain non-sleeping. Locks that must always be real true spinlocks should use raw_spinlock.

Finally today, Linus’ birthday. It was noted by a few posters that Linus’ birthday is December 28th. Linux Journal had an article, and as these things are want to do, Linus’ reply took the conversation off to a tangent about 387 co-processors, to which Avi Kivity added some remarks on the original design.

In today’s miscellaneous items: a new version of kFIFO (Stefani Seibold – and an even newer version on January 14th that is re-implemented and apparently does not require any changes to existing code users of the old kFIFO API), an XFS status update from Christoph Hellwig (who notes that ongoing work includes support for new event tracing code and mkfs.xfs default support for “lazy superblock counters”), a refresh of the jump labeling patches (v4) from Jason Baron, a patch from Dave Jones removing his name from checkpatch.pl (so people stop asking him about it), initial PCI support for Xilinx Microblaze (and a discussion about generic PCI support file locations), a modpost patch implementing support for ELF objects with greater than 64K sections – recall that this has to be handled specially in ELF – for use when compiling a kernel with -ffunction-sections and having e.g. an allysconfig, and a mini-rant from Stefan Richter concerning “Changelog quality” that is worth reading. Finally, Dan williams (the Intel one) is taking over maintainership of I/OAT.

In today’s announcements: LTTng version 0.186o for 2.6.32-rc8 and Userspace RCU 0.3.3 were both released by Mathieu Desnoyers on January 4th. He posted an updated Userspace RCU 0.3.4 on January 10th, that had some additional fixes.

LTP. Subrata Modak announced the December 2009 Linux Test Project release. It included a number of build system fixes for various distributions.

rt-tests version 0.60. Clark Williams announced the latest version of rt-tests on December 29th. It includes a new ‘pip’ (Priority Inheritance stress test) from John Kacur and adds an unbuffered output option to cyclictest for those parsing the output at runtime. The source is available on git.kernel.org.

smatch 1.54. Dan Carpenter announced version 1.54 of his “smatch” static source code checker tool for C programs such as the Linux kernel. His intention is for “smatch” to become “a smarter version of checkpatch.pl”. It includes cool things like a check for DMA use on the kernel stack.

SystemTap version 1.1. David Smith announced the 1.1 release of SystemTap. This includes better support for gcc 4.5 “richer” debuginfo, amongst other fixes. It’s available from http://sourceware.org/systemtap. Frank Ch. Eigler followed up to note that this release also includes a fix for CVE-2009-4273.

util-linux-ng v2.17. Karel Zak announced the latest 2.17 release of util-linux-ng, which contains a number of new features (an fallocate command, unshare command, wipefs command), and updates to libblkid, blockdev, fdisk, and other fixes.

The latest kernel release is 2.6.33-rc4. There was an rc3 too, but that was pretty small “due to the holidays”. 2.6.33-rc4 had few updates, but 40% of those were in DRM (nouveau and radeon in the staging tree, i915 updated too), and Linus called that out as being “unusual”. There were also some bootloader issues on non-x86 systems, but Linus figured “we’ll sort it out”.

Andrew Morton posted an mm-of-the-moment (mmotm) for 2010-01-15-15-34.

Greg Kroah-Hartman posted some stable review patches for 2.6.32.4, 2.6.31.12 (there had previously been a 2.6.31.10 that was replaced with a build fix in 2.6.31.11), and 2.6.27.44.

Stephen Rothwell posted a linux-next tree for January 14th (and announced that there would be none for January 15th). Since Wednesday, there was a new mtd-current tree, Linus’ tree still had a build failure for which he reverted a comment, the net-current tree lost its conflict, and the tip-tree gained a conflict against the kgdb tree. The total sub-tree count increased to 156.

That’s a summary of today’s Linux Kernel Mailing List traffic, for further information visit www.kernel.org. I’m Jon Masters.

Pavel Machek: Thinkpads with crappy keyboards

Sat, 16 Jan 2010 09:26:23 +0000

Slashdot had a story about thinkpads with chicklet keyboards and FX keys only available as Fn-FX. Sad :-(.

Rik van Riel: A multi-band HF loop antenna

Sat, 16 Jan 2010 00:49:23 +0000

Amateur radio has 10 HF (shortwave) bands, from 1.8 MHz (160 meters) through 28 MHz (10 meters). The different bands are great for talking to amateur radio operators (hams) at different distances, different times of the day, different seasons and different times during the solar cycle. Only one problem: most hams do not have space in their yard for 10 different antennas. Fortunately it is possible to build one antenna that can be used effectively on most of the bands, for example this 80 meter loop antenna.

Pete Zaitcev: The evils of event-driven programming in CLD clients

Fri, 15 Jan 2010 19:50:54 +0000

I wanted to fix a small bug in tabled where the server conflicted with its own lock and hung if restarted too quickly. But before I was able to write the patch, I had to fill a Letter format piece of paper with pencil scratches. And that's for the code that a) I wrote myself and b) is only 1000 lines long. It's like the return to 1972 kind of programming.

The root of the evil here is that the event-driven API of CLD forced a regression to the programming with GOTO, because the mechanics of CLD consumed what C programmers use for abstraction otherwise: functions. And CLD is event-driven because it processes the protocol in a library, but someone else decided that spawning a processing thread would be too bourgeouse.

Until today I suffered this silently, but enough is enough. I will not rest until a sane API for CLD is established.

Pavel Machek: podcasts worth listening

Fri, 15 Jan 2010 19:42:12 +0000

Ok, so 'podcasts you love, from people you trust' tagline should have
warned me. 'FLOSS weekly' from same twit network is a bit full of
advertising, but otherwise ok. 'Security Now'... is quite another
thing. They don't seem to know what they are talking about, and use
even more advertising. 'Spin Write' they are pushing all the time seems
suspiciously like snake oil. But it gets worse. 'SameOrigin troubles'
episode, they kind-of defend Adobe for grave security hole. Flash
object with .gif extension and image/gif content type still gets
executed, and can steal your data. Adobe claims that can't be
fixed. Well, I always knew Flash is malware but, ...

Anyway, worst stuff came with 'elephant in the browser'
episode... They are pushing some remote aceess 'solution': "you just
mail the link to executable to them, they click on it, download .exe,
run it and you fix their computer, all completely secure". Hmm, sounds
so. Not.

So there's 'FLOSS weekly' podcast, then 'lkml summary podcast'. I listened to
'all things considered' music podcast, but I can't find it now, or any
other reasonable music podcast for that matter. So... whats worth
listening?

Pavel Machek: Android market sucks, part II.

Fri, 15 Jan 2010 19:18:12 +0000

Half applications have totally strange requirements -- like 'control
vibrator' and 'full internet access'. Do they phone home? I guess so.

Licenses are only available only after you have already wasted your
time/bandwidth downloading the crap. Like wikitude: it presents you
with welcome screen with one button 'yes I agree to eula'... (I'd welcome court decision saying that license has to be available before download, or is unenforcable. After all, the only copying you really do is from server...)

Pavel Machek: Android mystery

Fri, 15 Jan 2010 18:57:03 +0000

So I had android eat battery like crazy, and when I tethered PC, I was
getting like 10KB/sec (instead of 80KB/sec). I tried searching who is
using network, only to be told that it is 'Media' :-(.

I tried installing network monitor widgets, but it turns out there's
no really good monitor available, and some of those want to use
network themselves...

It seems acast was responsible -- it tried to download video podcast
in background. To add insult to the injury, the video was not even
playable on the android...

Pete Zaitcev: That Google

Thu, 14 Jan 2010 20:44:08 +0000

Regarding the Google's decision not to censor their search results anymore and Ted's praise of it, it strikes me as notable how they thought nothing of censoring until the agents of Chinese government directly attacked them (among other American companies and organizations). That sort of deducts the points Google receives for this decision. Also, since it was shown they were willing to censor results until something drastic happens to them, what else do you think they censor, even now? Tried to enter "Obama" or "Mormons" or "Net Neutrality" into Google lately?

Kernel Podcast: Where’s the podcast? (sick)

Thu, 14 Jan 2010 20:37:03 +0000

Folks,

I’m really not doing so well this week. Sure, I’m working, but I’m also having to sleep the moment I get home in the evenings. I expect to get caught up over the weekend. But there probably won’t be podcasts for last week, only this week. Sorry about that.

Jon.

Pete Zaitcev: yum blooper of the day

Thu, 14 Jan 2010 19:59:40 +0000

The "yum update" said:

kernel-PAE-devel-2.6.32.2-15.fc13.i686 is a duplicate of kernel-PAE-devel-2.6.31.9-174.fc12.i686

What does that even mean? How can different kernels be "duplicates"? Of course it's nothing that one little rpm -e cannot fix, but still.

Pete Zaitcev: Sun F5100

Thu, 14 Jan 2010 18:54:19 +0000

Seen at Axboe's:

Sun F5100 flash array [] packs 80 SLC flash modules (or FMods, in Sun speak) into a 1U box. Each module is essentially identical to the ones on the F20 card, which I already have used a lot for testing. It's 24GB in size, has a sequential throughput of approximately 200MB/sec reads and 150MB/sec writes, or 25K/15K IOPS. That's pretty fast by itself, multiply that by 80 and you are now burning holes in the fabric of time - 1.6 million IOPS, or well over 10GB/sec throughput.

This box has John Corbin's feel to it. John understood why MB/s were not everything and why IOPS mattered. I wonder if he was actually involved. The litmus test would be to verify if something clever is done to support 4KB blocks (flash usually has 64KB blocks and its small-block write performance is suboptimal; striping when chasing MB/s can easily make it worse).

UPDATE: LinkedIn says John is at NetApp now, so someone else made F5100.

Arnaldo Carvalho de Melo: Cross platform perf.data analysis support

Thu, 14 Jan 2010 14:55:38 +0000

There are still some problems related to loading vmlinux files, but those are unrelated to the feature implemented in this patch, so will get fixed in the next patches, but here are some results: 1. collect perf.data file on a Fedora 12 machine, x86_64, 64-bit userland 2. transfer it to a Debian Testing machine, PARISC64, 32-bit userland acme@parisc:~/git/linux-2.6-tip$ perf buildid-list [...]

Jens Axboe: New storage toy

Thu, 14 Jan 2010 08:57:58 +0000

Since I do a lot of work on the IO side of the Linux kernel, I have an assortment of various types of disk drives and storage. I've been lucky enough to receive various types of SSD drives, which enabled me to research and test new ideas for these types of devices. I love doing performance work and squeezing the most out of what you have, and the performance characteristics of SSD devices rekindled that interest.

Earlier this week I received a new storage toy that'll take this to a whole new level - a Sun F5100 flash array. This beast packs 80 SLC flash modules (or FMods, in Sun speak) into a 1U box. Each module is essentially identical to the ones on the F20 card, which I already have used a lot for testing. It's 24GB in size, has a sequential throughput of approximately 200MB/sec reads and 150MB/sec writes, or 25K/15K IOPS. That's pretty fast by itself, multiply that by 80 and you are now burning holes in the fabric of time - 1.6 million IOPS, or well over 10GB/sec throughput.

Internally the modules are connected through SAS expanders, of which there are 4 on the back, each with 4 ports. Ideally you'd connect 16 controllers to reach the full potential of the box, setting up zoning to drive 5 modules per port. The box essentially has its own UPS with 4 energy modules at the front, so you never need to use barriers on this device. And it's running surprisingly quiet, which is always a big plus for things you have a few meters away from the desk :-)

Currently the box has 40 modules installed, 40 more are on the way. I have just a single SAS controller, more are on their way as well. Once the additional controllers arrive, I'll be putting it through the paces and posting some numbers. I have some soft multi-queue block IO support in progress that has been waiting for a potent device to be unleashed. This will be interesting...

Pavel Machek: Hackable microwave oven

Wed, 13 Jan 2010 21:00:32 +0000

After hackable phones and hackable watches, here comes... hackable microwave. Running android.

Actually... having cookbook on kitchen appliance kind of makes sense...