Kernel Planet

Dave Jones: Daily log June 17th 2013

Tue, 18 Jun 2013 04:27:16 +0000

Busy day. Hit a whole bunch of new bugs.

Daily log June 17th 2013 is a post from: codemonkey.org.uk

Dave Jones: Daily log June 14th 2013

Fri, 14 Jun 2013 20:10:38 +0000

Switched out a broken SSD in test machine for a hybrid drive. Hopefully it’ll last longer even if it is slightly slower.
Hit a trinity bug where a child process would sleep for a really long time, and the watchdog had already exited. Reworked things so the watchdog never leaves while there child processes still running so that it can SIGKILL ‘stuck’ children. Stupid bug..
Spent some time cleaning up the linked list uses in trinity. Not happy with the resulting patch so didn’t commit it. Maybe next week.
Short bi-weekly kernel meeting in #fedora-meeting. Nothing exciting.

Looked over my 3.10-rc5 outstanding issues. Some more patches got merged, so things are starting to look better, as long as I don’t find any new problems next week.

GPF in aesni_xts_crypt8. (fix)
top reports cpus maxed while idle (Fix is in linux-tip:timers/urgent).
XFS log recovery bug
Trinity triggered bugs:
- udpv6 oops (Been around for a long time, still no fix).
- Assorted RCU/NOHZ_FULL bugs.
  - rcu_read_lock() used illegally while idle!, and again. Possible fix is d6284099 in linux-rcu.git rcu/urgent
  - WARNING: at kernel/rcutree.c:388 rcu_eqs_enter+0x9b/0xb0()(potential fix),
  - Context tracking warning(fix).
- Assorted perf bugs.
  - perf function tracing uses rcu_lock()
  - perf_lock_task_context deadlock
- WARNING: at kernel/trace/ftrace.c:1605
T430s Lid events no longer put machine to sleep (actually an old bug since 3.9-rc1, haven’t had time to bisect)

Daily log June 14th 2013 is a post from: codemonkey.org.uk

Dave Jones: Weekly Fedora kernel bug statistics – June 14th 2013

Fri, 14 Jun 2013 16:06:43 +0000

	17	18	19	rawhide
Open:	244	424	134	72	(874)
Opened since 2013-06-07	2	25	12	7	(46)
Closed since 2013-06-07	10	9	3	1	(23)
Changed since 2013-06-07	16	39	32	13	(100)

Weekly Fedora kernel bug statistics – June 14th 2013 is a post from: codemonkey.org.uk

Matt Domsch: Fedora Project Board Town Hall Thursday 1900 UTC

Thu, 13 Jun 2013 15:38:12 +0000

I have the pleasure of moderating the Fedora Project Board Town Hall today, 1900 UTC, having served on the board for five years previously. Held on IRC, these Town Halls give project members a chance to ask questions directly of the five Board candidates, so that you can make a more informed decision when casting your vote. I hope you can join us.

Dave Jones: Daily log June 12th 2013

Thu, 13 Jun 2013 04:58:54 +0000

Did some more work on my test harness for setting up an assortment of disk configurations.
While testing it, hit a bug during RAID5 reconstruction. This was a pain to get a trace out of, because it hits a BUG_ON. When that happens usb-serial stops spitting out characters, which seems suboptimal, but might be tricky to fix. I ended up just converting it to a WARN_ON to get the trace. A patch fixing this has existed for almost a month. sigh. At least it got merged by the end of the day.

Daily log June 12th 2013 is a post from: codemonkey.org.uk

Dave Jones: Daily log June 11th 2013

Wed, 12 Jun 2013 04:03:05 +0000

Hit an AES bug that prevented my laptop booting 3.10-rc. Reported to the author of the only relevant change, and got a patch back quickly which solved it. If only all bugs had that kind of turnaround time.
Poked at my fedora kernel bug triage script. Realised that right now, it’s incomplete because python-bugzilla doesn’t support setting keywords, only querying. Someone with python-fu want to implement that (and component reassign) ?
Finally got fed up with all the reports from the soft lockup detector from users of vmware/kvm, and wrote a patch to automatically disable it there. It appears that if you create enough load in the host, guests don’t get scheduled within the time range necessary for the softlockup detector to keep ticking.

Daily log June 11th 2013 is a post from: codemonkey.org.uk

Andy Grover: New screencast: 10 New Features in LIO and targetcli

Tue, 11 Jun 2013 16:38:04 +0000

I posted a new screencast that talks about ten new ease-of-use features that are new in Fedora 18.

10 New Features in LIO and targetcli

Easier storage->ACL setup
Name shows up as LUN model name
Tags for initiator aliases and grouping
‘info’ command
IPv6 portal support
WWNs normalized
Only show HW fabrics that are present
10 previous configs saved
More info in summary
iSER support
Better sorting

Dave Jones: Daily log June 10th 2013

Tue, 11 Jun 2013 04:12:47 +0000

Usual Monday morning email catch-up.
Rebased my testing trees to rc5, restarted tests.
Watched the Apple keynote. Wondering how long before we get a “new macbook air doesn’t work on Linux” bug report.
More RCU bugs, which was actually a repeat of an -rc2 era bug. Hitting old bugs again is actually giving me some hope that there’s not anything too scary remaining in this code. Spent a while detangling all the different RCU/NOHZ bugs reported over the last few weeks.
While chasing that stuff down, hit a new ftrace bug.

My 3.10-rc5 outstanding issues.

top reports cpus maxed while idle (Fix is in linux-tip:timers/urgent).
XFS log recovery bug
Trinity triggered bugs:
- NETLINK mmap corruption bug. (Fix known, but not committed yet)
- udpv6 oops (Been around for a long time, still no fix).
- Assorted RCU/NOHZ_FULL bugs.
- Assorted perf bugs.
  - perf function tracing uses rcu_lock()
  - perf_lock_task_context deadlock
- WARNING: at kernel/trace/ftrace.c:1605
T430s Lid events no longer put machine to sleep (actually an old bug since 3.9-rc1, haven’t had time to bisect)

Daily log June 10th 2013 is a post from: codemonkey.org.uk

Dave Jones: git commit statistics.

Tue, 11 Jun 2013 01:17:38 +0000

Linus posted a git one-liner this morning that ended up intriguing me.
git log --pretty=%aD --author=davej@redhat.com | cut -c1-3 | sort | uniq -c | sort -n

The output looks like this..

     37 Sat
     42 Sun
     50 Fri
     73 Wed
     78 Mon
     79 Thu
    112 Tue

What I found interesting, is that on every single git repo I ran that command on, Tuesday was my most productive day.
As much as I hate Mondays, I think the real reason there is that I treat Mondays as ‘catch up’ day for the most part. As can be clearly seen, I don’t really work on weekends any more, so Mondays tend to be dealing with a backlog of email/git commits, bringing kernels on test machines up to date, and paging in context from whatever I was working on the prior week.

git commit statistics. is a post from: codemonkey.org.uk

Pavel Machek: Dear William Hague

Mon, 10 Jun 2013 09:00:10 +0000

If you are not a criminal, please publish your PIN, bank account password, email/facebook passwords and complete medical history. You don't have anything to hide, do you?

Eric Sandeen: Running the numbers on Minnesota’s solar mandate

Sat, 08 Jun 2013 22:23:41 +0000

At the end of the 2013 legislative session in Minnesota, legislators passed an omnibus energy bill which included, among other things, a requirement that investor-owned utilities in Minnesota (Read: Xcel Energy) must generate 1.5% of their electricity from solar by 2020. There were a lot of other things in there as a result of the ~~sausage~~ law-making process for the solar mandate, including some that I’m not very fond of, but the bottom line of encouraging more solar development is a good thing in my book. (Also, it was signed into law on my birthday!)

1.5% doesn’t sound like a whole lot, but what does it really mean in terms of physical solar PV deployments? Numbers have been tossed around that this will require 450MW of new capacity in the next 7 years.

Assuming the 450MW number is correct, and picking 250W panels as a common panel size today, that’s 450,000,000 / 250 = 1,800,000 or 1.8 million panels installed by 2020. That’s about 700 panels installed every day for 7 years.

If commodity sized (65x39cm) panels are used, that’s about 112 acres of panels (if they were laid out flat and edge to edge, which of course they aren’t) ;) That’s roughly equivalent to 112 US football fields.

Is this possible? Sure. Austria installed 230MW in 2012 alone. New Jersey installed 415MW in 2012. And Minnesota gave itself 7 years to accomplish this goal.

Is 450MW the right number? According to the NREL PVWatts calculator for Minneapolis, 450MW of optimally situated, fixed solar PV could be expected to generate 578,512 MWh of solar energy in the course of a year.

According to the EIA energy data browser, all utilities (including co-ops etc) in Minnesota generated 42,586,000 MWh in 2012. 578,512MWh is about 1.3% of that number. Xcel is by far the largest generator, so if we take out the smaller co-ops etc, 450MW does seem like a reasonable ballpark number.

There are already large companies ready to jump at this. Geronimo Energy has submitted a proposal to provide up to 100MW of capacity at up to 31 sites ranging from 2 to 10MW. I honestly hope this isn’t the predominant mode of development. We have an awful lot of flat roofs which would be well suited – for example, Ikea put 1MW on their Minnesota store last year. 100 acres or so isn’t all that much land, but I’d still rather see this go up on the built environment before we start using farmland & green space.

I’m excited to see how this works going forward. Will my friends in the small-scale solar installation business stay busy? Will SolarCity come to town? Will companies like Geronimo make up the bulk of this with giant installations? Will it reduce the need for new gas peaker plants? Time will tell, but it’s an exciting time for solar in Minnesota, for sure.

Dave Jones: Daily log June 7th 2013

Sat, 08 Jun 2013 04:07:16 +0000

Woke up to a patch fixing the NETLINK_MMAP corruption bug I’ve been chasing. Seems to work.
Some more polishing on trinity towards a new release (next week possibly).

My 3.10-rc4 outstanding issues.

top reports cpus maxed while idle (Fix is in linux-tip:timers/urgent).
XFS log recovery bug
Trinity triggered bugs:
- NETLINK mmap corruption bug.
- udpv6 oops (Been around for a long time, still no fix).
- Assorted RCU/NOHZ_FULL bugs.
- Assorted perf bugs.
  - perf function tracing uses rcu_lock()
  - perf_lock_task_context deadlock
T430s Lid events no longer put machine to sleep (actually an old bug since 3.9-rc1, haven’t had time to bisect)

Daily log June 7th 2013 is a post from: codemonkey.org.uk

Dave Jones: Weekly Fedora kernel bug statistics – June 07 2013

Fri, 07 Jun 2013 17:54:08 +0000

	17	18	19	rawhide
Open:	250	417	126	66	(859)
Opened since 2013-05-31	0	26	17	1	(44)
Closed since 2013-05-31	16	11	5	2	(34)
Changed since 2013-05-31	41	41	26	11	(119)

Weekly Fedora kernel bug statistics – June 07 2013 is a post from: codemonkey.org.uk

Dave Jones: Daily log June 6th 2013

Fri, 07 Jun 2013 05:33:52 +0000

Despite yesterdays SSD death, turned out the XFS bug I was seeing is actually an XFS bug.
Noticed the busy-while-idle bug again. Gathered info and pinged upstream. Annoying. Tested a patch currently in linux-tip.
Reinstalled test machine with spare SSD.
Once that was done, I chased down the root cause of the mystery mmap corruption. Investigation ongoing with upstream author, though it looks like this is 3.10-rc only. More details on that soon. Chasing this down has taken all week, and highlighted a number of areas I need to improve trinity in.

Daily log June 6th 2013 is a post from: codemonkey.org.uk

Dave Jones: daily log June 5th 2013

Thu, 06 Jun 2013 14:36:16 +0000

Another SSD died (Intel 520 this time). Annoyingly the one that I was chasing an XFS bug in. Lots of time spent transplanting it into different machines to try and salvage uncommitted changes without luck. RMA time again.
Bad allergies combined with a headcold made the rest of the day a sudafed & red bull induced haze.

daily log June 5th 2013 is a post from: codemonkey.org.uk

Dave Jones: daily log June 4th 2013

Wed, 05 Jun 2013 03:40:32 +0000

Managed to hit the XFS ‘dir block has an entry not in the hash index’ bug again. Just like last time, it prevented booting, and running xfs_dump (or xfs_repair) on the block device from the rescue shell locked up. Ended up having to remove the SSD and put it in another machine over USB.
Created a xfs dump for Dave Chinner to ponder over.
Built 3.10-rc4 for rawhide
Continued attempts at narrowing down weird mmap bug.
Prepping some more debug patches for rawhide over the next few days.

daily log June 4th 2013 is a post from: codemonkey.org.uk

Harald Welte: Attending HITCON and COSCUP in Taipei

Wed, 05 Jun 2013 02:00:00 +0000

It is my pleasure to attend the HITCON 2013 and COSCUP 2013 conferences in July/August this year. They are both in Taipei. HITCON is a hacker/security event, while COSCUP is a pure Free/Open Source Software conference.

At both events I will be speaking at the growing list of GSM related tools that are available these days, like OpenBSC, OsmcoomBB, SIMtrace, OsmoSGSN, OsmoBTS, OsmoSDR, etc. As they are both FOSS projects and useful in a security context, this fits well within the scope of both events.

Given that I'm going to be back to Taiwan, I'm looking very much forward to meeting old friends and former colleagues from my Openmoko days in Taipei. God, do I miss those days. While terribly stressful, they still are the most exciting days of my career so far.

And yes, I'm also going to use the opportunity for a continuation of my motorbike riding in this beautiful country.

Dave Jones: daily log June 3rd 2013

Tue, 04 Jun 2013 03:58:13 +0000

Spent pretty much the whole day chasing one bug. Which meant a bunch of trinity experiments to try and narrow down the cause. Still no real results, but feels like I’m getting closer. Need to start thinking about more generic ways to do this frequent “narrow down test cases” problem.
Merged a couple pending trinity patches.

daily log June 3rd 2013 is a post from: codemonkey.org.uk

Andy Grover: Fedora for short-lifespan server instances

Mon, 03 Jun 2013 16:00:58 +0000

I read Máirín Duffy’s coverage of the Fedora Board’s userbase discussion. Really interesting. I wanted to add my take.

tl;dr: Puppet/Chef make Fedora’s short support period much less of an issue.

The OS is a building block

I’ve been watching a lot of videos on DevOps lately. Several close friends of mine are sysadmins and I’ve been learning a lot from them about the transformation that their profession is undergoing. From this year’s ChefConf, Adam Jacob’s keynote and the talk by Sascha Bates really impressed on me the big change in how admins should view machines — They’re not permanent, or even semi-permanent. They are ephemeral snowflakes that may live a year, or just an hour, so don’t get too attached.

Part of why admins like VMs is because the isolation they provide between different services. I used to run mail, DNS, and httpd from a single machine. Everything was mostly separate but not quite everything. They had separate userids but everyone’s config was in /etc, even touching the same files, sometimes. A full disk affected everybody. /var/log/messages didn’t split up their logging cleanly (by default, anyway.) It all just was built assuming there would be an admin at a command shell who could use their brain to resolve the conflicts to make everything play nice on a single OS image.

One service per instance

Admins adopted VMs for isolation, increased density, and better per-service resource allocation, but then ran into other problems. The setup that they did by hand once per new-hardware now was once per instance. (Editing /etc/sudoers for the fiftieth time gets old.) The tools then evolved further, until today one may keep no persistent state in an instance. Now, an instance is kickstarted into existence and configured automatically for the one job it will ever do. The sysadmin’s job isn’t to herd boxen any more, it’s to build ‘em, run ‘em, and then reap ‘em.

All the OS mechanisms for co-existing server processes, they’re now either obsolete or vestigial to some degree. What is important is the malleability of the OS to assume all the tasks it may be asked to – rather like a stem cell needs to be able to become a nerve or muscle, but never needs to be both.

Never upgrade, just redeploy

Let’s come back to the odd fact that Fedora is both a precursor to RHEL, and yet almost never used in production as a server OS. I think this is going to change. In a world where instances are deployed constantly, instances are born and die but the herd lives on. Once everyone has their infrastructure encoded into a configuration management system, Fedora’s short release cycle becomes much less of a burden. If I have service foo deployed on a Fedora X instance, I will never be upgrading that instance. Instead I’ll be provisioning a new Fedora X+1 instance to run the foo service, start it, and throw the old instance in the proverbial bitbucket once the new one works.

Cheap and easy virt and config management gives admins what they’ve always wanted — stability when they want it (run a LT support distro image, or for the VM host) or the latest stuff for their fast-moving business-oriented instances, by running a fast-update or rolling-release distro.

What Fedora should do

We’re already working on some of these to some degree — I think we should try to do even more to ensure Fedora is useful for the fast-update instance role.

First, Fedora needs to be able to be small. Nobody’s going to read the manpages on a throwaway instance, nobody’s even going to run vi. Image size matters when multiplied for each instance. Can we get by without /usr/share/doc/* and its thousands of copies of the GPL text? Fedora seems pretty good but there must be more we can do.

Second, we need to ensure Fedora supports the packages people are really using these days. Latest Ruby. Latest OpenStack. Vagrant. Django. Chef. Puppet. All the weird JS stuff that’s popular now on GitHub. Continue to improve packaging tools so it’s easier for new contributors to do their first package, as well as for long-time packagers to maintain more packages. And not just package for contribution to Fedora, but for admins to package for solely internal distribution. Like Sascha Bates stresses in her talk, packaging is a huge benefit to automation, but it does require effort. It can be easier.

Finally, I think we need to continue to look at how easy it is to configure and manage an instance of the OS, and tailor it more for automated configuration. I believe the key to this is adding programmatic interfaces where they are lacking. See my “All Plumbing needs an API” talk. Since we’re probably being configured by another piece of code rather than a person at the shell, we need clear, unambiguous programmatic interfaces with good error handling. Chef should not be calling cmdline tools and checking error codes, there should be a Ruby configuration library that natively controls the whatever-it-is directly! We want configuring Fedora to be fast, straightforward, and reliable.

Conclusion: Stable+fast-update is better than stable+self-built

Practically the whole history of Linux distros has been the conflict between stability and new features. With virtualization, one still must make this choice, but at a much finer granularity than before. If you’re going to re-instance within 6 months anyways, why manually build your latest-Ruby and whatnot to support your app on top of a stable distro image? Maybe just use Fedora for those.

Matthew Garrett: Dealing with UEFI non-volatile memory quirks

Mon, 03 Jun 2013 15:25:23 +0000

Since I wrote this, we've made some worthwhile progress on avoiding damaging Samsung hardware. The first is that the samsung-laptop driver appeared to be causing the firmware to attempt to write to an area of memory that was marked in the chipset, triggering a Machine Check Exception. That was what generated the pstore output that caused the problem originally. The driver now refuses to load if EFI is enabled, which avoids the problem. It's not ideal, since it's currently the only mechanism we have for certain functionality on Samsung laptops, but there you go.

The second problem was that avoiding crashing on boot didn't actually fix the problem in any fundamental way. Even with pstore disabled, it was possible for userspace to fill the nvram and trigger the same problem. Our first approach to this was to prevent any writes to nvram if the UEFI QueryVariableInfo() call reported that more than 50% of the nvram storage space would be used. That was safe, but led to another issue. The nvram storage area is typically implemented as part of the same flash chip as the firmware. Flash isn't arbitrarily accessible - changing the contents of a block typically involves rewriting the entire block. It's impractical to rewrite the entire nvram area on every write, so what actually happens is that deleting variables just results in them being marked as inactive but doesn't actually free up the space. The firmware can later perform some sort of garbage collection to free it up.

This caused us problems, since inactive space that hasn't been garbage collected yet isn't actually available, and as a result firmware implementations tend to count it as used. Say you had 64KB of nvram and wrote 32KB of variables. We'd then refuse to write any more because you'd drop below 50%. So you delete 16KB of the variables you've created and try again. Unfortunately, the firmware still thinks that there's 32KB in use and Linux would still refuse.

If you were lucky, rebooting would trigger a garbage collection run. If you weren't, it wouldn't. Problematic. Our next approach was to try to account for the space actually actively used by the variables, rather than relying on what the firmware told us via QueryVariableInfo(). This seems simple enough - just add up the size of all the variables and subtract that from the overall size to determine how much of the "used" space is actually just old inactive variables that can be ignored. However, there's still some problems there. The first is that each variable has some additional overhead associated with it, and the size of that overhead varies depending on the system vendor. We had to make a conservative guess, which could cause problems if systems had large numbers of small variables. The second is that the only variables the kernel can see are those that are flagged as runtime-visible. There may also be a significant quantity of nvram used to store variables that are only visible in boot services code. We could work around this by adding up sizes while we're still in boot services code, but on some systems calling QueryVariableInfo() before ExitBootServices() results in later calls to GetNextVariable() jumping to invalid addresses and crashing the kernel. Not a great approach.

Meanwhile, Samsung got back to us and let us know that their systems didn't require more than 5KB of nvram space to be available, which meant we could get rid of the 50% value and replace it with 5KB. The hope was that any system that booted with only 5KB of space available in nvram would trigger a garbage collection run. Unfortunately, it turned out that that wasn't true - some systems will only trigger garbage collection if the OS actually makes an attempt to write a variable that won't otherwise fit.

Hence this patch. The new approach is to ask the firmware how much space is available. If the size of the new variable would reduce this to less than 5K, we attempt to create a variable bigger than the remaining space. This should cause the firmware to realise that it's out of room and either (depending on implementation) perform a garbage collection run at runtime or set a flag that will cause the system to perform garbage collection on the next reboot. We then call QueryVariableInfo() again to see whether a garbage collection run actually happened, and if so check whether we now have enough space. If so, we go ahead and write the variable. If not, we tell userspace that there's not enough space.

This seems to work in all the situations I've tested, and it should avoid ending up in a situation where a Samsung can end up bricked. However, it's firmware, so who knows whether it's going to break things for someone else.

comments

Harald Welte: Rest In Peace, Atul Chitnis

Mon, 03 Jun 2013 02:00:00 +0000

Today, very sad news has reached me: Atul Chitnis has passed away. Most people outside of India will most likely not recognize the name: He has been instrumental in pioneering the BBS community in India, and the founder and leader of the Linux Bangalore and later FOSS.in conferences, held annually in Bangalore.

I myself first met Atul about ten years ago, and had the honor of being invited to speak at many of the conferences he was involved in. Besides that professional connection, we became friends. The warmth and affection with which I was accepted by him and his family during my many trips to Bangalore is without comparison. I was treated and accepted like a family member, despite just being this random free software hacker from Germany who is always way too busy to return the amount of kindness.

Despite the 17 year age difference, there was a connection between the two of us. Not just the mutual respect for each others' work, but something else. It might have been partially due to his German roots. It might have been the similarities in our journey through technology. We both started out in the BBS community with analog modems, we both started to write DOS software in the past, before turning to Linux. We both became heavily involved in mobile technology around the same time: He during his work at Geodesic, I working for Openmoko. Only in recent years his indulgence in Apple products was slightly irritating ;)

Only five weeks ago I had visited Atul. Given the state of his health, it was clear that this might very well be the last time that we meet each other. I'm sad that this now actually turned out to become the thruth. It would have been great to meet again at the end of the year (the typical FOSS.in schedule).

My heartfelt condolences to his family. Particularly to his wonderful wife Shubha, his daughther Anjali, his mother and brother. [who I'm only not calling by their name in this post as they deserve some privacy and their Identities is not listed on Atuls wikipedia page].

Atul was 51 years old. Way too young to die. Yet, he has managed to created a legacy that will extend long beyond his life. He profoundly influenced generations of technology enthusiasts in India and beyond.

Pavel Machek: Scary copy&paste issues

Sat, 01 Jun 2013 13:12:29 +0000

This is scary. I wonder if it is worth filling security bugs against web browsers?

Dave Jones: daily log May 31st 2013

Fri, 31 May 2013 22:11:13 +0000

Bisection revealed the source of the mystery mmap bug. Still unclear as to why it happens. Mailed the upstream author.
Replacement SSD turned up yesterday, so spent a while installing it, and reinstalling laptop. Noticed that they still ship new SSDs with out-dated firmware.
Spent a while creating a bootable USB key to upgrade it.
bi-weekly #fedora-meeting.
Finally got to the point where I had to install the aircon unit again.

Amusing URL of the day.

My 3.10-rc3 outstanding issues.

tickbroadcast bootmem allocator trace.
XFS xfs_setattr_size assertion pending fix
XFS unhashed dirblock bug.
Mystery unprivileged ‘crash other processes’ bug.
Trinity triggered bugs:
- udpv6 oops (Been around for a long time, still no fix).
- Assorted RCU/NOHZ_FULL bugs.
  - Mysterious ‘rcu_preempt’ goes to max cpu while idle bug.
  - [1],
  - [2](potential fix),
  - [3]
  - [4](fix).
T430s Lid events no longer put machine to sleep (actually an old bug since 3.9-rc1, but laptop is out of action until next week)

daily log May 31st 2013 is a post from: codemonkey.org.uk

Dave Jones: Monthly Fedora kernel bug statistics – May 2013

Fri, 31 May 2013 17:39:31 +0000

	17	18	19	rawhide
Open:	267	404	110	65	(846)
Opened since 2013-05-01	21	103	43	14	(181)
Closed since 2013-05-01	28	62	53	25	(168)
Changed since 2013-05-01	51	155	54	22	(282)

Quite a few CVE’s this month. Made some progress through the btrfs backlog.
The 3.9 rebase revealed some new “Can’t boot” cases, which looks like yet another UEFI bios bug.

Monthly Fedora kernel bug statistics – May 2013 is a post from: codemonkey.org.uk

Dave Jones: Weekly Fedora kernel bug statistics – May 31st 2013

Fri, 31 May 2013 17:32:42 +0000

	17	18	19	rawhide
Open:	267	404	110	65	(846)
Opened since 2013-05-24	7	36	14	1	(58)
Closed since 2013-05-24	14	11	8	4	(37)
Changed since 2013-05-24	16	63	24	4	(107)

Weekly Fedora kernel bug statistics – May 31st 2013 is a post from: codemonkey.org.uk

Dave Jones: daily log May 30th 2013

Fri, 31 May 2013 04:43:16 +0000

end of month administrivia. status reports etc.
booked flight for FLOCK.
Last nights storm knocked out power a few times. One machine didn’t come back up until I left it off overnight. More strange, is that one LCD wouldn’t power up until I replaced its IEC cable. Took advantage of the situation to recable a bunch of stuff in my office.
Reorganisation included removal of several switches that were linked to other switches. Even though these were gigabit switches, latency between machines ‘feels’ subjectively better. I never took ‘before’ measurements to prove it one way or another.
For no obvious reason, since moving from one side of the room to the other, firewall thinks ctrl is stuck on keyboard.
Continued bisecting mystery mmap bug.

daily log May 30th 2013 is a post from: codemonkey.org.uk

Dave Jones: daily log May 29th 2013

Thu, 30 May 2013 00:59:56 +0000

Up early. Went into office. On arrival, found that laptop wouldn’t boot (now I remember why this was the ‘backup’ laptop).
Spent an hour trying to coax it to until giving up and attempting reinstall. Which failed.
Did some bugzilla triage on an ipad that I happened to have with me. (Surprisingly not a horrible way to do this, first time for everything). Noticed we seem to be getting more filesystem/vfs related bugs than we used to. Going to need to scrounge up some more disks/controllers soon I think.

Started feeling ‘not right’ around lunchtime. Skipped lunch. Headaches and nausea mid afternoon.
Not the most productive of days.

daily log May 29th 2013 is a post from: codemonkey.org.uk

Dave Jones: daily log May 28th 2013

Wed, 29 May 2013 03:28:04 +0000

email backlog of doom.
Another week, another xfs bug.
Found (yet another) perf/rcu bug. It never ends.
Investigating a bug which appears to allow a user to kill other users (even root) processes. (All day, ongoing).
Made a change to trinity to allow calls to syscalls with no arguments. Some other small fixes.
Infrastructure work for some IO testing we’ll be doing soon.
Reinstalling backup laptop (Dell Adamo) in absence of new SSD for thinkpad.

daily log May 28th 2013 is a post from: codemonkey.org.uk

Matthew Garrett: Secure Boot isn't the only problem facing Linux on Windows 8 hardware

Tue, 28 May 2013 21:41:43 +0000

There's now no shortage of Linux distributions that support Secure Boot out of the box, so that's a mostly solved problem. But even if your distribution supports it entirely you still need to boot your install media in the first place.

Hardware initialisation is a slightly odd thing. There's no specification that describes the state ancillary hardware has to be in after firmware→OS handover, so the OS effectively has to reinitialise it again. This means that certain bits of hardware end up being initialised twice, and that's slow in some cases. The most obvious is probably USB, which has various timeouts as you wait for hardware to settle. Full USB support in the firmware probably adds a couple of seconds to boot time, and it's arguably wasted because the OS then has to do the same thing (but, thankfully, can at least do other things at the same time). So, looking for USB boot media takes time, and since the overwhelmingly common case is that users don't want to boot off USB, it's time that's almost always wasted.

One of the requirements for Windows 8 certified hardware is that it must complete firmware initialisation within a specific amount of time, something that Microsoft refer to as "Fast Boot". Meeting these requirements effectively makes it impossible to initialise USB, and it's likely that certain other things will also be skipped. If you've got a USB keyboard then this obviously means that your keyboard won't work until the OS starts, but even i8042 setup takes time and so some laptops with traditional PS/2-style keyboards may not set it up. That means the system will ignore the keyboard no matter how much you hammer it at boot, and the firmware will boot whichever OS it finds.

For a newly purchased device, that's going to be Windows 8. It's not too much of a problem with a fully installed Windows 8, since you can hold down shift while clicking the reboot icon and get a menu that lets you reboot into the firmware menu. Windows sets a flag in a UEFI variable and reboots the system, the firmware sees that flag and does full hardware initialisation and then drops you into the setup environment. It takes slightly longer to get into the firmware, but that's countered by the time you save every time you don't want to get into the firmware on boot.

So what's the problem? Well, the Windows 8 setup environment doesn't offer that reboot icon. Turn on a brand new Windows 8 system and you have two choices - agree to the Windows 8 license, or power the machine off. The only way to get into the firmware menu is to either agree to the Windows 8 license or to disassemble the machine enough that you can unplug the hard drive[1] and force the system to fall back to offering the boot menu.

I understand the commercial considerations that result in it ranging from being difficult to impossible to buy new hardware without Windows pre-installed, but up until now it was still straightforward to install an alternative OS without agreeing to the Windows license. Now, installing alternative operating systems on many new systems will require you to give up certain rights even if you want nothing other than to reach the system firmware menu.

I'm firmly of the opinion that there are benefits to Secure Boot. I'm also in favour of setups like Fast Boot. But I don't believe that anyone should be forced to agree to a EULA purely in order to be able to boot their own choice of OS on a system that they've already purchased.

[1] Which is a significant and probably warranty-voiding exercise on many systems, and that's assuming that it's not an SSD soldered to the motherboard…

comments

Daniel Vetter: i915/GEM Crashcourse: Overview

Sun, 26 May 2013 14:42:15 +0000

Now that the entire series is done I've figured a small overview would be in order.

Part 1 talks about the different address spaces that a i915 GEM buffer object can reside in and where and how the respective page tables are set up. Then it also covers different buffer layouts as far as they're a concern for the kernel, namely how tiling, swizzling and fencing works.

Part 2 covers all the different bits and pieces required to submit work to the gpu and keep track of the gpu's progress: Command submission, relocation handling, command retiring and synchronization are the topics.

Part 3 looks at some of the details of the memory management implement in the i915.ko driver. Specifically we look at how we handle running out of GTT space and what happens when we're generally short on memory.

Finally part 4 discusses coherency and caches and how to most efficiently transfer between the gpu coherency domains and the cpu coherncy domain under different circumstances.

Happy reading!
Update: There's now also a new article with a few questions and answers about some details in the i915 gem code.

Daniel Vetter: i915/GEM Q&A

Sun, 26 May 2013 14:40:45 +0000

So apparently people do indeed read my my i915/GEM crashcourse and a bunch of follow-up questions popped up in private mails. Since I'm a lazy bastard I've clean some of the common questions&answers up to be able to easily point at them. And hopefully they also help someone else to clarify things a bit.

Question: What’s the significance of i915_gem_sw_finish_ioctl ? It seems to flush cpu caches, but only conditional on obj->pin_count != 0. Why does it no unconditionally flush the cpu caches like e.g. when we move an unsnooped/not LLC-cached object into a gpu domain?

Answer: i915_gem_sw_finish_ioctlis only used to flush out cpu rendering to the display (and in current userspace it's not used at all). obj->pin_count != 0 is used as a proxy for "this a scanout buffer". Obviously more intelligent userspace should know whether it is doing cpu rendering to a displayed buffer or not and force the expensive clflushing with e.g. the set_domain ioctl only when really required. But the sw_finish ioctl is called from the libdrm cpu mmap unmap function, which does not have this knowledge at hand, hence the check in the kernel. Furthermore for efficient integration of cpu rendering into the gpu render pipeline we want to use snoopable objects even on non-LLC platforms which means that this ioctl shouldn't really be used any more for new code.

Question: So the cpu can only access a GEM object through the GTT when it's in the mappable part of the GTT, i.e. when gtt_offset + size <= gtt_mappable_end. But the i915_gem_object_set_to_gtt_domain function does not check that whether this condition is satisfied or not and simply goes ahead with the domain change. Why is that done so, even though the cpu won't be able to access the buffer object at its current place?

Answer:The GTT domain is purely about coherency, i.e. a buffer object is in the GTT domain if reads/writes through the GTT would see the correct values. The other big domain is cpu domain, i.e. the data (when accessed directly in the physical memory location, not going through the GTT) is coherent with cpu caches. Shifting between these two domains requires flushing/invalidating cpu caches.

Note that on recent kernels that doesn't even mean that there's a global GTT mapping allocated for that buffer object: This is used to optimize away the redundant cache flushing when moving an object around, e.g. when moving it into the mappable range to serve a cpu access page fault. In the future this will be even more common once we have proper per-process GTT address spaces. Then an object could be fully coherent with the GTT domain, read by the gpu through a PPGTT mapping, but don't have an offset allocated for it in the global GTT at all.

The mappable GTT address range on the other hand is a different concept and simply means the object has a GTT mapping visible to the cpu (on gpus without PPGTT the global GTT can be up to 2g, but only 256m are usually visible in the pci bar). Note that GEM object can be mappable but can be (at the same time) in the cpu domain. This happens when userspace writes to the buffer object through the cpu mappings.

Question: How does the the i915_gem_fault function handle a page fault when it itself is invoked through a page fault in the i915 GEM kernel code? Like suppose if fault_in_pages_readable function is called which dereferences a user pointer - won't that cause issues with deadlocks?

Answer:Yes, this can happen and we need to be careful that we cannot possible deadlock with our own pagefault handlers. And it's not just theoretical, it happens in the wild when a GL client tries to use a pointer obtained from one of the texture mapping funtions (which can use a GTT memory mapping internally) to upload data (which could use the pwrite GEM ioctl).

These potential deadlocks are resolved by instructing the linux memory subsystem to not serve pagefaults when accessing userspace memory but instead fail it. Then our code can release any resources and locks required by our own page fault handler and retry the operation in a slowpath. Often this requires that we copy the data into a (unfaultable) temporary buffer in kernel's memory space. These atomic sections are often implicit, but we have a few places where we need to explicitly disable page fault handler with pagefault_disable/enable() calls.

Question: Is obj->fenced_gpu_access ever set on modern platforms - it seems not? Or could this cause a stall waiting for the gpu when all fences are in use and we need a few fence to handle a GTT page fault?

Answer: No, this is only ever set on Gen2/3 devices. Those gpus use the same GTT fences used on all platforms for detiling cpu access also for gpu access, at least for some gpu rendering functions. So this is irrelevant on modern platforms and can't lead to a stall in the pagefault handler when accessing an otherwise idle buffer object.

Question: What is this wedeged stuff - there's lots of references to it in the i915 GEM code?

Answer: This is part of the gpu hang detection and reset handling code, which I didn't really cover in my crashcourse. It is set when we've detected a hang but failed to reset the gpu. It will cause all subsequent command submission from userspace to fail with -EIO, which is used by userspace as a signal to fall back to software rendering. The i915 hang detection and reset code has been (and still is) under pretty active development and is nowadays a rather complex piece of code. I plan to cover it more in-depth hopefully soon.

Question: In the use_cpu_reloc function, why is the obj->cache_level != I915_CACHE_NONE condition used?

Answer: That's just crazy optimization - it's always faster to write relocations through cpu maps if LLC caching is enabled. But without caching it's faster to use global GTT access - but then only if we have the mappable mapping already set up. Note that pwrite ioctl code has similar tricks.

Eric Sandeen: Enphase Solar Microinverter Clipping Analysis

Sat, 25 May 2013 21:08:52 +0000

I wanted to look at how much the “clipping” behavior of power-limited solar microinverters affected my annual energy production. The TL;DR version is: at worst, only about 0.6% loss due to clipping. For more, read on.

A photovoltaic inverter is a device which converts the DC energy from the panel into AC energy for the grid; it also manages optimum power point tracking. Traditionally this was done with a big central inverter for all panels combined; recently companies such as Enphase Energy have started making microinverters, which are per-panel devices. One advantage of these devices is that each panel operates independently so that if one panel is shaded, damaged, or dirty, it doesn’t affect the rest of the array.

I have 11 230W Solar PV panels on my roof with an Enphase M190 micronverter on each. These are nominally 190W devices, though in practice they have a maximum output of 199W. (Note, these are 3 year old models; Enphase now has microinverters with more capacity). The fact that the panels can produce more than the microinverter can handle might seem like an issue; indeed on a cool, clear day we can see the effect:

So from the graph above it’s clear that I am losing a little energy production during that clipping. What would normally be a smooth curve is flattened out at the top as I hit the 11x199W = 2189W limit. (A few factors affect whether this clipping happens; obviously we need a clear day, but optimal sun angle and, perhaps more than anything, panel temperature affects it greatly).

I wanted to try to quantify this a bit – how much am I really losing from this behavior?

One cool thing about the Enphase units is that they report data every 5 minutes, and this data can be queried via a standard data API. So I pulled down the past 365 days worth of data to see how often I was clipping. I grabbed 5-minute data files for each day, and looked for when clipping seems to start, by looking at watt output around the clipping point, and how many 5-minute entries there were for each wattage:

$ egrep -w "21[789][0-9]" *.json | awk -F : '{print $8}' | sort | uniq -c
   9 2170,"enwh"
  10 2171,"enwh"
  15 2172,"enwh"
  11 2173,"enwh"
  17 2174,"enwh"
  17 2175,"enwh"
  11 2176,"enwh"
  20 2177,"enwh"
  24 2178,"enwh"
  14 2179,"enwh"
  17 2180,"enwh"
  21 2181,"enwh"
  32 2182,"enwh"
  51 2183,"enwh"
 107 2184,"enwh" <-- actual clipping start?
 166 2185,"enwh"
 134 2186,"enwh"
 119 2187,"enwh"
  97 2188,"enwh"
  62 2189,"enwh" <-- nominal clipping, 11x199
  21 2190,"enwh"
   2 2191,"enwh"

There’s a pretty big jump at 2184W, so I went with that as a definition of “when clipping starts” vs. the nominal 2189W. Adding up the occurrences of clipping, I got 708 5-minute intervals of clipping out of the last 365 days. That’s about 59 hours.

So how much energy is that? My panels can nominally make 11x230W = 2530W of output, so 2530-2184 = 346W lost, at most, during clipping. That’s actually too high; not every instance above is clipping, and not every interval would have been making the maximum output. So we’ll take that as a high estimate.

346W x 59 hours is 20,414 watt-hours, or about 20kWh. At around $0.10/kWh, that’s about $2.00 of lost value. Over the same year period, my array made 3,356 kWh, so 20kWh lost is about 0.6% of that. I would hope that the microinverters made up at least that much by virtue of keeping the array going over the winter when some panels were snow-covered, etc. Remember, many of my assumptions above make this a high estimate.

One other interesting datapoint is to see when this clipping occured. By month, here’s how it looks:

March was far and away the highest; this is probably due in large part to the cooler temperatures, which make the panels more efficient.

Dave Jones: daily log May 24th 2013

Fri, 24 May 2013 23:20:13 +0000

Woke up to find a fix in my inbox for the XFS setattr bug. Bug turned out to be a few years old, and also highlighted a broken test in xfstests. Two bugs for the price of one.
Applied the daily “fix trinity compile on old distros” bug.
Briefly looked at Boehm-Demers-Weiser garbage collector with a view to maybe using it in Trinity in the absence of better allocation tracking.
Updated a triage script to add all the aliases we have for fedora kernel bugs. Hoping that in time that script can grow to be even more useful.
Tried out another potential fix for the RCU problems I’ve been seeing. Seems good so far.
Looked at a bunch of “can’t boot” bugs that came in since F18 got rebased to 3.9. Found a thread upstream that seems to be discussing the same bug.
Spent the afternoon compiling OCLint out of curiousity. Had no idea how long it would take, or how much it would increase the temperature in my office. After four hours of sitting in a simulated sauna, llvm finished building, but oclint wouldn’t build. Ran out of time to play with it. Something for next week.
Seeding improvements from Kees to trinity.git

and now: three day weekend. \o/

daily log May 24th 2013 is a post from: codemonkey.org.uk

Andy Grover: Using qla2xxx with LIO on Fedora

Fri, 24 May 2013 16:29:32 +0000

In addition to turning your Fedora 18 box into an iSCSI target, LIO also supports other SCSI transport layers (‘fabrics’), such as Fibre Channel, with the qla2xxx fabric.

The most crucial bit is to verify that the qla2xxx driver has initiator mode disabled — it should be operating in target mode only. You can check this with:

cat /sys/module/qla2xxx/parameters/qlini_mode

It should say ‘disabled’. If it doesn’t, create a file called /usr/lib/modprobe.d/qla2xxx.conf and put:

options qla2xxx qlini_mode=disabled

in it. Then, run ‘dracut -f’ to rebuild your initrd, and reboot.

Some of you may be wondering: why /usr/lib/modprobe.d instead of /etc/modprobe.d ? This is because qla2xxx is likely loaded from the kernel’s initial ramdisk (initrd), and dracut, the initrd building tool, omits “host-specific” settings in /etc/modprobe.d. While you’re mucking around, also make sure the firmware package for your qla device, such as ql2200-firmware or similar, is also installed.

targetcli won’t let you create a qla2xxx fabric if qlini_mode is wrong. Once it lets you create the qla fabric, you can add luns to it and grant access permissions to acls exactly in the same manner as the other LIO fabrics.

Dave Jones: Weekly Fedora kernel bug statistics – May 24th 2013

Fri, 24 May 2013 16:10:15 +0000

	17	18	19	rawhide
Open:	276	380	104	66	(826)
Opened since 2013-05-17	4	32	6	3	(45)
Closed since 2013-05-17	3	11	4	3	(21)
Changed since 2013-05-17	16	48	14	5	(83)

Weekly Fedora kernel bug statistics – May 24th 2013 is a post from: codemonkey.org.uk

Dave Jones: daily log May 23rd 2013.

Fri, 24 May 2013 04:25:45 +0000

Continued chasing the xfs bug. Confirmed that with the pending fix in the XFS tree, I can no longer reproduce the bug I saw. Then 3-4 hours later, it popped up again.
Annoying. Finally got some debug info to Dave Chinner at the end of the day.
Timezone differences making it even more annoying to debug.
Fixed up a compile problem on older distros in trinity since the recent perf changes.
Vince Weaver’s perf specific trinity fork is finding more bugs in the perf syscall already [1], [2].
Merged trinity patch from Daniel Borkmann to add randomized seccomp filters generated by markov chain. Interesting stuff.
Planning for some interesting testing work next week.
Started some prep work for trinity 1.2

My 3.10-rc2 outstanding issues:

RCU/NOHZ_FULL bugs [1], [2]
RCU bootmem allocator trace.http://codemonkey.org.uk/?p=477&preview=true
tickbroadcast bootmem allocator trace.
XFS Slab corruption (pending fix)
XFS xfs_setattr_size assertion
T430s Lid events no longer put machine to sleep (actually an old bug since 3.9-rc1, but laptop is out of action until next week)

Puzzling website of the day: pain registers.

daily log May 23rd 2013. is a post from: codemonkey.org.uk

Dave Jones: daily log May 22nd 2013.

Thu, 23 May 2013 03:51:12 +0000

Going to try and continue yesterdays daily log format for a while.

Grumbled at openvpn changing pathname for ‘plugin’ to ‘plugins’ breaking my vpn script.
Bugzilla seemed unhappy. Gave up trying to look at it after it kept timing out.
Continued poking at the XFS assertion from yesterday. Downgraded the compiler from f19′s 4.8.0 to 4.7.3. No luck. Couldn’t reproduce on 3.9, so started bisecting. Seemed to be caused by a patch I added recently to work around another XFS bug (slab corruption). I can’t win. Dave Chinner confused by my diagnosis. Bisect take 2 on that tomorrow.
Vince Weaver posted a perf_event fuzzer based on trinity. Spent a while reading it over. Neat. Glad to see people taking an idea and running with it in new directions. The more test programs the better.
Diagnosed yesterdays “microcode loader got slow” bug. Turned out that I had somehow inadvertently set CONFIG_FW_LOADER_USER_HELPER, which incurs a 60 second timeout.
While waiting for bisections, looked over some bugs in coverity’s database. Around 1500 untriaged. Would like to find time to work on that some at some point.

Spent so much of the day bisecting/building/rebooting that I didn’t write much new code today. Ho-hum.

daily log May 22nd 2013. is a post from: codemonkey.org.uk

Michael Kerrisk (manpages): More man pages now rendered at man7.org

Wed, 22 May 2013 13:18:55 +0000

As detailed in this blog post, I've expanded the set of man pages rendered in HTML at http://man7.org/linux/man-pages/ to include pages in addition to those provided by the man-pages project. This change has several purposes. One main purpose is to provide a up-to-date and regularly updated HTML renderings of these man pages. (Most online man page renderings are out-of-date to some extent--in some cases, extremely out of date.) The other main purpose is to provide information on where to report bugs in each man page. To this end, each HTML rendering includes a COLOPHON that describes the origin of the page, notes the date when it was extracted, and provides information on where to report bugs in the page. (The man-pages project has already done this since December 2007, with the result that many more man page bugs are nowadays reported.)

Currently, man pages from nearly 40 projects are rendered, raising the number of pages rendered at man7.org from around 950 to around 1750. The projects that I have so far included have a bias that matches my interests: man-pages, projects related to low-level C and system programming (e.g., the ACL and extended attribute libraries), toolchain projects (e.g., gcc, gdb, Git, coreutils, binutils, util-linux), and other relevant tools (kmod, strace, ltrace, procps, expect) and tools relevant to manual pages (e.g., groff, man-db). The full list of projects and the corresponding man pages that are rendered can be found in the man pages by project index. I'm open to adding further projects to the rendered set, if they seem relevant. If you think there is a project that should be added, take a look at this blog post.

Dave Jones: a day in the life..

Wed, 22 May 2013 03:36:41 +0000

Got back from vacation today (since last Thursday). Here’s how I spent the day.

Caught up (skimmed) the 1500 postings to Linux-kernel and related mailing lists that had accumulated.
Reviewed, applied and cleaned up my patch backlog for trinity.
Caught up with direct mail that needed a response.
Brought my test machines up to 3.10rc2, and restarted tests.
Caught another pair of RCU/nohz bugs pretty quickly. [1][2].
Checked on the RMA for my failed SSD. Still awaiting shipment of replacement.
Received my ultrabay adaptor for my thinkpad. Surprised to find out that a full height SSD would fit into it.
Pushed out a 3.9.3 update for F18
Looked at bugzilla backlog. Swore a lot. 3.9.x rebase bugs started to trickle in.
Rewrote a bunch of code surrounding trinity’s rand() usage.
Finally got F19 installed via NFS on new test machine.
Hit an XFS assertion.
Then hit an i915 pineview kms console blanking bug.
Noticed that x86 microcode loading had gotten really slow. It seems to be waiting a whole 60 seconds for each core.

a day in the life.. is a post from: codemonkey.org.uk

Greg Kroah-Hartman: Updated history of the 2.6.16-stable kernel

Fri, 17 May 2013 16:34:00 +0000

A few years ago, I gave a history of the 2.6.32 stable kernel, and mentioned the previous stable kernels as well. I'd like to apologize for not acknowledging the work of Adrian Bunk in maintaining the 2.6.16 stable kernel for 2 years after I gave up on it, allowing it to be used by many people for a very long time.

I've updated the previous post with this information in it at the bottom, for the archives. Again, many apologies, I never meant to ignore the work of this developer.

Pete Zaitcev: Joe Arnold on Software-defined Storage

Thu, 16 May 2013 16:19:46 +0000

At Havana summit they were giving away a paper version of Joe Arnold's "Software Defined Storage with OpenStack Swift". Very useful book for anyone dealing with Swift, I would be glad to pay the cover price of $25. But even more interestingly than tips on care and feeding of Swift, Joe opens the whole book thus:

[...] a de-coupled management system so customers could achieve (1) amazing flexibility in terms of how (and where) they deployed their storage, (2) control of their data without being locked-in to a vendor and (3) private storage at public cloud prices.

These features are the essence of Software Defined Storage (SDS), a new term the meaning of which is being defined. [...] Key aspects of SDS are scalability, adaptability, and the ability to use most any hardware. Through this de-coupling, operators can now make choices on how their storage is scaled and managed and how users can store and access data — all driven programmatically for the entire storage tier, regardless of where the storage resources are deployed.

Parts of the above prompt questions. Firstly, what good is de-coupling in respect to lock-in? SwiftStack effectively locks in by owning the de-coupled management. Sure, you own your data and could, in theory, manage your Swift with another management plane... I do not expect anyone crazy enough to try switching by anything less than standing up a new cluster. In any case, that part is not important, IMHO. The important part is programmatic control.

The phrase "SDS" jumps off "Software-Defined Networking". When SDN came into OpenStack, I was quite skeptical about it. It seemed too much like vendor-driven marketing bullshit. However, as users deployed the Project Formerly Known as OpenStack Quantum, it became clear that SDN answers their needs. The chief need was the ability to shape networks programmatically, overlaid on top of the physical networking plant, in service of the VMs.

Before SDN, when all this cloud thing came about, practitioners also struggled with the definition of it, and in particular the difference from the plain old datacenter virtualization. The difference is the programmatic control throughout. RHEV (now oVirt) eventually grew an API, which blurred the lines. But in OpenStack it was the main feature from the start. So you can manage everything and anything programmatically, including, for example, running on bare hardware. One can say that cloud is "Software-Defined Computing".

So, how does this programmatic thing apply to Swift? Joe had interesting insights cunningly hidden in the book, like these:

In an SDS system, reliability is the responsibility of the software, not the hardware. Replication and data integrity tactics are used to ensure that data does not become corrupt and that lost data is recovered.

[...]

A crucial function of an SDS system is to orchestrate capacity — storage, networking, routing & services — for entire cluster.

Swift covers the first part well already. The second is missing, or "de-coupled".

For galactic fairness, he also wrote things that seem wrong-headed to me:

There is no application sharding or managing volumes which can drive operational knowledge and complexity into applications because the SDS system is one cohesive system. Users do not need to ask for or know 'which storage pool' should be used because there is only one namespace.

The problem with hiding the pools outside of namespace is that they become invisible to the programmatic control as well, and such control is essential to the very definition of SDS. Someone at Amazon made a brilliant decision to make buckets a unit of replication in S3, so they can be linked to a region. In effect this hides the complexity but exposes knowledge that an application needs. Thus, any S3 client can do what Joe coniders SDS, but without any de-coupling, through the namespace and inside the API (or it can chose not to do it and just use a default region, for simplicity).

Joe's employees are hard at work implementing the vision as he outlined it, using the concept of regions that are internal to Swift cluster. The problem for everyone else, however, is how the programmatic control of that stack is exclusive to SwiftStack (with some useful things leaking into Swift, such as changeable replica count).

So, in the end, today Swift offers a solid foundation and parts of an SDS system, but the orchestration is "de-coupled" away elsewhere. Seems like a clear challenge to OpenStack to (re-)create the missing pieces.

P.S. I'd love to see the missing parts inside the Swift API and even namespace, although we have a problem here. Our Accounts and Containers are not guaranteed to live anywhere specifically or even on the same nodes. Changing that would be a step that I prefer. But Joe prefers to give up on plugging programmatic orchestration into the Swift API and just "de-couple" the heck of it. John, our benevolent PTL, seems to toe that line. Maybe they are right.

P.P.S. The deal with the programmatic orchestration is something that "unified" storage projects have to address too. E.g. in GlusterFS a program can issue mkdir(2). Is this programmatic control? No, not enough. Okay, they have glusterfsd nowadays, I can create volfiles in there, is that SDS? That is getting closer!

Dave Jones: PSA: OCZ Vector SSD firmware.

Thu, 16 May 2013 16:16:27 +0000

My bad luck with hardware continues.

At the beginning of this year, I bought an SSD for my laptop

I previously wrote about the need to update smartmontools, which should now be updated everywhere. One thing I was not aware of at the time however, was that there’s a firmware update available. Had I known this, I would have applied it, because as soon as I hit the “400GB of lifetime writes” counter (coincidence?), it lost the ability to write to any block. It won’t even respond to secure erase commands.

The failure is exacerbated by the fact that the disk contains journalling filesystems in need of recovery. So if anything tries to mount them, it tries to write to the disk, and then falls off the bus requiring a power cycle to even see the disk again. The recovery tools provided by OCZ apparently try to mount every partition it finds during boot up (derp).

So now it’s on its way back to OCZ for reflashing/replacement. Lesson learned.

If you have one of these, and hdparm -I shows you have firmware 1.03, you might want to update it to 2.0. There are flashing tools on ocz’s site.
(in the form of bootable linux images, using an insane desktop that looks like what hacker movies in the 1990s looked like). There’s no guarantee that the new firmware actually fixes whatever problem I’ve hit, due to the lack of changelogs, but given it was the first thing they asked me to try, I’m going to say there’s a strong possibility it’s a known bug.

PSA: OCZ Vector SSD firmware. is a post from: codemonkey.org.uk

Dave Jones: CVE-2013-2094. Another day, another fuzzed bug.

Thu, 16 May 2013 15:01:42 +0000

Last month Tommi found a kernel bug in perf_swevent_init using trinity, and posted a fix upstream. This apparently turned out to be a local root. Someone released an exploit for it this week. (interesting dissection of the exploit by spender here).

The code to fuzz perf_event_open was added to Trinity in November 2011. Yet for some reason, we only started to hit this recently. The sanitise routine for this syscall is still pretty basic, even after I added a little more to it yesterday. There’s probably more fruit on that branch somewhere.

There’s a date in the exploit code that claims it was written shortly after the affected code was merged upstream in 2010. Assuming that’s true, it’s taken way too long to find this. Trinity should have found this a lot sooner.

CVE-2013-2094. Another day, another fuzzed bug. is a post from: codemonkey.org.uk

Dave Jones: 3.10rc1 testing status

Wed, 15 May 2013 14:40:13 +0000

3.10rc1 came out a few days ago. At 12,000 changesets, lwn calls it the busiest such ever. Statements like that usually make me nervous. But things are generally in pretty good shape. Much better than 3.9rc1 was.

There has been nowhere near the same level of fallout from trinity this cycle. The only bug I’m reliably hitting has been around for a while (connect vs sendmsg udpv6 oops)
I hit a few crash-in-early-boot bugs that were a pain to debug. (fixes still pending merge)
Some slab corruption found in XFS. (again, fixes pending merge). There’s some talk on lkml about an ext3 issue with the same symptoms, but I’ve not managed to reproduce this (yet?).

and that’s been about it.

Generally feeling pretty solid. Fedora 19 is still going to ship with 3.9, but we’ll likely have a 3.10.x update on day of release.

3.10rc1 testing status is a post from: codemonkey.org.uk

Eric Sandeen: Distributed solar variability

Wed, 15 May 2013 02:39:50 +0000

One of the common arguments against solar as an energy source is that it’s just too variable. You can never count on it when you need it. What if clouds roll in and out? [1]

One counter-argument might be – well, you never know when anyone will turn on their AC, either, at least not minute-by-minute. The grid is a balancing act; unpredictable, random loads have the same effect as unpredictable, random generators.

To which one might then counter yes, but there are so many AC units out there, they average out, more or less, turning on and off at random times and smoothing things out in aggregate.

To which the solar advocate might reply OK, then with enough solar the peaks and valleys of generation should cancel out too, as clouds move out of one area into another. Does this seem likely out in practice?

To find out, I grabbed 5 minute data from about 40 Enphase systems in the twin cities on a highly variable, sporadically cloudy day. Because we don’t yet have a whole lot of solar here, and I didn’t want the one or two large commercial systems in the group to swamp the smaller residential systems, first I normalized them all to a % of their max output. (This might be cheating a little, but with a lot more systems randomly distributed in size and geography, the swamping-out effect should be minimiized.) Here’s what just 4 of those systems looks like; each is indeed pretty messy and unpredictable at the 5-minute range:

Then I averaged all of the systems. Here’s what the average looks like, compared to one of the individual systems:

It appears that things certainly do smooth out when we look at geographically distributed systems. If I were a grid operator, I might feel a lot better about that.

The caveats might be that this is a very wide geographic range – I grabbed systems from all of the twin cities and suburbs. And that’s probably larger than the various sub-grids within the cities; what the variability is within those subgrids is, or how this solar variability affects them, I’m not sure. And of course my initial normalization of all systems to the same size could be argued with.

There have been much more rigorous papers and presentations written on this as well, see for example “Quantifying PV Power Output Variability” by Thomas E. Hoff and Richard Perez in 1999, and “Implications of Wide-Area Geographic Diversity for Short- Term Variability of Solar Power” by Andrew Mills and Ryan Wiser at LBNL in 2010. But with the advent of 5-minute monitoring from systems like Enphase, I wonder if even better results could be found from this wealth of data.

[1] I’ll submit that a sporadically cloudy day is more trouble to a grid operator than a generally cloudy day. We often know if a day will be cloudy well ahead of time, and that doesn’t yield the minute-to-minute variations of a sporadically cloudy day. The grid is better, I think, at responding to these longer-term variations.

Pavel Machek: Slides from openmobility talk

Tue, 14 May 2013 12:54:47 +0000

...are here. Recording was going on, but I'm not sure if it is online somewhere...

James Morris: Slides from my Security Subsystem Overview at LinuxCon Japan 2012

Mon, 13 May 2013 11:14:29 +0000

Whoops. Looks like I forgot to post my slides from last year’s LinuxCon Japan talk on the Linux kernel security subsystem.

Here they are:

http://namei.org/presentations/kernel-security-state-linuxconjp-2012b.pdf

I’ll be giving an update at the upcoming LinuxCon Japan in Tokyo in a couple of weeks.

Dave Jones: Weekly Fedora kernel bug statistics – May 10th 2013

Fri, 10 May 2013 16:18:54 +0000

	17	18	19	rawhide
Open:	273	348	117	73	(811)
Opened since 2013-05-03	7	17	8	6	(38)
Closed since 2013-05-03	7	20	14	4	(45)
Changed since 2013-05-03	20	41	20	7	(88)

Nothing terribly exciting in this weeks new bugs. Backlog continues to slowly get beaten down. Next week should see a rebase to 3.9 for F18.

Weekly Fedora kernel bug statistics – May 10th 2013 is a post from: codemonkey.org.uk

Pete Zaitcev: Viva la testing revolution

Thu, 09 May 2013 20:12:25 +0000

This is not something to brag about, but apparently I managed to program computers for about 30 years without writing unit tests. Today it's recitified by adding a test to one of my projects voluntarily. I encountered the goodness of build-time testing when working on Jeff Garzik's Project Hail. And of course, OpenStack, including Swift, had them since forever. Those weren't my projects, however.

Pavel Machek: From squeeze to wheezy and back, and how not to backup your / filesystem

Thu, 09 May 2013 12:20:07 +0000

I'm still running squeeze on my X60... and I decided that with wheezy becoming "stable", it was good idea to upgrade. Before I started, I did back up my root filesystem (fortunately), with

cp -a --one-file-system / somewhere

Upgrade was a bit of fight (like aptitude trying to take hours of cpu time), but eventually I succeeded... Only to realize that system no longer boots into GUI and (worse) that gnome2 is gone. I'm not great fan of gnome3; definitely on X60, anyway. Its animations feel excessive even when system is unloaded, if there's some background load it quickly becomes unusable. I googled a bit, and it did not look like going back to gnome2 is not exactly easy.

So I went back from the backup. First, chromium refused to run because new version broke the config files. I restored those from backup. But next... strangely my self-compiled 3.9 kernel stopped working. Stock debian kernel kept running, but own kernel ran init then rsyslogd broke the boot.

Can you guess what went wrong?

pc jvgu bar svyr flfgrz bcgvba vf ernyyl onq vqrn; vg jvyy abg pbcl rirelguvat sebz lbhe / svyrflfgrz, va cnegvphyne vg jvyy abg pbcl /qri, orpnhfr gurer'f gz csf zbhagrq bire vg. Bhpu.

Matthew Garrett: A short introduction to TPMs

Tue, 07 May 2013 17:18:31 +0000

I've been working on TPMs lately. It turns out that they're moderately awful, but what's significantly more awful is basically all the existing documentation. So here's some of what I've learned, presented in the hope that it saves someone else some amount of misery.

What is a TPM?

TPMs are devices that adhere to the Trusted Computing Group's Trusted Platform Module specification. They're typically microcontrollers[1] with a small amount of flash, and attached via either i2c (on embedded devices) or LPC[2] (on PCs). While designed for performing cryptographic tasks, TPMs are not cryptographic accelerators - in almost all situations, carrying out any TPM operations on the CPU instead would be massively faster[3]. So why use a TPM at all?

Keeping secrets with a TPM

TPMs can encrypt and decrypt things. They're not terribly fast at doing so, but they have one significant benefit over doing it on the CPU - they can do it with keys that are tied to the TPM. All TPMs have something called a Storage Root Key (or SRK) that's generated when the TPM is initially configured. You can ask the TPM to generate a new keypair, and it'll do so, encrypt them with the SRK (or another key descended from the SRK) and hand it back to you. Other than the SRK (and another key called the Endorsement Key, which we'll get back to later), these keys aren't actually kept on the TPM - the running OS stores them on disk. If the OS wants to encrypt or decrypt something, it loads the key into the TPM and asks it to perform the desired operation. The TPM decrypts the key and then goes to work on the data. For small quantities of data, the secret can even be stored in the TPM's nvram rather than on disk.

All of this means that the keys are tied to a system, which is great for security. An attacker can't obtain the decrypted keys, even if they have a keylogger and full access to your filesystem. If I encrypt my laptop's drive and then encrypt the decryption key with the TPM, stealing my drive won't help even if you have my passphrase - any other TPM simply doesn't have the keys necessary to give you access.

That's fine for keys which are system specific, but what about keys that I might want to use on multiple systems, or keys that I want to carry on using when I need to replace my hardware? Keys can optionally be flagged as migratable, which makes it possible to export them from the TPM and import them to another TPM. This seems like it defeats most of the benefits, but there's a couple of features that improve security here. The first is that you need the TPM ownership password, which is something that's set during initial TPM setup and then not usually used afterwards. An attacker would need to obtain this somehow. The other is that you can set limits on migration when you initially import the key. In this scenario the TPM will only be willing to export the key by encrypting it with a pre-configured public key. If the private half is kept offline, an attacker is still unable to obtain a decrypted copy of the key.

So I just replace the OS with one that steals the secret, right?

Say my root filesystem is encrypted with a secret that's stored on the TPM. An attacker can replace my kernel with one that grabs that secret once the TPM's released it. How can I avoid that?

TPMs have a series of Platform Configuration Registers (PCRs) that are used to record system state. These all start off programmed to zero, but applications can extend them at runtime by writing a sha1 hash into them. The new hash is concatenated to the existing PCR value and another sha1 calculated, and then this value is stored in the PCR. The firmware hashes itself and various option ROMs and adds those values to some PCRs, and then grabs the bootloader and hashes that. The bootloader then hashes its configuration and the files it reads before executing them.

This chain of trust means that you can verify that no prior system component has been modified. If an attacker modifies the bootloader then the firmware will calculate a different hash value, and there's no way for the attacker to force that back to the original value. Changing the kernel or the initrd will result in the same problem. Other than replacing the very low level firmware code that controls the root of trust, there's no way an attacker can replace any fundamental system components without changing the hash values.

TPMs support using these hash values to decide whether or not to perform a decryption operation. If an attacker replaces the initrd, the PCRs won't match and the TPM will simply refuse to hand over the secret. You can actually see this in use on Windows devices using Bitlocker - if you do anything that would change the PCR state (like booting into recovery mode), the TPM won't hand over the key and Bitlocker has to prompt for a recovery key. Choosing which PCRs to care about is something of a balancing act. Firmware configuration is typically hashed into PCR 1, so changing any firmware configuration options will change it. If PCR 1 is listed as one of the values that must match in order to release the secret, changing any firmware options will prevent the secret from being released. That's probably overkill. On the other hand, PCR 0 will normally contain the firmware hash itself. Including this means that the user will need to recover after updating their firmware, but failing to include it means that an attacker can subvert the system by replacing the firmware.

What about using TPMs for DRM?

In theory you could populate TPMs with DRM keys for media playback, and seal them such that the hardware wouldn't hand them over. In practice this is probably too easily subverted or too user-hostile - changing default boot order in your firmware would result in validation failing, and permitting that would allow fairly straightforward subverted boot processes. You really need a finer grained policy management approach, and that's something that the TPM itself can't support.

This is where Remote Attestation comes in. Rather than keep any secrets on the local TPM, the TPM can assert to a remote site that the system is in a specific state. The remote site can then make a policy determination based on multiple factors and decide whether or not to hand over session decryption keys. The idea here is fairly straightforward. The remote site sends a nonce and a list of PCRs. The TPM generates a blob with the requested PCR values, sticks the nonce on, encrypts it and sends it back to the remote site. The remote site verifies that the reply was encrypted with an actual TPM key, makes sure that the nonce matches and then makes a policy determination based on the PCR state.

But hold on. How does the remote site know that the reply was encrypted with an actual TPM? When TPMs are built, they have something called an Endorsement Key (EK) flashed into them. The idea is that the only way to have a valid EK is to have a TPM, and that the TPM will never release this key to anything else. There's a couple of problems here. The first is that proving you have a valid EK to a remote site involves having a chain of trust between the EK and some globally trusted third party. Most TPMs don't have this - the only ones I know of that do are recent Infineon and STMicro parts. The second is that TPMs only have a single EK, and so any site performing remote attestation can cross-correlate you with any other site. That's a pretty significant privacy concern.

There's a theoretical solution to the privacy issue. TPMs never actually sign PCR quotes with the EK. Instead, TPMs can generate something called an Attestation Identity Key (AIK) and sign it with the EK. The OS can then provide this to a site called a PrivacyCA, which verifies that the AIK is signed by a real EK (and hence a real TPM). When a third party site requests remote attestation, the TPM signs the PCRs with the AIK and the third party site asks the PrivacyCA whether the AIK is real. You can have as many AIKs as you want, so you can provide each service with a different AIK.

As long as the PrivacyCA only keeps track of whether an AIK is valid and not which EK it was signed with, this avoids the privacy concerns - nobody would be able to tell that multiple AIKs came from the same TPM. On the other hand, it makes any PrivacyCA a pretty attractive target. Compromising one would not only allow you to fake up any remote attestation requests, it would let you violate user privacy expectations by seeing that (say) the TPM being used to attest to HolyScriptureVideos.com was also being used to attest to DegradingPornographyInvolvingAnimals.com.

Perhaps unsurprisingly (given the associated liability concerns), there's no public and trusted PrivacyCAs yet, and even if they were (a) many computers are still being sold without TPMs and (b) even those with TPMs often don't have the EK certificate that would be required to make remote attestation possible. So while remote attestation could theoretically be used to impose DRM in a way that would require you to be running a specific OS, practical concerns make it pretty difficult for anyone to deploy that at any point in the near future.

Is this just limited to early OS components?

Nope. The Linux kernel has support for measuring each binary run or each module loaded and extending PCRs accordingly. This makes it possible to ensure that the running binaries haven't been modified on disk. There's not a lot of distribution infrastructure for setting this up, but in theory a distribution could deploy an entirely signed userspace and allow the user to opt into only executing correctly signed binaries. Things get more interesting when you add interpreted scripts to the mix, so there's still plenty of work to do there.

So what can I actually use a TPM for?

Drive encryption is probably the best example (Bitlocker does it on Windows, and there's a LUKS-based implementation for Linux here) - while in theory you could do things like use your TPM as a factor in two-factor authentication or tie your GPG key to it, there's not a lot of existing infrastructure for handling all of that. For the majority of people, the most useful feature of the TPM is probably the random number generator. rngd has support for pulling numbers out of it and stashing them in /dev/random, and it's probably worth doing that unless you have an Ivy Bridge or other CPU with an RNG.

Things get more interesting in more niche cases. Corporations can bind VPN keys to corporate machines, making it possible to impose varying security policies. Intel use the TPM as part of their anti-theft technology on education-oriented devices like the Classmate. And in the cloud, projects like Trusted Computing Pools use remote attestation to verify that compute nodes are in a known good state before scheduling jobs on them.

Is there a threat to freedom?

At the moment, probably not. The lack of any workable general purpose remote attestation makes it difficult for anyone to impose TPM-based restrictions on users, and any local code is obviously under the user's control - got a program that wants to read the PCR state before letting you do something? LD_PRELOAD something that gives it the desired response, or hack it so it ignores failure. It's just far too easy to circumvent.

Summary?

TPMs are useful for some very domain-specific applications, drive encryption and random number generation. The current state of technology doesn't make them useful for practical limitations of end-user freedom.

[1] Ranging from 8-bit things that are better suited to driving washing machines, up to full ARM cores
[2] "Low Pin Count", basically ISA without the slots.
[3] Loading a key and decrypting a 5 byte payload takes 1.5 seconds on my laptop's TPM.

comments

James Morris: Linux Security Summit 2013 (New Orleans) – Call for Participation

Mon, 06 May 2013 09:59:59 +0000

The CFP for the 2013 Linux Security Summit has been announced.

The summit will be held across the 19th and 20th of September in New Orleans, co-located again with LinuxCon and Linux Plumbers. Note that presenters and attendees at LSS must be registered as LinuxCon attendees.

We’ll be following a similar format to last year, with a day of refereed presentations, followed by subsystem updates and break-out sessions on the second day. We’ll probably finish up around lunchtime on the Friday for people needing to head home that day, but check the final schedule for details once it’s published.

The CFP is open until 14th June, with speaker notifications to be posted by 21st June.

If you’ve been doing cool and interesting work in Linux security, be sure to submit a proposal!

Andi Kleen: TSX profiling

Sat, 04 May 2013 02:53:22 +0000

I published a quick overview on how to do TSX profiling with Linux perf: Intel TSX profiling with Linux perf

This is a technical overview that assumes some prior knowledge of profiling. I apologize for the cumbersome title.

Dave Jones: Weekly Fedora kernel bug statistics – May 03 2013

Fri, 03 May 2013 16:10:26 +0000

	17	18	19	rawhide
Open:	270	345	126	70	(811)
Opened since 2013-04-26	4	24	9	6	(43)
Closed since 2013-04-26	12	28	8	9	(57)
Changed since 2013-04-26	15	52	18	7	(92)

Weekly Fedora kernel bug statistics – May 03 2013 is a post from: codemonkey.org.uk

Michael Kerrisk (manpages): man-pages-3.51 is released

Tue, 30 Apr 2013 07:31:48 +0000

I've released man-pages-3.51. The release tarball is available on kernel.org. The browsable online pages can be found on man7.org. The Git repository for man-pages is available on kernel.org.

This is a relatively small release that has various fixes across a number of pages. Among the more notable changes in man-pages-3.51 are the following:

Documentation of various /proc interfaces was added in a number of pages.
Various architecture-specific details were added to the syscall(1) and clone(2) pages.

Andi Kleen: Modern locking

Tue, 30 Apr 2013 04:21:06 +0000

I wrote a blog post and contributed to a paper on modern locking on Intel Xeon systems. My recent talk on this has been also covered by LWN here (still behind paywall for a few days)

Summary is more or less: batch your locks. Don’t make critical sections too small. Having the smallest locks is not cool anymore.

Dave Jones: Monthly Fedora kernel bug statistics – April 2013

Mon, 29 Apr 2013 17:41:07 +0000

	17	18	19	rawhide
Open:	274	336	130	66	(806)
Opened since 2013-04-01	31	271	64	16	(382)
Closed since 2013-04-01	37	351	139	19	(546)
Changed since 2013-04-01	55	163	119	27	(364)

Huge number of bug closures this month. Unfortunately several hundred of them are the automated ‘faf’ bugs that were pretty useless.
(1. lots of tainted/virtualbox reports. 2. old kernels. 3. no human attached to them if we need to ask questions, which most of the time we do).
Even discounting those bugs, it’s been quite a productive month, with the total open count around 170 bugs lower than it was a month ago.

Monthly Fedora kernel bug statistics – April 2013 is a post from: codemonkey.org.uk

Dave Jones: Weekly Fedora kernel bug statistics – April 26th 2013

Sat, 27 Apr 2013 17:18:26 +0000

	17	18	19	rawhide
Open:	276	344	126	65	(811)
Opened since 2013-04-19	11	208	26	9	(254)
Closed since 2013-04-19	12	223	64	8	(307)
Changed since 2013-04-19	14	57	34	6	(111)

Weekly Fedora kernel bug statistics – April 26th 2013 is a post from: codemonkey.org.uk

Pavel Machek: What should computers learn

Fri, 26 Apr 2013 13:01:23 +0000

Ok, so I have small herd of computers -- Thinkpad X60, AMD desktop, ARM "tablet", Nokia N900, Nokia 6151, TI Chronos smartwatch. All but 6151 and chronos run Linux, all are programmable, and I have root access on all but Nokia 6151.

Yes, I should probbly get some light and cheap android as 6151 replacement. Light and cheap because it will go on horseback with me, and should have powerful at least 60mW led... because that's important in woods. I'm thinking about Alcatel v860 (old android) or Gigabyte Gsmart g1342 (unsure rooting)... suggestions welcome..

I somehow assumed that having machines under control and running Linux would make them well-behaved, solve world hunger, and make them behave as they do in movies... and yes, it is nice to be able to ssh around and to run same app on most machines.

But happy new world did not happen.

Phones have concept of "behaving themselves", like quiet profile, and I did program automatihc profile switching -- sl4a for android is nice, and that's part of reason I ported it to maemo. Unfortunately, desktops/notebooks do not even have that much... and tend to wake me, or worse (wake my girlfriend).

Another thing PCs should learn from phones is concept of loud speaker and headset. Using headset is pretty much requirement for VoIP telephony, but incoming rings should go to the loudspeakers... even if headset is plugged in. (Currently, I have USB speakers connected for loudspeakers. Yes, twinkle can select soundcards for ringtone and voice separately; but these should be system-level options, not app-level.)

What more would I like? It should be easy to do remote access between my machines; I should not have to search for cell phone to read incoming SMS, or do anything with the small keyboard when I'm near PC. Yes, ssh exists and I use it a lot, and vnc for n900 exists, too... but it has all to be configured separately. When I open web page on a PC and start reading, it should be easy to finish reading on other computer. I now use chromium, so maybe it is doable, but I'd really prefer to use my server, not google.

I'd like my machines to synchronize contact and calendar entries. At least this seems to be mostly working, thanks to owncloud. I'd like machines to automatically back each other up. I have some scripts for that, but they are a bit hacky and far away from automatic. And there's more that should be done...

Dave Jones: USB debug cables.

Thu, 25 Apr 2013 17:27:54 +0000

A few years ago, I was fortunate enough to get given a USB EHCI debug cable. With traditional serial ports being a thing of the past that I haven’t seen on a new machine in a long time, it’s been a lifesaver. The number of kernel crashes I’ve been able to capture through using that cable that would have otherwise been lost is some ridiculously high immeasurable number. I’m saying I like this thing, a lot.

So much so that I wanted to buy more of them, so I could not have to keep replugging it around between test machines.
With multiple test machines constantly running, it’s not really a practical solution.

The first problem, they aren’t cheap. $95 each. For basically two USB->serial chips, and some circuitry to make them handshake.
The bigger problem, is that only one place seems to sell them and they’ve been
“out of stock, and in redesign” for a long time now.

I tried emailing the manufacturer Ajaystech, who seem to completely ignore their sales@ email address.

Disappointing.

In the absence of a replacement, I’m going to have to hope that netconsole works well enough on older machines, and in the future, dumps to pstore.

USB debug cables. is a post from: codemonkey.org.uk