strcpy data corruption on Core i7 with Linux 64bit

If you’re C programmer, does this code look OK to you?

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char* argv[])
  char szBuffer[80];
  strcpy(szBuffer, "abcdefghijklmnopqrstuvwxyz");
  printf("Before: %s\n", szBuffer);
  strcpy(szBuffer, szBuffer+2);
  printf(" After: **%s\n", szBuffer);

  return 0;

Here is the output on my server, a Core i7 running Debian 6:

Before: abcdefghijklmnopqrstuvwxyz
After: **cdefghijklmnopqrstuvwzyz

What the program does is dropping two characters from a text string in a buffer, moving the rest of it left by two characters. You expect the moved characters to stay in sequence, but if you compare the last three characters of the output you see that that isn’t the case. The ‘x’ has been obliterated by a duplicate ‘z’. The code is broken.

It’s a bug, and not a straightforward one, as I’ll explain.

I first came across it a couple of months ago, as I was moving some code of mine from an Athlon 64 Linux server to a new Intel Core i7 server. Subsequently I observed strange corruption in data it produced. I tracked it down to strcpy() calls that looked perfectly innocent to me, but when I recoded them as in-line loops doing the same job the bug went away.

Yesterday I came across the same problem on a CentOS 6 server (also a Core i7, x86_64) and figured out what the problem really was.

Most C programmers are aware that overlapping block moves using strcpy or memcpy can cause problems, but assume they’re OK as long as the destination lies outside (e.g. below) the source block. If you read the small print in the strcpy documentation, it warns that results for overlapping moves are unpredicable, but most of us don’t take that at face value and think we’ll get away with it as long as we observe the above caveat.

That is no longer the case with the current version of the GNU C compiler on 64-bit Linux and the latest CPUs. The current strcpy implementation uses super-fast SSE block operation that only reliably work as expected if the source and destination don’t overlap at all. Depending on alignment and block length they may still work in some cases, but you can’t rely on it any more. The same caveat theoretically applies to memcpy (which is subject to the same warnings and technically very similar), though I haven’t observed the problem with it yet.

If you do need to remove characters from the middle of a NUL terminated char array, instead of strcpy use your own function based on the memmove and strlen library functions, for example something like this:

void myStrCpy(char* d, const char* s)
  memmove(d, s, strlen(s)+1);
  char szBuffer[80];
  // remove n characters i characters into the buffer:
  myStrCpy(szBuffer+i, szBuffer+i+n);

I don’t know how much existing code the “optimzed” strcpy library function broke in the name of performance, but I imagine there are many programmers out there that got caught by it like I was.

See also:

Solar energy, USA vs. Germany

Recently I came across an article that quoted a Forbes commentary (“Sue OPEC? Congress Should Sue Itself”, 2008-07-09) comparing solar energy development in the USA (or lack thereof) with the situation Germany, where 2010 was a veritable boom year for photo-voltaic panels.

Two maps and one quote underneath caught my attention:

Check out the map above. With the exception of Seattle, the entire continental U.S. is much sunnier than Germany. Yet Germany has 17 times the installed solar base per capita.

According to the map, Germany received amounts of sunlight comparable to the region around notoriously cloudy Seattle and arctic Alaska, while most of the states along the Canadian border got 50% more sun than the Southern half of Germany.

This did not seem plausible to me. While it’s true that most of Germany lies further North than the 49th parallel that marks most of the US-Canadian border and should therefore receive less sun than most of the US, most of Germany’s climate is far sunnier than Seattle, which lies about as far North as Mannheim or Nuremberg (Nürnberg) in Southern Germany. Based on latitudes and annual rainfall, solar insolation (the amount of solar energy radiated onto a given area) should be largely comparable between at least southern Germany and the northern US outside the Pacific Northwest. I’ll give you some data to verify this theory.

Here is a map of insolation for the entire US, showing kWh per square metre per day at latitude tilt (multiply by 365 for annual figures like in the Forbes map):

As you can see, most of the US gets between 4 and 5.5 kWh/day (yellow-grey to dark yellow), or 1450-2000 kWh per year.

And here is a map of insolation in Germany, showing horizontal irradiation in kWh per square metre per year:

Note that the colour scale is not the same. The southern part of Germany gets 1200 kWh and more per year, the northern part less than that.

But that’s not the whole picture. If you you paid attention, you noticed the “at latitude tilt” (US) versus “horizontal irradiation” (Germany). It makes a big difference, because without taking it into account, the comparison of the raw numbers become an apples to oranges comparison: The numbers in both maps don’t actually measure the same thing!

The further you move north, the lower the sun stands at midday to the South. Consequently, when you install solar panels anywhere but in a tropical country, you don’t install them horizontally but make sure to tilt them at the right angle to catch the most sun per square metre of expensive panel, based on the average position of the sun at noon throughout the year, which depends on your distance from the equator. It will be more tilted at a more northern location than somewhere further south.

The US solar data is measured per square metre of panel. The German data however is per square metre of shadow the panels cast on the ground, which is not the same. The two ways of measuring insolation only match at the equator.

Let’s look at an example: In Chicago, at around 42 degrees North, a solar panel tilted 42 degrees towards south is exposed to 37% more sun than a flat piece of lawn of the same size (1/cos(42 degrees) = 1.37). So a one square metre panel’s 1500 kWh/year in total solar irradiation in Chicago is basically the same as 1100 kWh of horizontal exposure, which is the same or less than you catch just about anywhere in Germany. Far from being comparable only to rain-swept Seattle, Germany’s annual exposure to the sun is actually not too different from the US east of the Mississippi, except for the Southern sunbelt from Texas to Florida, which does get more sun.

So what do we learn from this?

1) As our physics teacher always used to tell us: Watch your units! Numbers don’t tell you anything unless they go with the proper unit.

2) The German solar subsidy program is no crazy boondoggle at taxpayer’s expenses. Solar energy does make sense in Germany, as it does in most other countries in temperate climates.

Of course it makes even more sense in dry climates like Spain, Turkey, North Africa and the arid US Southwest, where it’s ideal. On the other hand, Seattle or Iceland may not be the greatest places for it.

Ideally, solar energy investment should start where it brings the highest returns, accompanied by sufficient investment into a power grid that can carry power wherever it’s needed.

In 2010, Germany produced about 1.9% of its electricity usage from solar panels, almost double what it was the year before. By 2020 this is expected to more than triple again, to 40 GWh or 6%. As mass production brings costs down while oil prices keep going up, German solar electricity is expected to become cost-competitive with conventional power by about 2015, which will be great for cutting dependence on dwindling oil reserves.

Also, let’s not forget that solar energy is not the only renewable game in town: Wind for electric power is already much more competitive on a cost per kWh basis than photovoltaics and deserves more attention. Many regions that have relatively poor solar prospects have excellent wind opportunities, for example Scotland. A mix of solar and wind often works better than just one or the other.

Germany has great potential for wind power, both land based and off-shore, and so do the US. What is holding wind power back particularly in the US (but to some extent also in Germany), is an under-sized grid that can not move enough power over the necessary distances from areas with plentiful wind to where the power is needed. Investment in a 21st century power grid will be essential for a low-carbon future.

Bike Friday Pocket Rocket

Yesterday I ordered a new bicycle, a Bike Friday Pocket Rocket. It’s a folding bike that rides like a good road bike.

My house in Setagaya, Tokyo is tiny and the three other members of the family already own four bikes between them, so we’re really short of space.

A while ago an Alex Moulton bike in a park in Tokyo caught my eye and that started me looking at small-wheeled fast bikes.

Moulton bikes have suspensions and use small wheels (20 inch or smaller). Some of them use a space frame, like on the TSR model pictured above and can be separated into two halves for transport or storage. Unfortunately, some Moultons are obscenely expensive (for the list price of the top of the range Moulton Double Pylon — JPY 1,795,500 — you could almost get a Honda Insight hybrid car). A Moulton TSR-30, the model I would have been most interested in, sells for JPY 390,000 (about US$5,000) over here, which is way over my budget limit.

Therefore I started looking at various other small-wheeled bikes, including Dahon, Birdy (Riese & Müller, Germany) and Bike Friday (Oregon, US).

I then test rode a Bike Friday Pocket Rocket at Amanda Sports (one of their dealers in Japan) here in Tokyo last summer and liked it. Bike Friday bikes are definitely not cheap either, but the bikes are well designed and well made and carry a life-time warranty on the frame. The company is famous for its customer service. They use a lot of standard parts to make sure their bikes are easy to repair and in all the bike forums I visited, I never read a single bad word about them from any of their owners: Their customers seem very loyal.

I probably would have ordered a Bike Friday this spring, but then the M9.0 Tōhoku earthquake and tsunami hit Japan and triggered the Fukushima nuclear disaster, prompting me to take my family abroad until the situation in Japan had stabilized again a bit. I returned and now I finally placed the order.

With the help of one of Bike Friday’s bicycle consultants, I selected a 27-speed model using Shimano STI integrated shifters, with drop handles, narrow high pressure tires, fenders and a kick stand, in a colour called “Enno Orange”.

The bike will be delivered in early October. Stay tuned for pictures and ride reports with even more pictures!

Cycling since high school

Though born and raised in Germany, I’ve never been interested in football (soccer) or anything that involves a ball or opposing teams. The only sports I really enjoyed were swimming, walking or running longer distances and cycling. These are sports that you can do not only when you’re young, but throughout your life.

When I was in high school I bought my first 10-speed bicycle, a French Motobecane with drop handles. During one summer I did 7,500 km (4700 miles) over a period of 150 days (50 km/day average). I enjoyed going for rides of several hours, especially on weekends. My longest single day distance was 220 km.

Eleven years ago, when I moved to Yokohama, I bought a mountain bike. I gave it away to friend when I moved to my small house in Tokyo last year, because I had no space for it. Since then I’ve been using either my wife’s mamachari (housewife’s bike) or a small rusty foldable with squeaky brakes that we originally got as freebie from SoftBank Mobile when we signed up for iPhone contracts (seen folded in the centre of the picture of my home at the top). I am not a bike snob. I’ll ride just about anything with two wheels and usually the only kind of bicycle wear I put on is a proper bicycle helmet.

In Tokyo, a bicycle will often get you from A to B almost as quickly as a car (sometimes quicker) and you won’t be paying an arm and a leg for parking, plus it’s more fun than driving a car on our notoriously congested city roads. That’s why I do most of my shopping by bicycle these days.