This is a composite created from three source images: one exposed for the sky, one exposed for detail in the shadows, and one in-between. The composite image retains color information from all areas of the image. Pretty cool, huh?

There is also a ridonkulously big version of the image. For the curious, I used the tutorial from Instructables (intended for Gimp users).

The comment system is particularly painful. Now, given the quality of discourse on YouTube, it’s perhaps of debatable importance that the comment system is horrific to use. On the other hand, perhaps the system is to blame for the poor discussions it fosters.

Example: A couple days ago I was watching M.I.A.’s music video, Paper Planes, and I noticed a comment where someone asked what song she sampled for the beat. Trying to be helpful, I clicked the Reply link for that particular comment, and answered, “The Clash - Straight to Hell”. YouTube helpfully took my comment out of context, and placed it at the top of the comment stream, instead of below the question it answered. (Why it did this, I have no idea. It seems like sometimes replies will be in context, other times they’re treated like normal non-reply comments, and placed at the top.)

Great, now I posted a pretty useless comment, in light of the new placement. It makes no sense as a standalone comment. I notice shortly later that my comment has been given a thumbs down but another user. (Which makes sense, given how little it contributes out of context.) A couple days later, I have this in my inbox, from a miffed M.I.A. fan:

wow seriously….Name a god damn band that hasn’t used a melody or beat from a different song and her representatives must have paid shitloads to use it….dont talk its a very good song not a rip off

YouTube set me up to fail in this instance, and then subjected me to abuse, primarily by having a terrible user interface. Their user experience is so bad that it is actually causing its users pain, in the sense that being sent an angry reply like that is hurtful. Sure, I can laugh it off as a misunderstanding, but, for what it’s worth, it aggravated me enough to write this entry.

But of course, that’s only one problem. In fact, that barely scratches the surface of the mountain of flaws is YouTube. I could detail a lot more problems, but as it turns out, someone else already has, and did a far more thorough job than I could have. If it wasn’t such a damn useful site, I could just, you know, stop going there.

So Google, what’s up? You’re known for being the king of simple, usable applications. When are you going to fix this insanely popular, but badly broken service?

At first glance, it seems like you are actually losing something good when you switch to a dynamic language. You lose compile-time type checking, which is a useful way of detecting errors before ever running the program. In a dynamic language, the compiler (if there is one) doesn’t know the types of any objects until runtime, so it cannot verify the validity of the operations performed on them until execution time. As a beginner in the language, this can be surprising and frustrating. For example, your program may work normally until you hit one less-used branch of the code, and it bombs out with a TypeError.

I’ve had a couple realizations about this. The first is this: If you properly tested your code, there would be almost no paths of code that you haven’t dynamically explored. Ideally, there shouldn’t be any lines of code that you haven’t executed via your testing. In practice, however, there are always edge cases that are hard to reproduce. But this is the second prong of the testing realization: dynamic languages make it ridiculously easy to create mock objects to simulate an environment for the tested object.

The other realization is that all of these errors are presented as exceptions, which you can handle yourself at runtime. This means that, if one of your untested paths has a type error in it, you can handle the error gracefully and keep your application running. Since untested code is broken anyway, it seems to me like you’re not losing very much, if anything, by going dynamic. Finally, tools are evolving that will perform static analysis on dynamic programs and look for the types of errors a compiler would catch (and more). One example of this is PyLint.

In return for having a little less type safety, dynamic typing gives you a pretty powerful new tool in your programming arsenal. Last week, I wrote a Python module that generates a callback API for our C++ application, involving, ironically, the generation of several redundant type declarations, function prototypes, and a bunch of glue to connect the API to a set of function pointers. (Why this was needed is another story.)

The Python module reads in an XML description of the API and generates all of the redundant C++ code. One of the more interesting parts was that much of the client code itself is generated by a C++ application. This meant that my Python objects had to be able write themselves to a file directly, but also write to a C-style quoted string, that could be included by the C++ generator. I solved this in about 5 minutes by writing a QuotedWriter object, that just wrapped a file object (or anything with a write method), and wrapped the given text in quotes.

class QuotedWriter:
    """
    A wrapper around the file class (or any class that has a similar
    write method) that writes text in C-style quoted strings.
    """

    def __init__(self, fp):
        """Create a new writer using the specified file object."""
        self.fp = fp
        self.buffer = ""

    def write(self, text):
        """
        Quote the text and write it to the destination file object.
        Note: Nothing is written until a newline is seen.
        """
        self.buffer += text
        if "\n" in text:
            lines = self.buffer.split("\n")
            for i in range(0, len(lines)-1):
                line = lines[i].replace("\\", "\\\\")
                line = lines[i].replace(’"’, ‘\"’)
                self.fp.write(’    "’ + line + ‘\\n"\n’)
            self.buffer = lines[len(lines)-1]

As an aside, I could have easily delegated all other methods to the fp object, but it wasn’t needed for my simple module. The above code was sufficient. It also escapes quotes and backslashes, and ends each line in the string with a newline, so the generated code looks just as pretty as the generator itself.

Thanks to the power of duck typing, I could give my Python objects a normal File object, or a QuotedWriter object, and they wouldn’t know the difference. They can happily generate their code in either format, and it only took a few minutes.

For the C++ guys (and gals) reading this, just imagine subclassing std::ostream to accomplish the same thing. I just tried, for the purposes of this post, and gave up after 15 minutes, because it still wasn’t working right. (For the record, here’s the code.) It segfaults somewhere in the standard library code, before my write function is ever called. There are obviously nuances to subclassing std::ostream, and I have little desire to dig deeper. Here’s another guy who tried; check out the complexity of the response given by SuperKoko in that thread.

The notion of the power of a programming language is rather vague. When you get used to working in a low level language, it’s easy to think that’s all the power you need. I’ve had the title, “What’s so great about Python?”, sitting on my whiteboard for a while now. I think the simplicity and flexibility of dynamic typing is part of the answer to that question.

First, a quote from the famously eloquent Donald Rumsfeld:

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.”

There’s a real problem among programmers. (Well, there are several, but we’ll stick to just one for now, okay?) We know they’re cocky, but this is generally regarded as a strength. However, it plays badly with the next fact, care of Rummy: they don’t know what they don’t know. There are unknown unknowns.

The thing is, there is a sequence of events that plays out through your entire life as a coder. It goes something like this:

1. Look at the pure crap you wrote a couple years ago.
2. Realize how much you’ve improved since then.
3. Think, “Wow, I’m a top notch programmer now.”
4. In a couple years, repeat step 1.

So, a year after you learned C++ in college, you thought you were on top of your game. You knew how to manage your memory (and dammit, you liked it that way), you created pure virtual interfaces, you’ve internalized Design Patterns and invented a couple of your own. And, because you’re cocky, you figure those Java guys are just dumb because they need a crutch like garbage collection.

Fortunately, programmers have one more quality beyond hubris and laziness: they also get bored easily. After C++ starts to get pretty mundane, you think maybe I’ll broaden my horizons and write a Java program. So you learn the basics, and you decide to write a small utility you’ve been wanting for a while now. You figure it’s probably a two-week job, and start on it one Saturday morning. The next day, you realize, “Wait… I’m done?”

Suddenly, you notice the vast set of standard libraries. The garbage collection doesn’t seem so lame, now. The realization dawns on you, “Well, I guess most of the stuff I write isn’t really performance-critical…” After a little more research, you dig around and find out, to your surprise, that garbage collection is actually getting to be faster than hand-written memory management. And with every iteration, it gets better (some real-time JVMs have an O(1) collector). Not to mention that it’s not nearly as error-prone as managing allocation by hand.

The average person, at this point, will often admit, “Hmm, maybe I was wrong about garbage collection.” (Truthfully, some people never get this far.) But typically, this doesn’t lead to any major epiphanies, and the same person will cling to all the other ego-validating beliefs about everything else, including those pesky unknown unknowns. Something that would have taken the old, C++ aficionado weeks is accomplished in a day; this ought to blow your little mind.

Unfortunately, like that pizza and mountain dew (or is it curry and lattés these days?), clinging to old beliefs activates those pleasure centers in your brain. Eventually, you encounter one of those new, dynamic languages like Python, and snort, “Sure, it’s good for a little script here and there, but no one writes real programs in a scripting language!”

I’ll skip the part where you learn Python (or Ruby, or whatever) and start loving it. By now, it’s predictable, right? Well, here’s the real kicker: for the past half a century, while you were just pecking at the walls of your little object-oriented eggshell, the Lisp and Scheme guys have quietly been producing more and more interesting projects using weird things like closures, first class functions, and macros.

Programmers are people who specialize at seeing the general pattern behind a few concrete instances of something. Well, here’s a general pattern I think a lot of them are missing. I’m personally not very fluent in functional languages (yet), but I refuse to believe that, just because I’ve survived without many of the functional concepts so far, that they’re not awesome. At some point, a software developer needs to realize that there are bounds to one’s knowledge, and they are the bars of a cage that keep one from doing something amazing.

Hmm, what’s this Clojure thing all about…

First of all, the updater application worked well. I had to expand the window to show the terminal when it updated OpenVPN, which would hang waiting for me to enter my username and password. (This actually happened twice). Also, it is occasionally interactive, asking you to look at changes it wants to make to certain config files. This is a good thing, but it would be nice to have all the questions come at the beginning or the end, so the update can be unattended (since it takes several hours).

System doesn’t boot

The moment of truth: I allow the updater to restart my computer. Grub appears momentarily, then it tries to start the 2.6.24 kernel. Kubuntu’s splash screen appears and the progress bar pulses back and forth, seemingly forever. I give up after seeing no HDD activity, and reboot the machine via the keyboard. I switch back to kernel 2.6.22, and the system boots, but has some issues with nVidia’s driver and X.org.

Ignoring these for the moment, I disable the boot splash screen and the quiet boot options in my /boot/grub/menu.lst, and reboot. The kernel outputs a bunch of messages like:

ata3: failed to IDENTIFY
ata3: failed to recover some devices

It retries this several times, and then dumps me to a busybox shell. So far, I’m not impressed with this update. Ubuntu is supposed to be the OS that anyone can use, and it’s taking an awful lot of my Linux experience to figure this out. After googling the messages, I find this bug report, which turns out to be the problem. The solution is to add the option “pci=nomsi” to my boot parameters. I readily admit I have no idea what this does, but it fixed my problem. The 2.6.24 kernel now boots cleanly.

Files overwritten

Even though the updater was kind enough to ask my permission to update some files, there were a couple cases where allowing it seemed like the right idea. I allowed it to update my php.ini and my xorg.conf files. It screwed them both up. So I had to fix my modelines for my display (detected native resolution of 1792×1344 on my 1920×1200 monitor… wtf?), and fix my php.ini to get my wiki working again.

Firefox 3.0 Beta

Apparently, they decided to ship a beta version of Firefox with this new release. It’s sort of logical from the point of view that this is an LTS release (long-term support), which means the Ubuntu crew will support this release for the next three years. They don’t want to be supporting Firefox 2 for that long, and that makes sense. However, as an end user, I specifically held off on Firefox 3 because it’s still beta. Even Firefox 2 still feels like a beta product, with its memory usage pattern and frequent locking up on pages with Flash or heavy Javascript use. Now I’m an FF 3 beta user, let’s hope that goes well.

Update June 23, 2008: Firefox 3.0 eventually came out officially. Well, it’s driving me nuts. The flash plugin dies five or six times a day, requiring me to restart the browser. Also, the UI locks up during heavy disk activity (like when I’ve got a big compile job going at the same time), and freezes for 5 to 10 seconds at a time. Sometimes, it never wakes up. Today, I’d finally had enough ,and installed the “firefox-2″ package, and I’m happy again. I can’t wait for a nice, small WebKit browser.

Tracker

After the update, my system seemed slower than usual, especially launching applications from a cold start. top reveals a process called trackerd as the culprit. Tracker is some kind of filesystem indexing program, to optimize searches. It seems like it’s a Gnome product, so it really shouldn’t be running on my KDE system. Well, after the update, sure enough, trackerd is running, and consuming a ridiculous amount of resources.

Trackerd was steadily using 15 to 25% of my CPU, and as far as I can tell, a much higher percentage of my hard drive bandwidth. Killing it from the command line sped up my desktop, but it rises from the dead shortly thereafter, hogging my resources all over again. Googling turned out to be not very helpful, but I ultimately figured out how to disable it: run gnome-control-center from the command line, under “Other” open “Searching and Indexing”, and in there, you can disable the indexing service. It seems to be gone for good now; we’ll see.

GTK glitch

For some reason, checkboxes and radio buttons in GTK applications (such as Firefox) don’t show their state when focused. This means you click a checkbox, and it doesn’t appear checked until you click somewhere else. Terrible. I fixed this by opening the KDE System Settings → Appearance → GTK Styles and Fonts, and changing to “Use another style”. Several of them appear to work fine, and I settled on “Glossy” as a moderately attractive style. Ubuntu has an open bug for this issue.

Crappy fonts

A lot of the text I’m seeing now looks worse than it used to. I’ve never been an expert in the Linux font area, so I don’t know what changed. But the default anti-aliasing looks super fuzzy:

I tried disabling anti-aliasing, and it looks even worse:

Not all fonts look this bad, but all of them seem to look at least slightly worse after the update. Check out the kerning in “Select which”, great stuff. This is the kind of thing that tempts me to get a Mac.

Update: Found this reddit thread (total coincidence that I was using reddit as my example) via the Ubuntu forums, and his solution fixes my sans-serif fonts in Firefox. After much trial and error, I also substituted “DejaVu Serif” for “Liberation Serif”, and my serif fonts look much sharper. This is how reddit looks now, with the sans-serif fix:

RAID rebuilding

For some reason, as I write this, my RAID arrays are currently rebuilding. All four, in sequence. This could be due to the boot issue where the ata3 and ata4 devices weren’t detected, so it was perceived as a RAID drive failure. I’m not experiencing any data loss or stability issues as a result, but it does affect HDD performance. The arrays appear to be in good shape.

Conclusion

Make sure you have a day to waste when you upgrade to hardy. I suspect a lot of my issues are due to KDE being a second-class citizen in the Ubuntu world. There also seems to be the mindset that Ubuntu absolutely has to release every April and October, regardless of the amount of testing it has received.

I give a lot of credit to the Ubuntu team for keeping this great OS going, but the bottom line is it’s still Linux, and it’s still a maintenance nightmare.

I updated our Linux build machine with the new nightly build script. At the same time, I was configuring our new, faster Windows build machine to replace our old, pokey one. Since the old Windows machine is expected to be rendered obsolete, I didn’t bother installing the new build script on it. So Windows just continued running the normal build, unit tests, and system tests. We knew the integration tests worked on Windows anyway, because we ran them by hand just recently (right?).

This Sunday, about a week later, the new Windows machine came online. I installed all the prerequisites (which only took a few hours, because we have an awesome wiki full of HOWTOs), connected it to our network, and installed the updated build script. By now, you know what happened next: I ran the build script, and everything worked except the integration tests. The code checked out, compiled, unit tested, and system tested just fine. But the one thing that wasn’t tested nightly — the integration test suite on Windows — was broken.

Our strategy of automating everything continues to pay dividends. And it shows what happens when you don’t. If you aren’t able to regularly test your software in some kind of automated fashion, it is broken. You just don’t know it yet.

Title is a quote attributed to Bruce Eckel.

• How I Built a Working Poker Bot, Part 1
• How I Built a Working Online Poker Bot, Part 2: Interlude

It’s reminiscent of the days when IBM proved the power of computer software by facing off against chess grandmaster Gary Kasparov. Kind of exciting in that sense. According to the author, their artificial intelligence isn’t better than the best poker players… yet.

I’ve personally written one program that plays an online flash game, by screen scraping and mouse pointer movement. It didn’t require any AI component, and it was purely for fun; no profit motive. Needless to say, this is intriguing; of course, I know I’m supposed to use my powers for good.

Website note: I’m opening up comments generally. I noticed Wordpress gives me a neat summary of recent activity, so I can delete spam pretty easily.

“Whenever I see an adult on a bicycle, I have hope for the human race.” — H. G. Wells

For example: In my continuing efforts to be more environmentally conscious, less oil-dependent, healthier, and a bit more frugal, I’ve taken to bicycling to nearby destinations whenever possible. In many places in the world, this is the easy and obvious solution; after all, the bicycle is the most efficient known form of transportation. But things are different in the land of sprawl. In a country that, for the last 50 years, has designed its spaces around cars, attempting to use anything else can be a harrowing experience. Not only that, but many seem to take offense at bicyclists, shouting profanities or honking as they drive by. Whether they’re completely ignorant or just living in our familiar state of cognitive dissonance, who knows.

The writing’s been on the wall for as long as anyone’s been willing to look. Oil production has probably peaked in the last year or two, and will soon begin a series of declines, each worst than the last. The environment is changing in ways that no one completely understands, but most agree is for the worse. The suburban way of life is going to fade out, an anomalous period of history fostered by the era of cheap energy and insufficient foresight.

Yet, I’m still not sure how I’ll ever manage to live in a way that is balanced with nature. Riding bikes, switching to fluorescent lights, reusing grocery bags, even changing my diet; they seem like token gestures rather than true improvements. If everyone lived like me, we’d still need the resources of multiple planets to fulfill the demand. Not only that, but what I consider modest behavior changes, many still consider extreme! Doing anything other than throwing money at a problem is practically unamerican. Almost existentially, I begin to question the point of modifying my behavior in a culture that still chugs bottled water like, well, water. How am I supposed to care when my actions are regarded with disdain by the culture I’m purporting to help?

No matter what I do, things are going to change in my lifetime in ways I can’t anticipate. With news of food rationing and droughts, and the knowledge that we use 10 calories of hydrocarbon energy for every calorie of food we produce (not including transport or cooking), energy scarcity may be the least of our worries. The next wars may be fought not over oil, but clean water and arable land. But today, we misguidedly continue to repurpose farms as fuel supplies, converting food into ethanol while billions starve.

Here’s a fantastic talk (approx 20 min) by James Howard Kunstler, “The Tragedy of Suburbia” (ironically sponsored by BMW):

Some worthwhile documentaries on the topic:

• The End of Suburbia
• What a Way to Go

There are no quick fixes, and the difficult fixes that should’ve begun years ago haven’t yet. These topics always leave me feeling uneasy, never able to come completely to grips with reality. As we continue down the irrational path of consuming the planet that gave us life, I wonder, how do you come to terms with this? Consciously or not, we know that the world isn’t going to bear the strain much longer. It probably explains the hopes pinned on a Rapture-event, an ascension to a higher plane of existence. Because what else do you do, when you’ve burned up all you have?

Comments are open on this entry, if anyone has any thoughts to share.

This article is part of a series.

Hopefully you have a pretty good idea of what is meant by the term dynamic range now. Let’s get a little more hands-on.

Histograms

Figure 1: An extremely simple histogram.

First, you need to know how to read your image’s histogram. A histogram is a 2-dimensional graph of the frequency a particular value occurred. Remember our 8-bit cameras, with a grid full of pixels whose values could be anywhere from zero to 255? Now it’s time to start inspecting those values.

Imagine a really crappy black & white camera with only a 2-bit range, capable of storing values between zero (black) and three (white). If you took an image with this camera, it would look remarkably like something from the old GameBoy, having only four shades of gray to display. The histogram for that photo might look something like the example to the right.

The histogram displays the number of pixels in the image with each value. So maybe 50 pixels had the value zero, completely black. Then about 25 pixels were a dark gray (value=1), about 75 pixels were light gray (value=2), and maybe 90 pixels were white (value=3). This gives you a sense of what information a histogram is technically conveying, on a very simple level, but we’ll need a better example to start learning how to use histograms to improve our photos.

Figure 2: A real world histogram.

In most cameras and photo editing packages, histograms look more like the one displayed in Figure 2. First of all, there are no numbers or indications of scale. Second, the range of values is much larger than the one above, so the graph looks more like a curve than a bar graph. As a further complication, all three colors — red, green, and blue — are represented on the same graph. Confused yet?

So what is this histogram telling us? It seems that the majority of the pixels are bunched up at the low end of the range, meaning the majority of the picture is composed of darker pixels (including a whole lot of completely black ones). There are a moderate amount of middle range pixels, and then essentially zero bright values. (The lighthouse is definitely overexposed, but it’s such a small amount of pixels that you don’t even see the blip on the far right end of the histogram.)

Spend a minute reading this histogram and comparing it to the image it was derived from (at the left). Do this again when you start working with your own images — pull up the histogram and spend a minute studying it. Get comfortable with what the histogram is telling you, that experience will help you take better pictures in the future.

Note that in an RGB histogram like the one in Figure 2, the colors overlap. You will eventually get familiar with how colors add together, but for now, just know that red + green = yellow, blue + green = cyan, and blue + red = magenta. Areas where all three colors are combined would be white in the image, but for some reason, Gimp decided to use black in the histogram. On cameras that provide a histogram, the background is usually black, so all three colors add up to white, as you’d expect.

Setting black- and white-points

In the previous post, we talked about how the camera’s dynamic range can be too small to fit all the detail into an image. This is particularly common with a high contrast scene, such as the sunny day example. Well, just as often, the range is sufficiently big to capture everything. This happens in situations where you have lower contrast, and your subject(s) are lit relatively evenly.

In this case, you’ll often find that the entire range of brightness fits into your camera’s range, and the histogram shows this in the example below. The top half shows the image as the camera recorded it, and the corresponding histogram. Reading the histogram tells us that the darkest part of the image has a value above zero, and the brightest part of the image is well below maximum. The result is a flat, grayish image, as in the top half of FIgure 3.

Figure 3: Image before and after levels adjustment.

Now you might be tempted to find some kind of “contrast” knob, like you were fixing the picture on a TV. Both Photoshop and Gimp give you a contrast/brightness tool, and you probably found that already when you were trying to fix your flat images in the past. The problem is the contrast tools don’t give you any sense of what you’re doing to the values in the image. You don’t know when you’ve reached the point of clipping in either direction. You also don’t know if you’re still far from that point. You can achieve decent results, but they’ll almost always be less than optimal.

The correct tool to use for this job is the Levels adjustment. The bottom half of Figure 3 shows the image after I’ve adjusted the levels, setting true black- and white-points. You can see that the corresponding histogram has been stretched to fill the entire range of values now, so that the darkest points in the image are truly black, and the brightest points of the image are at the maximum brightness in the image.

The effect is dramatic, and truly brings life back into what was a pretty stale image. In fact, this is generally the first thing I do to any image I plan on developing. Note that you don’t always want to bring the bright end of the spectrum up, it depends on the subject and your goals. But setting a black-point is pretty much universally useful.

To set black- and white-points in your images:

In Photoshop, under the Image » Adjustments submenu, you have several options. You can try Auto Levels, Auto Contrast, and Auto Color, to see if they produce the results you’re looking for. These are the easiest options, but give you the least control. Auto Constrast, in this case, gives basically the result I was looking for. The other options affected the colors too much for my taste, so that the sky lost the reddish hues.

Alternatively, you can select Image » Adjustments » Levels (or hit CTRL-L), and adjust the levels by hand. Then follow the instructions below for Gimp, as the Photoshop levels dialog behaves almost identically.

Figure 4: Adjusting levels in Gimp

In Gimp, right click on your image, and select Colors » Levels. In the dialog that appears, slide the two arrows circled in Figure 4 until they frame the range of values in the histogram. I’ve marked the locations I used in this example with red arrows. As long as Preview is checked, you can watch the image change as you adjust the black- and white-points. You can also toggle the Preview checkbox to see how much you’ve changed the image from the original.

After you click Ok, the colors in your image will be scaled to the new range you specified. Values lower than or equal to the left arrow will be completely black, and values equal to or higher than the right arrow will be at their maximum brightness. All the values in between will be scaled proportionally.

Getting to know how the dynamic range affects your pictures is an important step in producing better photographs. As you get more comfortable with histograms, you’ll be able to produce better pictures simply using the histogram display on your digital camera. Additionally, when you get the pictures home, you can create a more pleasing range of values using the “levels” capability of your image editing software.