Geek Blight

Many times, the hardest part of creating a computer program is to write the code that deals with user input in particular or the real, outside world in general. Let’s suppose you are coding a program that calculates the inverse matrix of a given one. You may have a smart and elegant algorithm to do that work, and you can create a very beautiful program to implement it. However, the program’s elegance will be spoiled by the code that deals with reading the matrix from a file or from the keyboard. That code is ugly because it’s not a controlled environment. Once you have read the matrix, you have the control. You start operating and, if you do everything right, you only have to catch a few possible errors, if any.

On the other hand, the user may give you anything. You ask them to give you the matrix dimension and the matrix values, and anything can fail. They may input text that is not a number, or a too big number, or too small. Or something that starts with a number but has more text at the end. Or you can reach EOF while reading the matrix data, or its dimensions. What if you read all the values correctly but there’s more text in the file? Do you consider that an error or maybe not, so the user can write comments in the files that store the matrixes? What about displaying a warning? Let’s face it: that code is not beautiful. There are beautiful and smart programs which are all about dealing with user input, like interpreters or compilers, but those are the exceptions. And, even then, the code is many times ugly anyway.

So, how to write good-looking code to deal with user input? I’ll give you my opinion about it. First off, it’s much better when the programming language’s standard library has high level functions to do the work. Ok, usually there’s not a function to say "Read a matrix from this file". But let’s suppose it has a function that can tell you if a string represents an unsigned integer (from the beginning to the end) or a floating point number, and to split a string using blank space as separators. And some functional programming tools to transform lists. You can read the full file to a string and split it. See if the matrix dimensions are valid quite easily, see if you have enough elements to read, and if all of them are valid floating point numbers, and you can create a matrix easily from that list.

The more high level the functions are, the better. A good example of this is the getopt library, to deal with the always unpleasing task of processing command line options. In particular, the Python implementation of it in the module optparse is very flexible and powerful.

Exceptions can also help in these situations. There are people in favour and against exceptions. It’s somehow controversial. Many people prefer functions that return status codes, and do not like exceptions because catching them is slower, and designing and using the different program layers is more complicated. Other people like them, but warn you about creating programs that handle errors with exceptions. If the program has several layers, they tell you, you must create a set of exceptions for each layer, but you should never, under any circumstances, let an exception appear if it’s not in the layer just above. This is not as simple as it sounds. For example, in my experience programming Putmail I found out that the smtplib Python module sometimes raises exceptions from the socket module. That shouldn’t happen, as socket exceptions should be dealt internally inside the smtplib module, and be transformed into smtplib exceptions. In any case, exceptions let you write naive code that supposes everything is correct, without dealing with any error situation, followed by a block in which you capture possible problems. That code looks more elegant, in my opinion, than dealing with exit status codes and having error checks all over your code, which is the usual situation with user input.

In general, it’s not about user input only. It’s about problems that can’t be modeled easily. Being given a large text and seeing if the text fits a very particular format is not a problem that can be easily modeled mathematically. Sometimes you are asked to build a program that needs a database, but the real world situation of the entities and relationships between them can’t be translated easily to an ER model, and you end up with a very ugly database design. And let’s not forget operating system kernels, which have to deal with (sometimes faulty) real world hardware all the time, thousands of different devices and subsystems that try to interact nicely, internal APIs designed to be good enough for a given subsystem, until a new type of device is invented and you need to include more functions, or redesign that internal layer. I’m sure that’s a pain in the ass.

You may know I’m the author of youtube-dl, a very simple Python script to download videos from YouTube.com. This program is essentially about dealing with the real world. Accessing web servers and retrieving data. It’s all user interaction. I won’t say its code is good, because it’s not. But I think I managed to keep it sane, at least. And this is thanks to Python having a module called urllib2 that does all the hard work for me, and internally deals with many different real world errors that are filtered and presented to me in the shape of a very simple group of exceptions I can deal with easily.

Some days ago I made a small code change which inspired me to write about this topic. In previous versions, I downloaded the video data in 10 KB chunks. I chose that block size because it was good enough for dial up connections, and if you had a very fast connection capable of downloading at 300 or 600 KB per second you only needed to iterate 30 or 60 times per second, which is not too much. That was the rationale behind choosing that number. However, I wanted to make youtube-dl more robust and make it able to deal with any connection speed more or less gracefully. I kept 10 KB as the initial block size, but I tried to implement dynamic block sizes. My objective was to make one iteration per second. I would measure the time it takes to download one block. Dividing the block size by the time it took to download it, I expected to find the download speed in bytes per second. As I wanted one iteration per second, the next time I needed to try to download a block of that size. If the speed was 100 KB per second, I’d try to download a 100 KB block next time. It didn’t work.

With my code, I had just stepped over to the real world and I was there to find a painful reality. In the time it takes to process one block, part of the next is being downloaded. Also, the operating system sits in between my application and the network, which as also bad for my initial idea. The result is that, sooner or later (usually sooner), I would ask for a block size that was already available for the most part. The measured time would be very small, which led the program to believe I was able to download several MB per second. Nice thing this happened more sooner than later, because this allowed me to quickly spot the problem in the first test I performed. In the end, the changes I introduced were not very ugly. I put a limit in the speed the block size could change. The size could be doubled or halved, at most, in the next iteration. This allows for exponential growth, which is a very fast change and can adapt quickly to any network connection speed in a handful of iterations, while at the same time providing a filter so I wouldn’t hit that faulty behaviour. As I said, not too ugly, but the initial ideal model failed to represent the problem appropriately.

By the way, don’t you hate it when you are writting a loop (any loop) and you realize you have to put the first iteration out of it? That gets on my nerves.