Sloppy APIs and getrandom

Posted on .

Not long ago, the getrandom system call was introduced to Linux with a patch from Ted Tso. It was included for the first time in kernel 3.17, released a few days ago. It attempts to provide a superset of the functionality provided by the getentropy system call in OpenBSD. The purpose is having a system call that will let you obtain high quality random data from the kernel without opening /dev/urandom. This helps when the process is inside a chroot and protects the process from file descriptor exhaustion attacks.

If you’re a C programmer and as paranoid as I am about clean APIs and integer types, you may have noticed getrandom is a bit weird in this regard, and you’ll also notice getentropy is much cleaner. getrandom takes a buffer pointer and the number of requested bytes as a size_t argument. size_t is unsigned and is supposed to be able to represent the size of any object in the program. Specifically, size_t is usually an unsigned 64-bits integer in the typical 64-bit Linux system.

However, the return type of getrandom is a simple integer (32-bit, signed) that may indicate the number of bytes that were actually read. Out of context, I think such an interface is a bit sloppy. The call to getrandom may result in a short read just because it wouldn’t be able to return the proper result due to data types. Short reads are possible in other situations with getrandom, but are mostly mentioned in the context of passing a flag to read from /dev/random instead of /dev/urandom.

In context, it does make sense. If you request a very large number of random bytes you’re going to wait a long time while they are being computed. This may catch you by surprise. So it probably doesn’t make sense to request a large number of random bytes in the first place. In fact, if you check Ted Tso’s patch, you’ll notice getrandom returns an error if the requested size is over 256 bytes.

The interface in OpenBSD solves all these problems right away. First off, the manual page mentions you shouldn’t be using it directly, and provides references to better APIs. In any case, an error will be returned if you request more than 256 bytes (minor complaint: provide a named constant for this just in case the value changes in the future). This is mentioned explicitly in the manual page. If not, there will be no short reads, and the integer value returned will only be used to signal errors, and not to tell how many bytes were actually read. Super-clean. If you want to request more than 256 random bytes, you’re responsible of splitting the requests over a loop, but generally you won’t need to do so. 256 bytes are "more than enough for everybody".

Contrast that with getrandom. Even if you request only 4 bytes, nothing in the API tells you there won’t be a short read. In practice, the implementation will not result in short reads for requests below 256 bytes with the default behavior, sure, but the API leaves that open. So you end up having to code something like this if you want a direct equivalent to getentropy that’s guaranteed to work well (note: coded on-the-go and not tested).

int getentropy(void *buf, size_t nbytes)
{
        if (nbytes > 256) {
                errno = EIO;
                return -1;
        }

        int ret;
        size_t got = 0;

        while (got < nbytes) {
                ret = getrandom(buf + got, nbytes - got, 0);
                if (ret < 0)
                        return ret;
                got += ret;
        }

        return 0;
}

Specifically, the simple getentropy equivalent from Ted Tso will work in practice, but it’s not solid according to the proposed API.

I think it’s unfortunate that a system call may fail, partly, just because of API data types. The surprising part is this happens for some common system calls too. The history trail is not specially good. For example, take a detailed look at the common standard read system call. It takes a size_t argument to indicate how much to read, yet returns the amount that was really read as a ssize_t, which is signed (hence the initial S) and only allows representing numbers half as big. This allows returning -1 to signal errors, but then you have to read further to notice that -1 will be returned and errno set to EINVAL if the requested size is larger than SSIZE_MAX.

The standard write system call has a similar problem. However, not all manpages in every system mention the SSIZE_MAX limitation. For example, in my Linux system the manpage for read mentions it, but the manpage for write does not. It may not have that limitation (I haven’t checked), but if you’re coding portably, the manpage won’t help you detect the problem.

In my very humble opinion, having the requested size as a size_t is very practical because it allows the following kind of code to work without casts.

struct foo a;
...
ssize_t ret = write(fd, &a, sizeof(a));

But would it really hurt to have one more argument to separate the error signaling from the amount of bytes written in a short write?

int write(int fd, void *buf, size_t nbytes, size_t *wbytes);

In systems and programming languages with exceptions this is not a problem. Real errors are signaled by raising exceptions and the API is not polluted. write could return size_t just like its input argument (note: no intention to start a flamewar about exceptions vs error codes). In the Go programming language, functions many times return two values, one is the normal return value and the other one is an error code.

End of the rant.

Typing this from a new computer

Posted on .

Last week I upgraded my computer changing everything except the hard drives. I’m typing this text from the new system already, having migrated my Linux installation to it, and having reinstalled Windows.

I always thought my previous system ran close to perfection, and the only reason for the upgrade was gaming. I wanted a new graphics card and, with it, I didn’t want other part of the system to be a performance bottleneck, so I ended up upgrading it almost completely. I’m now using an Intel Core i5-4690S, 8 GiB of RAM and an EVGA GTX 760 Superclocked ACX. The performance is just amazing, and the price very fair.

As I mentioned, I also migrated my Linux installation to this new computer. By migrating I mean I continue running the same system without reinstalling, preserving the state of the OS and personal data. It’s the third time I did such an operation, and it’s been at least ten or eleven years since I installed this system. I’ve been running the rolling release installation of Slackware Linux since then, across several computers.

I’ve never fully documented my migration procedure. The following notes serve mainly as a reference for myself in the future.

Kernel preparation

I start from the old system by first removing all custom kernel modules I have installed, which these days is mostly the binary NVIDIA driver. Then, I switch from a modular kernel with an initial RAM disk to a huge everything-included kernel that would be suitable to boot any system, including the new one for which I still don’t know which modules are essential to boot.

In Slackware there’s a specific package for that called kernel-huge. Just in case it’s useful, the kernel config and image file are easily available. I proceed to boot my old system with this kernel and without an initial RAM disk to check everything is fine.

Data transferring

I then proceed to transfer all data to the new computer. In this case it meant just plugging the old hard drive in the new computer, but it depends on the situation. The previous time I used an external USB hard drive to copy data from a laptop to my now-old desktop computer, which was booting from a Live-CD or Live-USB. If both systems have Gigabit Ethernet ports, a crossover cable can be a fast solution too.

In this step you may have decided to change the partitioning scheme to be used in the new system, so a bit of data juggling and temporary mounts may be needed. Once the new system has the data, 80% of the work is done and only the other 80% is left.

Finishing touches and booting

Time to try to boot the new system. For that, I usually chroot into the new system from a Live-CD or Live-USB. I first attempt to configure the bootloader correctly. I always install GRUB (these days v2) in the MBR of the appropriate hard drive the BIOS will try to boot. This will probably change with EFI, but I still run my system in Legacy BIOS mode (take into account my gaming OS is Windows 7).

The hard drive order as detected by the BIOS or GRUB may not match the order in the kernel. This happens in my new computer. hd0 and hd1, as detected by GRUB, match to /dev/sdb and /dev/sda respectively. I adjust the GRUB config file accordingly.

After installing GRUB, I pay attention to these other files and tweak everything until the system boots correctly.

  • /etc/fstab, paying attention to the new partitioning scheme, if any, and the hard drive names.

  • Remove autogenerated persistent udev configuration files. In Slackware they sit in /etc/udev/rules.d.

  • Tweak or disable personal commands run at boot time from rc.local and other scripts (no systemd yet in Slackware). Some of those may be related to specific kernel modules or hardware tuning.

  • Review the /etc/smartd.conf configuration file for possible changes or disable SMART until the new system is ready.

Post-booting

At this point I’m happy because I should be running my system in the new computer.

The next step is using mkinitrd-command-generator.sh from Slackware team member AlienBOB to find out which modules I need to boot the computer, and reverting to a modular kernel with an initial RAM disk.

Finally, re-enable or review everything that may have been temporarily disabled previously from boot scripts and SMART. Also, save a fresh copy of the new ALSA settings with “alsactl store”.

I then reinstall the NVIDIA driver and other binary kernel modules, re-configure X11 as needed and tweak any specific system scripts and tools I have stored in /usr/local/sbin and similar places, checking if anything needs to be changed or if there’s been a change in device names.

And that’s it. Typically it’s an evening’s work. Reinstalling Windows, on the other hand, and having to apply around 200 security updates takes ages, but at least it’s mostly unattended.

The fantastic youtube-dl team

Posted on . Updated on .

Sometimes people congratulate me for youtube-dl, specially on Twitter. I stepped down as the maintainer long ago, but I’ve given commit access to the repository to a few people who are doing a superb job at keeping the program up-to-date, supporting sites and adding whatever new features users come up with!

The list increases with time as the current maintainers ask me to add new developers. I wanted to have a post like this, so I can refer to it when people congratulate me.

The current list of people with commits access follows, and does not include occasional contributors, who should be thanked too.

mpv is now my default video player

Posted on .

After more than 10 years using MPlayer to play videos under Linux, I switched to mpv (source at GitHub) a few days ago.

mpv is essentially the same video player. It’s based on code from MPlayer and mplayer2, so it behaves more or less like you expect. That means it plays almost everything you throw at it thanks to the ffmpeg libraries, it starts instantly (sorry, VLC, I’ll still recommend you to friends and family!) and the command line options look familiar, even if they have been changed into more standard forms.

Deeper changes include major codebase cleanups, a high quality OpenGL video output driver with higher-precision color transformations, a reworked build system and using system libraries instead of including them in the program source code.

For someone like me who builds all the multimedia toolchain from sources, it’s an advantage. When ffmpeg releases a new version I build that first, and I don’t have to download a second copy and build it again while building MPlayer. mpv uses what’s available. It’s the same for libass or x264, the latter being an example of external library not included when you download the MPlayer source.

This results in much faster compilation times and a much improved situation for packagers, in my humble opinion.

The icing that tops the cake are periodic releases, so you know exactly when you should upgrade as developers mark interesting points in the project history.

And the only drawback is the surprising muscle memory. After more than 10 years typing “mplayer” and with “mpv” sharing a common two-letter prefix, it’s amazing how most times I have to focus a lot to type “mpv” correctly.

Dmitry Glukhovsky at this year's Celsius 232 festival

Posted on .

A couple of days ago my wife and I went to the talk Dmitry Glukhovsky gave in the Celsius 232 festival about his upcoming novel, Furu.re. We took the opportunity of getting a copy of Metro 2033 signed by its author, and the corresponding picture just like last year we did with David Simon. Coincidentally, the book signing took place in the same booth.

I must confess I haven’t read the book yet. I only knew the videogame, and interest in the novel only came to me after hearing he was coming to the festival. The videogame is pretty average, in my humble opinion, and I always thought it required too many resources for the graphics quality it offered, but recently I read several reviews suggesting the "hardcore ranger" mode made the game much more interesting and satisfying, emphasizing the survival aspect. Just a few months ago I replayed it and I have to agree. Naturally, it’s more difficult in that mode, but very far away from frustrating.

I have a copy of Metro: Last Light in my Steam account, but it’s still sitting in my pile of pending games.

Futu.re was enthusiastically presented by the talk director and praised as Glukhovsky’s best novel so far, bar none. The background is a future were humankind has finally reached biological immortality, and tries to answer the questions evoked by that premise.

The superstar of the festival this year was Patrick Rothfuss, author of The Kingkiller Chronicle trilogy, who got people waiting in line for hours to get a signed copy of one of his books, and a packed auditorium during his talk.