Getting used to IPv6

Posted on . Updated on .

Some days ago I tweeted my ISP router was assigning me a working public IPv6 address. I don’t know how long it had been that way when I wrote the tweet, but my guess is less than a week. I normally update a few personal repositories each Sunday and push changes in them to Bitbucket. That Sunday, while pushing, I noticed git was adding a few IPv6 addresses to my known_hosts file, which hadn’t happened the Sunday before.

Millions of people are already using IPv6 to access the Internet from their phones and computers, but this is significant because IPv6 adoption in Spain is lagging behind most other countries in Western Europe. My current ISP, Orange Spain, is one of the big players and it’s probably leading IPv6 adoption by number of users, as it’s been deploying it to more and more FTTH clients recently. I don’t know if there’s been a big transition lately or if they’re slowly proceeding by regions.

Obviously, I’m not trying to “sell you” Orange. It’s hated by many and adored by others. In my case it’s doing a good job. I have no complaints so far. I had to call for technical assistance once and they responded properly and fast but your mileage may vary. In any case, I think it’s important to give them (or anyone else) some credit when they do something well, like expanding IPv6 adoption.

Since the day I discovered I was using IPv6, I’ve been reading a bit about it and adjusting my systems and home network to the new situation.

The first topic I read about was how IPv6 addresses worked and the way they are assigned. The best reference I found, as usual, is Wikipedia’s article on IPv6 addresses, and in particular the section about the IPv6 address space and its allocation policies.

Then, I headed to a few IPv6 test pages to see if my IPv6 setup was working properly or if I was to expect problems in some areas. There are quite a few interesting IPv6 test pages, like the one from Google, which is very simple but trustworthy at the same time. There’s also the classic IPv6 test page from Jason Fesler, which reported a score of 10/10. However, I also found an interesting test at ipv6-test.com which warned me I was using SLAAC without privacy extensions.

I didn’t have any idea about what that meant, so I read a bit more about it. It turns out there are 3 normal ways a computer can get an IPv6 address. The first one is static assignment, like in IPv4. The second one is DHCPv6, which is similar to normal DHCP. The third one achieves the same but it’s a “native” standardized mechanism called SLAAC.

IPv6 requires all network interfaces to have a link-local address in addition to other IPv6 addresses they may have. With IPv6 it’s a given a network interface will have multiple addresses assigned. This link-local address starts with prefix “fe80”. Link-local addresses are very similar in concept to both the Ethernet MAC address (and in Ethernet interfaces it’s usually derived from it) and a local IPv4 address. The difference with the latter is link-local addresses are not routable.

Imagine your computer has two network interfaces. The first one has a link-local address with value fe80::1, and the second one has fe80::2. You have a second computer connected with a network cable to one of those interfaces, and its link-local address is fe80::3. If you try to contact fe80::3 (e.g. by trying to SSH into it), which one should be the outgoing interface and your local address in those packets? You don’t know. Fortunately, while not recommended, a lot of software out there lets you use link-local addresses to contact other computers by appending a percent sign and the name of the network interface you want to go out from, like fe80::3%enp3s0 (if enp3s0 is the name of the network interface you want to use to reach that address).

But I digress. Link-local addresses are also used to be able to send IPv6 packets from network interfaces automatically without having another, possibly public, address assigned yet. Using these link-local addresses, you can use SLAAC to send router solicitation requests and routers in your link neighborhood may reply with a prefix assignment you can use to create an additional globally routable address for your interface. At this point you’d normally have two IPv6 addresses assigned to your network interface: link-local and global.

Normally, routers assign a /64 prefix to your computer, and your computer fills the last remaining 64 bits in the address with something derived from the Ethernet MAC address. This is a privacy problem in today’s Internet. Because the way the last 64 bits are derived from the Ethernet MAC address, it’s possible for anyone to identify you and track you across ISPs, connections and services, without using HTTP cookies and without effort.

Enter Privacy Extensions. With them, the Ethernet MAC address will not be used to fill the remaining 64 bits in your assigned prefix to create the IPv6 address. It will be random and it will change every time the network interface is brought up, requesting an IPv6 prefix to the router.

In many Linux distributions this mechanism is not activated by default. At least in my Fedora installation, it was not. It should probably be used in most home desktops, laptops, phones and workstations.

The easiest way to use Privacy Extensions in Linux is by writing “2” to /proc/sys/net/ipv6/conf/default/use_tempaddr. This is normally achieved at boot time before the network interface is brought up by saving the following text to /etc/sysctl.conf or a new file in /etc/sysctl.d:

net.ipv6.conf.default.use_tempaddr = 2

sysctl.d is probably the best mechanism if your distribution allows it because your settings will survive system upgrades without having to manually merge file contents.

According to the Linux kernel documentation, a value of 0 disables Privacy Extensions, a value of 1 enables Privacy Extensions but prefers public addresses to temporary addresses. A value greater than 1 enables Privacy Extensions and gives preference to the temporary address when creating connections. Hence using 2 as the value.

After enabling these Privacy Extensions, you’ll notice your public address doesn’t disappear from the interface. Instead, a third IPv6 address is added to it and used by default.

If you let NetworkManager handle your IPv6 settings, it will interfere and probably ignore the sysctl parameter above. In any case, I think it’s useful to have it there as a fallback. If you tell NetworkManager to ignore the IPv6 part of your connection, the sysctl parameter will kick in. Otherwise, the easiest way to achieve the same effect is using a GUI like nm-connection-editor and setting “IPv6 privacy extensions” to “Enabled (prefer temporary address)” and “IPv6 address generation mode” to “EUI64”, but you could also set the latter to “Stable privacy” and neither public address would be derived from the Ethernet MAC address, with one being temporary and another one being stable.

nm-connection-editor screenshot

Most Android phones have Privacy Extensions enabled. If your home network has WiFi access and your phone uses it, you can make sure Privacy Extensions are enabled by visiting ipv6-test.com from it, but you can also check if your phone has 3 different IPv6 addresses in Settings > About phone > Status > IP address.

I also made some small adjustments to other settings, like using IPv6 DNS servers by default. Both Cloudflare’s 1.1.1.1/1.0.0.1 and Google’s 8.8.8.8/8.8.4.4 public DNS servers have IPv6 counterparts you can use.

I could talk about a few more details but this post is long enough already, so I can only wrap it up by saying I’m really happy to see IPv6 adoption expanding in Spain and being part of it. At the same time I’m surprised at how seamless and transparent the transition to IPv6 is being for many people all over the world, so congratulations to all those sysadmins and engineers behind it.

Fedora 29 takes longer to get to the virtual terminal prompt

Posted on .

Although the upgrade to Fedora 29 was smooth as silk, as I mentioned in a previous post, I quickly noticed my system was taking slightly longer to give me the virtual terminal login prompt when booting up. Linux in general and Fedora in particular has always booted really fast for me, specially after switching to SSD drives some years ago. After a cold boot, I get the login prompt in less than 5 seconds from the Grub menu, so a few extra seconds are easily noticed.

To give you a bit of background information, I’m the kind of person that boots to a virtual terminal by default instead of a graphical login prompt, and I use the “text” plymouth theme. Normally, just after logging in, I go ahead and type “xs”, which is a fast alias I have to launch an X session using “startx” with a few parameters, which in turn ends up launching i3. But booting to a virtual terminal is very handy when I need to exit the X session in order to uninstall the proprietary NVIDIA drivers to upgrade an X11 package, mesa or the kernel. It’s also a bit faster when I don’t need to launch an X session for a basic task I want to do quickly. The inconvenience of having to type x+s+Enter after logging in is tiny, because it takes a fraction of a second to perform those 3 keystrokes, and my hands are already at the keyboard after typing my user password.

After experimenting a bit and measuring actual boot time with a chronometer, I verified my findings. Indeed, it was taking a bit over 9 seconds to get to the login prompt, which is at least twice as long as what I was used to. After analyzing the boot process with systemd-analyze and doing various experiments, I booted with systemd’s log level set to “debug” and I was finally able to verify what was happening. The prompt is given by the [email protected] unit, and it was taking 5 seconds to launch “agetty” after the unit was considered started. See the following boot log:

Nov 06 22:49:57 localhost.localdomain systemd[1]: [email protected]: Passing 0 fds to service
Nov 06 22:49:57 localhost.localdomain systemd[1]: [email protected]: About to execute: /sbin/agetty -o '-p -- \u' --noclear tty1 $TERM
Nov 06 22:49:57 localhost.localdomain systemd[1]: [email protected]: Forked /sbin/agetty as 613
Nov 06 22:49:57 localhost.localdomain systemd[1]: [email protected]: Changed dead -> running
Nov 06 22:49:57 localhost.localdomain systemd[1]: [email protected]: Job [email protected]/start finished, result=done
Nov 06 22:49:57 localhost.localdomain systemd[1]: Started Getty on tty1.
Nov 06 22:49:57 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='[email protected] comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

[Thousands of lines...]

Nov 06 22:50:02 localhost.localdomain systemd[613]: [email protected]: Executing: /sbin/agetty -o '-p -- \u' --noclear tty1 linux

That appeared to be the exact cause of my login prompt delay, but I wasn’t sure about why it was happening. However, a few web searches later I arrived to the conclusion it was due to the [email protected] unit being declared with Type=idle. According to systemd.service(5):

Behavior of idle is very similar to simple; however, actual
execution of the service program is delayed until all active jobs
are dispatched. This may be used to avoid interleaving of output of
shell services with the status output on the console. Note that
this type is useful only to improve console output, it is not
useful as a general unit ordering tool, and the effect of this
service type is subject to a 5s time-out, after which the service
program is invoked anyway.

Indeed, changing the unit to type “simple” restored the old behavior. I’ve investigated a bit and that unit has had that type for a long time, so my guess is the change I observed in boot time has been triggered by other changes in the Fedora boot process. Just for completeness, this is the list of units with type “idle” in my system, but the only one that matters to me is [email protected].

/usr/lib/systemd/system/[email protected]:30:Type=idle
/usr/lib/systemd/system/emergency.service:24:Type=idle
/usr/lib/systemd/system/rescue.service:23:Type=idle
/usr/lib/systemd/system/[email protected]:35:Type=idle
/usr/lib/systemd/system/[email protected]:40:Type=idle
/usr/lib/systemd/system/console-getty.service:25:Type=idle

It’s worth noting I don’t get text garbage on screen by changing the unit type to simple, at least in Fedora. First of all, the wait process was timing out in any case, so I could still get a garbaged prompt. Furthermore, when agetty is launched, plymouth seems to go away and I get a clean login prompt. Waiting for an idle system is futile, more so if the system boots slower than mine. Those reasons prompted me to file a bug report that hasn’t been updated since filed. Fedora’s systemd package has 146 open bugs as of the time I’m writing this and many of them are sitting there marked as “New” without being confirmed or assigned.

To change the unit type to “simple” in a non-intrusive way that survives system upgrades, the best way is to create a so-called drop-in file for it. As root, use:

systemctl edit [email protected]

That will launch a text editor and create a file named /etc/systemd/system/[email protected]/override.conf where you can change parts of the unit configuration. The contents should simply be:

[Service]
Type=simple

If you also boot to a virtual terminal, I hope this helps or at least encourages you to experiment a bit to see if you can boot faster.

Creating a local repository for your own Fedora packages

Posted on .

Almost two years ago I wrote about a nice surprise in the Fedora freetype package that made it easy for users to rebuild the package and include subpixel rendering support. In the original article I also added some brief instructions with a few commands to run in order to create and prepare a specific user account used to build packages.

Expanding on that, I mentioned you’d want to install the build dependencies for the freetype package before rebuilding it. Back then I mentioned the packages explicitly. The list can be obtained from the package SPEC file, but there’s a better option. Any time you want to rebuild an official package you can install its build dependencies automatically by using dnf builddep, as in:

dnf builddep freetype

Furthermore, if you rebuild an official package you’ll notice sometimes Fedora wants to reinstall the original one from the official repositories. In order to avoid that, you can create a local repository containing your own packages and make it take precedence over the official packages. In any case, this is also handy if you have additional non-official packages you build yourself.

The first step is creating the repository metadata. Let’s suppose the build user is named “build”, and let’s suppose rpmdevtools has been set up normally for that user, creating a bunch of directories under /home/build/rpmbuild.

One of those is the directory containing RPM files for every package you create, called /home/build/rpmbuild/RPMS. It has one subdirectory for each architecture you create packages for, normally x86_64 and sometimes noarch too. You want to create a repository with the contents of that directory. It could hardly be easier:

createrepo /home/build/rpmbuild/RPMS

You can run that as the build user. It will create a subdirectory called repodata containing package metainformation. If you have a lot of personal packages or if you want something faster, you can replace createrepo with createrepo_c, a version of createrepo written in C that’s supposed to be much faster. For my simple use case, I barely notice any difference between the two. You’ll have to run that command every time you add or remove packages from the repository, to update their metainformation.

The final step is telling dnf there’s a repository there. That’s as easy as creating a file under /etc/yum.repos.d with information about it. I called mine localrepo.repo and it has the following contents:

[localrepo]
name=localrepo
baseurl=file:///home/build/rpmbuild/RPMS
enabled=1
gpgcheck=0
metadata_expire=60s
priority=80

The base URL mentions the path to the local directory. The repo is also enabled and its packages are not signed, so the GPG check is disabled. Metadata expires after 60 seconds and needs to be rechecked then. That’s handy so you won’t have to use dnf’s --refresh option all the time while working with your local repository. The last line contains a numeric priority value.

The lower the number, the higher the priority. The default priority is 99 according to dnf.conf(5) and that’s the priority of the official repos because they don’t set a value. If you use other non-official repositories like RPM Fusion, you’ll see they have their own priority value. You have to decide if you want your local repo to have higher or lower priority compared to others and put a numeric value there.

Going back to the rebuilt freetype package, dnf won’t insist anymore on reinstalling the official version, but it will update it if the official package version is considered more recent. Hopefully, you’ll notice font rendering has changed slightly, rebuild the package and reinstall it from your local repository.

Upgrade to Fedora 29 and Red Hat acquisition

Posted on .

I upgraded to Fedora 29 yesterday without any issues. The upgrade process was as flawless as the last few ones so I have to congratulate everyone involved once more for doing a superb job.

When I switched from Slackware to Fedora almost four years ago I mentioned it was probably as future-proof as Debian, the other option I had been considering. The recent acquisition of Red Hat by IBM has many people worried for several reasons, even if most people prefer to be prudent and wait to see how things turn out.

Fedora could be affected but, more importantly, Red Hat’s numerous contributions to a myriad of projects could be affected too. Kernel, GCC, glibc, Gnome, GTK, etc. It’s obvious if this operation doesn’t work it will have side effects on the whole Linux ecosystem, so it’s not a matter of just switching distributions.

In any case, if Fedora ceased to exist my next option would be Debian unstable. Almost four years ago I mentioned Debian testing, but I’ve been reading a bit on the difference between the two and I would probably prefer unstable now. My second option would be Ubuntu. It’s based on Debian unstable but it has a release schedule similar to the one Fedora has so it would be familiar to me at this point.

Disabling HyperThreading on Linux

Posted on .

In light of recent CPU vulnerabilities like TLBleed and L1TF you may want to disable HyperThreading on your computer if you have an Intel CPU with that feature. Note this is a personal decision. Most desktop computers run untrusted code in the form of untrusted JavaScript inside your web browser, but some people may consider disabling HyperThreading an excessive measure for a desktop computer. The situation for some servers is very different.

In any case, if you want to do it, your computer’s BIOS is your friend and the easiest solution. However, you may want a more flexible solution. Passing “noht” to the kernel command line used to work in the past but today it doesn’t seem to work properly. Following that link you can read a solution by “Paul M” that works reliably in more recent Linux systems. It’s also explained in some other articles, but I’ll repeat the explanation here.

Linux has a directory called /sys/devices/system/cpu that contains a subdirectory for each virtual CPU in the system, i.e. each thread you can run. Inside each directory, there’s a file called topology/thread_siblings_list that contains the list of CPUs that belong to the same thread group. In other words, the list of CPUs that are part of the same physical core. For example, if CPUs number 0 and 4 execute threads that run on the same core, the thread_siblings_list file for CPUs 0 and 4 contain the string 0,4 (always in that order). To disable hyperthreading, you need to disable the second CPU in each of those groups, which can be done by writing 0 to the online file in each CPU subdirectory.

This solution is flexible in the sense that the change can be reverted dynamically without rebooting by re-enabling the CPUs. I’d personally do it from /etc/rc.d/rc.local (or the equivalent file in your distribution). Note that file is normally executed as one of the last steps of the boot process, so hyperthreading would still be enabled for a few seconds after booting up. Normally, that’s OK in a desktop computer.

You’ll read many small variants of the following code:

# Disable Hyperthreading.
#
# /sys/devices/system/cpu/cpu*/topology/thread_siblings_list contains the list
# of sibling threads for each CPU core, always in the same order. For example,
# if cores 0 and 4 are threads on the same physical core, they both contain the
# string "0,4" in their thread_siblings_list file.
#
# The following script gets the second number of each list and disables that
# core by writing a 0 to its "online" control file.
#
for n in $( cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | cut -d, -f2 | sort -u ); do
    echo 0 >/sys/devices/system/cpu/cpu${n}/online
done