i summon one kim

I’ve been observing a really strange problem on Ubuntu Server 8.04: processes are sitting in limbo not doing anything. I first observed this as phones losing their SIP registrations periodically (but at random intervals).

When debugging the issue, I saw that a ping running on the system would often just sit there, not sending more packets, sometimes for seconds on end (I didn’t always wait to see if it would ever continue). Since hitting a key on the keyboard would immediately be echoed back, it wasn’t a problem with the terminal or ssh connection. Pinging the system from another would work without problems. Interrupting ping would report no packet loss (which was true — you can’t lose replies to packets you didn’t send).

My best guess is that the alternative scheduler chosen by the Ubuntu developers for the server kernels doesn’t work as intended on AMD CPUs (or just the Athlon XP, or some other part of the hardware I’m using).

My “fix” was to switch to Debian 5.0 (lenny), which doesn’t exhibit this problem at all.

I could have tried with the desktop edition of Ubuntu, but can’t really spare the time right now. Especially since I now have something that works reliably again.

I did try with different network hardware, though. I was originally using an SMC card based on ns83820 (but for some reason not reporting anything at all with ethtool — works fine with the gsip driver on NetBSD). I then switched to a Netwjork “brand” card based on r8169 (well liked by ethtool). No difference, but stayed with the latter (working ethtool is always nice).

Just a short note on getting a minimal Asterisk environment installed on current versions of Debian and Ubuntu:

aptitude install asterisk
m-a a-i zaptel
modprobe ztdummy

Also add ztdummy to /etc/modules so it gets loaded when the system starts.

If the m-a command is not found, install the module-assistant package.

I’ve been always a fan of the robust upgrade procedure documented in the Debian release notes, which has worked without problems even over ssh to remote machines. This has made upgrading very painless (at least since sarge — I haven’t used Debian actively longer than that).

Yesterday I ran into a problem, though, which was only saved by having remote console access. I have a system with a slightly more complicated disk setup: it has two SCSI disks running as a RAID 1 array using md and lvm. Upon rebooting after the upgrade, the system didn’t reappear on the network. What I found on the console was the initramfs panic shell: the root file system had not been found. Rebooting the old etch kernel worked fine.

The workaround proved to be very simple, once I distilled it from the search engine results. Just add rootdelay=10 to the kernel options in the bootloader. I’m using GRUB so this translates to editing the kopt line in /boot/grub/menu.lst and running update-grub.

I had also added raid1 to modules in /etc/initramfs-tools and regenerated the initrd, but that (alone) didn’t help. I’m not even sure it is needed at all — it might already be included anyway when using MODULES=most in initramfs.conf.

I ran into a model symptom of lacking multicast support with IPv6: a system wouldn’t answer to IPv6 traffic it didn’t initiate. But unless you’ve run into the problem before and managed to diagnose it successfully, you might not realize it’s about multicast.

It was my second Debian Linux system with IPv6 connectivity that gave me a start. I built one more system and got the same results. I could have sworn the first one had worked fine without any tricks. I considered it, but decided I’d rather not abandon IPv6 on Linux. After all, I had it running on all the other platforms (NetBSD, Mac OS X, Windows). (more…)