Jul 29, 2018

Blitz the Fritz: Optimize for Latency after setting up OpenWrt

Already usable with the target of the sibling devices (7320/7330), final support for the Fritzbox 7312 was committed to OpenWrt in July. Introduced to market as OEM version in 2012, the device received support until 2017 (Fritz!OS 6.54). It's basic1 but sufficient: ADSL2+ (Annex J), 1x 100 Mbit/s Ethernet, 802.11b/g/n and able to connect an analog telephone to VoIP at ~8 Watt. As people are leaving for higher bandwidth devices (with the 7412 being a successor), the average aftermarket price is low at ~10€ and less.

fritzbox 7312 frontside

I had a few reasons to flash it: being able to keep a current kernel and updated userspace on an EoL-device and to get in touch with some networking topics. Notably DNS, VPN, packet filtering and traffic shaping. I appreciate the efforts of the Freifunk people and OpenWrt is their base distribution, this can be a way to get to know more about their configuration.

The flashing process is outright trivial compared to doing similar to some old Android handhelds. Configuring the basics is very approachable in a few lines within files behind one directory path only: /etc/config/. The development snapshots do not include LuCI the webmin interface, but as one can do easily without, this is a cli-only guide.

Install

Download a snapshot, compare the checksum and maybe read the docs. In the commit, the ftp flash method from the 7330 is linked to. The box will accept FTP connections in the first 5 seconds after restart/boot. In the directory of the snapshot download, start a terminal:

ftp 192.168.178.1

and enter after successful connection to the polite Adam2 bootloader

User: adam2
Pwd: adam2
bin
passiv
quote MEDIA FLSH
put openwrt-lantiq-xway-avm_fritz7312-squashfs-sysupgrade.bin mtd1
quote REBOOT
quit

Give it some time after reboot and login to a yummy ascii banner, this must be the real thing!

ssh -l root 192.168.1.1
  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M

Config

I will quote config lines I changed or discuss, match accordingly. Run passwd and set your root pw, then attend to the configs.

1) /etc/config/dropbear: put your public (RSA2) key into /etc/dropbear/authorized_keys, chmod 600 the filepath and follow the securing guide. Enable pubkey-only auth and bind to a local interface if you do not intend to login from outside your network. If you want it public, moving the port reduces some noise but is ultimately not necessary. You could consider openssh instead too. If you install LuCI, you can do similar for the HTTP Server in this Guide: Secure your router's access. It binds uhttpd to the local interface too, and you will use a SSH tunnel to forward the http port.

config dropbear
    option PasswordAuth 'off'
    option RootPasswordAuth 'off'
    option Interface 'lan'
    option Port '5522'

Run /etc/init.d/dropbear killclients and /etc/init.d/dropbear restart after the edits.

2) /etc/config/network: add proper interface for wan and auth credentials, maybe dsl options

config dsl 'dsl'
    option annex 'b'

config interface 'wan'
    option ifname 'dsl0'
    option proto 'pppoe'
    option username '<123456789@isp.de>'
    option password '<password>'
    option ipv6 '0'

I use two /etc/crontabs/root entries with ifup/ifdown wan to reconnect dsl nightly. Run /etc/init.d/network reload to make config changes effective. If you have pppoe auth problems, add option pppd_options 'debug' to the wan interface.

To stop odhcp6c sending router advertisements (actually not sure what exactly is sent) to All DHCP Relay Agents and Servers on a wan that is not ipv6 capable, disable the wan6 section.

odhcp6c: Failed to send DHCPV6 message to ff02::1:2 (Permission denied)

3) /etc/config/wireless: is disabled by default. See the config lines explained in the wiki. Even for a 802.11 b/g/n device, hwmode '11g' is correct, the htmode 'HT20/HT40' is what enables the "n" in 11n. As I don't need much coverage, I checked with iwinfo wlan0 txpowerlist where the device is at, lowered the value and issued wifi reload. Check wireless.utilities for the scope of each util.

config wifi-device 'radio0'
    option disabled '0'
    option channel '11'
    option hwmode '11g'
    option htmode 'HT20'
    option txpower '9'

config wifi-iface 'default_radio0'
    option ssid '<your-wifi-name>'
    option encryption 'psk2'
    option key '<key>'

4) /etc/config/system: set your timezone according to this list to have your timestamps without offset. If you're an avid reader of logs, you can modify the log_size parameter.

5) telephony/VoIP/FXS setup: there exists a specific guide for the SoC family with dependencies ~3.5MB. Running asterisk after install put the device in an unresponsive state with a restart not booting to active networking. One of the new kernel modules caused a panic? I think a serial connection (USB-TTL) could help debug this. I had to flash again from Adam2, see backup advice.

6) use SSL for download of packagelists. This will need 1.3 MB. SSL is handy for other needs too. The http downloader is not arbitrarily interchangeable. Freifunk proposes uclient-ssl. Maybe it would suffice to only have the Let's Encrypt CA of downloads.openwrt.org included initially to save on space.

opkg install wget
opkg install ca-certificates
sed -i 's/http:/https:/g' /etc/opkg/distfeeds.conf
opkg update

7) System upgrades work via sysupgrade: files to be preserved over updates are configured (in /lib/upgrade/keep.d/) - but not all. As I'm not sure if folder-naming is sufficient, I listed paths explicitly in /etc/sysupgrade.conf.

/etc/dropbear/authorized_keys
/etc/config/dropbear

Manually installed software packages have to be reinstalled, changed config files are listed with opkg list-changed-conffiles

Even with those paths defined, create a backup of all your configs when your finished. Bootstrapping in case of unrecoverable errors will be quicker. Use either sysupgrade --create-backup /path/to/tarball.tar.gz and transfer the file, or use these lines:

DATE=$(date \+%Y%m%d); mkdir -p $DATE/etc
scp -r openwrt:/etc/{config,dropbear/authorized_keys,sysupgrade.conf} $DATE/etc/

8) with uci_commit and reload_config you can make changes in the /etc/config/ files effective (individually too, as in uci_commit network for example). It also made me notice defective config lines, otherwise rendering a almost correct config unusable. It will output the syntax error by line and position.

Utilities

Some of these are already installed, if missing get it with opkg install ..: bandwidth usage monitoring is provided by vnstat. The default "SaveInterval 30" (minutes) is light on writing to the db. vnstat --days -i dsl0 is a good summary, ifstat is good for quick look at current throughput. DSL diagnostics is handled by the ltq-adsl-app default package, called via /etc/init.d/dsl_control status. It displays the adsl termination unit vendor, dsl annex and line mode, SNR values and data rates. Wireless signal quality, channel and power level is attainable via iwinfo. wavemon will show you a nicer ncurse-based view. For general network debug, I rely often on mtr, nslookup (as dig will fetch the bind dependency) and tcpdump. wifischedule can turn your wireless off at designated time spans. Use the crontab to reconnect nightly:

/etc/crontabs/root 
0 5 * * * /usr/bin/logger "Scheduled WAN reconnect started" && /sbin/ifup wan

Base Packages

If I install from snapshots and do not compile myself, these are the packages I need.

opkg update
opkg install --force-overwrite dnsmasq-full
opkg install wget ca-certificates openssl-util sqm-scripts ddns-scripts 6in4 stubby vnstat ifstat mtr tcpdump ethtool diffutils screen wireguard
opkg remove dnsmasq

Debug and Workarounds

logread -f is a often used command to tail into syslogs. You can set a debug level of a service and follow its interactions this way. The uci command and ubusd are capable mechanisms and interesting topics.

ubus list
ubus call network.interface.lan dump

If you have wireless problem, raise loglevel and check with logread

uci set wireless.radio0.log_level=1
uci commit wireless
wifi up

If the auto-firewall-script is not working (the 2018-07-25 snapshot had a platform independent fw3 bug), these are some very simple forwarding iptables rules:

echo 1 > /proc/sys/net/ipv4/ip_forward
local="br-lan"
public="pppoe-wan"
iptables -t nat -A POSTROUTING -o $public -j MASQUERADE
iptables -A INPUT -i $local -j ACCEPT
iptables -A INPUT -i $public -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -j ACCEPT

I didn't have to touch /etc/config/dhcp. Probably due to a yet nonexistent IPv6 interface I receive odhcpd6c error messages and have to revisit this issue later: DNS and DHCP configuration.

Fix Bufferbloat

After basic setup, let's reap real benefit from having more control: consistent latency despite high network traffic. Bufferbloat is undesired latency introduced by too big and filled up queues. Your interactive voice stream will be fragmented by a bandwidth hog and noticeably degraded. Active Queue Length Management can help, checkout the research effort and a interview in a Packetpusher podcast.

opkg install sqm-scripts

Read up on tuning the parameters for download/upload/overhead. At my ADSL2+ uplink I took the numbers (Data Rate Up/Down: 11.283 Mb/s / 1.151 Mb/s) from dsl_control status and deducted 5%. I might need to script this if the rates are not stable.

/etc/config/sqm:

config queue 'wan'
    option enabled '1'
    option interface 'pppoe-wan'
    option download '10700'
    option upload '1100'
    option overhead '44'
    option qdisc 'cake'
    option script 'piece_of_cake.qos'
    option linklayer 'atm'

This will change your "queuing discipline" (qdisc) to "cake". See debug (logread) when restarting with new config

/etc/init.d/sqm stop ; SQM_DEBUG=1 SQM_VERBOSITY_MAX=8 /etc/init.d/sqm start
SQM: Starting SQM script: piece_of_cake.qos on pppoe-wan, in: 10700 Kbps, out: 1100 Kbps
SQM: cake link layer adjustments: atm overhead 44 mpu 0

To Measure ping latency (msec) during down- and upload, get this script and install netperf on a local machine. Then run ./betterspeedtest.sh | tee -a bufferbloat.log multiple times before and after change. By default, the script will ping against gstatic.com. The change to the latency was impressive.

qdisc Type Rate Min 10pct Median Avg 90pct Max
default Download 8.92 Mbps 29 534 845 828 1092 1224
cake Download 8.26 Mbps 28 30 33 34 36 62
default Upload 0.95 Mbps 28 32 48 56 65 83
cake Upload 0.85 Mbps 28 29 34 34 40 50

(tc -d qdisc can give details, and tc -s qdisc statistics)

Check out the .help files in /usr/lib/sqm/ for what the respective .qos ("QoS") scripts will do (latency differences were negligible, but I don't know about load. Apparently, layer_cake.qos with DiffServ is able to prioritize VoIP traffic compared to the very simple piece_of_cake.qos).

Flent is an easy install and can plot measurement graphs for visual comparison. The underlying irtt helps to also evaluate connection quality by inspecting instantaneous packet delay variation ("jitter") - I find this very interesting, especially if a VoIP client could do these measurements in the background and give you summary reports across environments.

cake queueing discipline at work

Left is fq_codel vs cake on the right on a netperf tcp download benchmark.

flent tcp_download -p totals -l 60 -H <netperfserver> -f plot --figure-dpi 100 --horizontal-legend --zero --bounds-y 12 --bounds-y 120 -o <outputfile>

Albeit another topic: asking the ISP to turn off interleaving for the DSL connection shaved off 7-10 ms from my latencies, making for ~6 ms inner-city pings. So all in all, it was a great day to be a ping.

Wraping up

After Openwrt is setup and control of latency ensured, what functionality is beyond? See a follow up with DynDNS, IPv6 and Wireguard VPN and a post on encrypted DNS traffic.

You can watch a good introduction to the project with the talk "OpenWrt/LEDE: When Two become One" by Florian Fainelli. He explains more about the build and package systems of the distribution.

As of writing, this lengthy Guide to Openwrt has a detailed section on Wifi optimizations and other topics, hinting at being updated with further content.

Recap

May 2019

  • for the first few months I was plagued by PPPoE hangups and then being stuck in reconnection phase with a Timeout waiting for PADO packets. It got explained in the references of openwrt/openwrt#555c5923. Basically the 5/1 defaults for lcp-echo-failure/lcp-echo-interval were too strict for my copper line, a higher lcp-echo-interval helped and it is stable ever since (option keepalive '5 5' in the interface wan section of /etc/config/network)
  • you can use the latest lantiq dsl firmware from the Fritz!OS 7312 AVM image (extract both, the avm.image and then kernel.image with 7z e to get to the ar9-B-dsl.bin file)
  • installing the voice subsystem packages kmod-ltq-vmmc/kmod-ltq-tapi to do VoIP on the FXS port ensures a bootloop. This is unresolved. Not being able to use a connected line phone is the only nitpick
  • the VDSL2 capable successor Fritzbox 7412 has an OpenWrt port since 2019

  1. MIPS 34Kc¹ CPU@393 MHz with 16 MB of Flash and 64 MB RAM, see current and old wiki. The Lantiq Xway ARX188 is based on the ARX100 "AR9" SoC Family, and packages are built for it in the mips_24kc instruction set. 

  2. ed25519 not yet possible, but might come soon