OPENWRT SURGERY

After a while when I didn't have ideas for new phlog posts, now I 
have ideas and no time to write them. But I did finally get around 
to setting up my final OpenWRT configuration for my home router, v. 
23 within my router's 32MB of RAM much better than the default. 
It's final because, as noted in my post 2025-01-09Hit_and_MIPS.txt 
(oh dear, this took me six month to get back to!), OpenWRT 24 has 
dropped the build option for router boards in my router's hardware 
family, which at least gives me the excuse to hack deeper into the 
works without fearing breakage after upgrades.

I'd like to write my usual long ramble, but since allowing time for 
writing this post is under the excuse of it being notes for my own 
later reference, I'll go into quick dot-point mode:

* Building with OpenWRT Image Builder on x86_64 VPS (because the 
image builder is a huge download so I don't want to do that over my 
home internet connection)
  
https://openwrt.org/docs/guide-user/additional-software/imagebuilder
  - Make directory named "files" in the root of the image builder 
directory, pointed to with the FILES argument to make. In here go 
all files to be added/changed compared with the default router file 
system, with all their final directory paths and permissions (you 
need root permissions to make them).
  - Instructions say not to run the Image Builder Makefile as root, 
but if you want to include custom files that are only readable by 
root, it will fail to read them unless you run it as root. It runs 
fine as root.

* OpenWRT releases supported for six months after release of next 
major version:
    https://openwrt.org/docs/guide-developer/security#support_status
  OpenWRT 23 therefore supported until July this year. Latest 
release is 23.05.5.

* The OpenWRT Wiki page for my router pointed to the "SMP" images 
(until it became officially unsupported after OpenWRT 19 due to 
lack of RAM). This was wrong, it only has one processor so an SMP 
kernel is bloat. I should have been using the image builder 
configured without SMP support, called "generic":
  
https://downloads.openwrt.org/releases/23.05.5/targets/bcm63xx/generic/openwrt-imagebuilder-23.05.5-bcm63xx-generic.Linux-x86_64.tar.xz
  https://openwrt.org/docs/techref/targets/bcm63xx

* Openwrt comes with DHCPv6 via a separate DHCP server package. I'm 
not using IPv6 so I don't need that and can just use the IPv4 DHCP 
server in dnsmasq:
 - Add "option dhcpv6 'disabled'" in /etc/config/dhcp.
 - Also disable automatic IPv6 support for USB modem in 
/etc/config/network
 - Also comment-out IPv6 rules in /etc/config/firewall
  - "disable_ipv6" option is not supported since OpenWrt 22.
 - Remove default packages odhcp6c, odhcpd-ipv6only, and odhcpd6c, 
from image builder PACKAGES
 - Disabled odhcpd in /etc/config/dhcp by setting in the odhcpd 
section:
      option dhcpv6 'disabled'
      option ra 'disabled'

* Disable other unused default services
  - ntpd (NTP daemon)
   - Router doesn't have a RTC so this means it won't know the 
correct time anymore (will count from the date of the current 
OpenWRT version's release after booting), but I don't care.
   - Can be disabled in /etc/config/system or put "sysntpd" in 
DISABLED_SERVICES for image builder.
  - urngd (Enthropy harvester daemon)
   - This might be important for encrypted WiFi, for which drivers 
sometimes need a fresh enthropy source), but I'm not using that so 
it gets the axe.
   - Remove default urngd package from image builder PACKAGES.
   - Put urngd in DISABLED_SERVICES for image builder.
  - logd (syslog daemon)
   - Networking wouldn't come up when I disabled this in the old 
image, so I left it enabled at boot but put this at the start of 
/etc/rc.local to kill logd at the end of the boot process:
     # Wait until firewall confguration is finished
     while killall -q -0 hotplug-call || killall -q -0 mac80211.sh || killall -q -0 fw4
     do
       sleep 5
     done

     # Kill syslog daemon
     killall logd
   - Use "/etc/init.d/log start" to start it after boot for 
debugging purposes. Read syslog afterwards with the "logread" 
command.

* Ditch Dropbear and SSH/SCP. Telnet and FTP have less overhead and 
won't break when new clients demand newer server versions than the 
last OpenWrt 23 dropbear package.
  - Remove "dropbear" package from image builder PACKAGES.
  - put "dropbear" in DISABLED_SERVICES for image builder.
  - put GNU Inetutils telnetd and inetd static binaries (see 
2025-01-09Hit_and_MIPS.txt) in custom files path:
     /usr/bin/telnetd
     /usr/bin/inetd
  - Make symlinks:
     /usr/local/bin -> /bin
     /usr/local/var -> /var
  - Disable vsftpd service, since I'll use it in inetd mode which 
takes up much less RAM, by putting vsftpd in DISABLED_SERVICES for 
image builder.
  - Make /etc/inetd.conf:
     telnet stream tcp nowait root /usr/bin/telnetd telnetd
     ftp stream tcp nowait root /usr/sbin/vsftpd vsftpd
  - Set "listen=NO" in /etc/vsftpd.conf
  - Create this directory for vsftpd at start-up with this command 
in /etc/rc.local:
     mkdir -m 0755 -p /var/run/vsftpd
    (/var/run is in tmpfs so can't put the directory in FILES)
  - Inetd now needs a custom init script to start it at boot, in 
/etc/init.d/inetd:
      #!/bin/sh /etc/rc.common

      # start after and stop before networking
      START=19
      STOP=50

      start() {
        SERVICE_USE_PID=1 service_start /usr/bin/inetd /etc/inetd.conf
      }

      stop() {
        service_stop /usr/bin/inetd
      }

      reload() {
        service_reload /usr/bin/inetd
      }
    - Make the script executable and owned+writable by root only. 
The symlinks to this script in /etc/rc.d are created by the image 
builder.

* If the router does run out of RAM, processes get killed by the 
kernel "OOM Reaper", but respawned by the (OpenWRT-specific) init 
system. I tweaked the "respawn" settings to be more forgiving and 
better suit my usage. Following docs here:
   
https://openwrt.org/docs/guide-developer/procd-init-scripts#service_parameters
  /etc/init.d/network - in start_service(): procd_set_param respawn 3600 20 0
   - This sets a 20sec wait between respawning if the network 
process is killed, and
     retries indefinitely.
  /etc/init.d/dnsmasq - in dnsmasq_start(): procd_set_param respawn 3600 40 0
   - This sets a 40sec wait between respawning if the DNS server 
process is killed, and
     retries indefinitely.
  /etc/init.d/log - in start_service_daemon(): #       procd_set_param respawn 5 1 -1
   - Disable respawning the log process if it is killed by 
commenting out this line.

* Probably not an issue anymore, but before the above changes I had 
trouble with running out of RAM at boot because the OpenWRT init 
system starts all services in the background and therefore has 
their init scripts all running in parallel. That's great for boot 
speed (at least if I had multiple processors), but terrible for RAM 
usage since there needs to be room for everything to run at once (a 
really odd choice from the OpenWRT developers). Disabling dnsmasq 
in DISABLED_SERVICES then launching it at the end of the boot 
sequence before starting the mobile broadband modem avoids the 
choke point, and can be achieved with this in /etc/rc.local (first 
part already included under "Disable other unused default services" 
above):
     # Wait until firewall confguration is finished
     while killall -q -0 hotplug-call || killall -q -0 mac80211.sh || killall -q -0 fw4
     do
       sleep 5
     done

     # Start these services at the end of the boot process
     # to avoid running out of RAM
     sleep 30
     /etc/init.d/dnsmasq start
     sleep 15
     ifup wan

     # Restart firewall if it was killed during ifup
     sleep 45
     /etc/init.d/firewall boot

* Finally, my idata.sh script (from 2024-03-10Off_and_On_Line.txt) 
used SSH to run the data-collector script on the router. Looking at 
the inetd docs, I discovered I could turn that into its own 
protocol using TCPMUX and make requests using "nc" instead of 
dropbear without changing the router-side script at all.
 - Put these lines in /etc/inetd.conf:
   tcpmux stream tcp nowait root internal
   tcpmux/+idata stream tcp nowait root /root/send_idata.sh idata
 - But all the best things in computing are obsolete, so I needed 
to add the tcpmux protocol to the /etc/services file which is 
missing it by default:
    tcpmux		1/tcp
 - Now in idata.sh, instead of:
     # Grab data stats from router via SSH
     newdata="`DROPBEAR_PASSWORD=$ROUTER_PASSWORD dbclient $ROUTER_SSH $ROUTER_SCRIPT`"
   I've got:
     # Grab data stats from router via TCPMUX
     newdata="`echo idata | nc $ROUTER 1 | tail -n 1`"

* This is my final OpenWRT Image Builder build command. Note that 
LuCI was never included in these builds by default (hence I've only 
ever known OpenWRT configuration via SSH), so it doesn't need to be 
excluded. I'm not sure if that's the case for other build targets:

make image FILES="files" PROFILE="brcm_bcm96358vw" PACKAGES="chat comgt hostapd-basic hostapd-common iw-full iwinfo kmod-b43 kmod-bcma kmod-cfg80211 kmod-crypto-hash kmod-mac80211 kmod-mac80211-hwsim kmod-nls-base kmod-usb2 kmod-usb-acm kmod-usb-core kmod-usb-ehci kmod-usb-ohci kmod-usb-serial kmod-usb-serial-option kmod-usb-serial-wwan kmod-usb-uhci libiwinfo librt libusb-1.0 terminfo usb-modeswitch vsftpd wireless-regdb wireless-tools coreutils-stty picocom kmod-usb-net-sierrawireless kmod-usb-serial-sierrawireless kmod-mii -odhcpd-ipv6only -odhcpd6c -odhcp6c -urng -dropbear" DISABLED_SERVICES="urngd dnsmasq sysntpd dropbear vsftpd"
 - Note that the "kmod-" packages are specific to my router 
hardware. No longer needing to figure out when these change 
(without documentation of the changes) is another big win from not 
upgrading to new major OpenWRT versions anymore.

* Still takes me a few tries to flash the firmware via TFTP (via 
serial console command). Dunno why the connection always seems to 
fail the first few times, but it works eventually.

The stats for this new image compared to the previous default SMP 
one which I'd tried to optimise after installation are significant.

Specs: 32MB RAM, 8MB Flash

Old OpenWRT 23 (SMP) installation:
  Total RAM (minus kernel size):	23504KB
  RAM available after boot:		4148KB
  Flash free after boot:		568KB

New OpenWRT 23 (non-SMP) installation:
  Total RAM (minus kernel size):	24712KB
  RAM available after boot:		9512KB
  Flash free after boot:		1960KB

But most important, it now starts up in a reasonable time again (it 
was taking over 5min before) and no longer has random crashes when 
the mobile broadband modem signal drops out. Before it would run 
out of RAM while trying to reconnect, sometimes failing, or killing 
other important services in the process. Also the router I thought 
was broken when I first installed OpenWRT 23 on it is actually 
fine, it was just an early case of those issues causing the 
firewall process to be killed which made it look like the network 
interface was intermittent. It has been running solidly for days 
since installing this lightweight build.

Oh and typically now that I've finally discovered the RAM/storage 
advantage of the Linux kernel without SMP support, the kernel 
developers are talking about removing that option:
https://www.phoronix.com/news/Linux-6.17-Maybe-SMP-Uncond

Linux seems to be looking less attractive lately, clearly aiming to 
serve a different user-base than me. Also see this tally of how the 
size of OpenWrt has increased over the years:
https://openwrt.org/supported_devices/432_warning#analysis_of_firmware_size_growth

I wonder if a BSD-based router OS would have done better, such as 
ZRouter ( https://zrouter.org - looks dead)? Although drivers would 
be a roadblock with that anyway. Anyway I don't think there's any 
good reason that a basic router now needs better specs than these 
ones have. Plus newer basic ISP-supplied models that are common 
second-hand in Australia have little-better specs and never got any 
sort of OpenWRT support to begin with.

 - The Free Thinker