404 Not Found
This is my custom 404 Not Found page, for HAProxy on OpenVMS!Visit for custom error message <404>. [ALERT] 105/222650 (1069) : Error(s) found in configuration file : /sys$startup/haproxy.cfg [ALERT] 105/222650 (1069) : Fatal errors found in configuration. At first I thought that it couldn't find the file. File paths on OpenVMS are way different than on UNIX / Linux systems. I looked at a few `CWSD` (Apache for OpenVMS) manuals and config files and that seemed to be the correct syntax. I also didn't start haproxy with the command above, I just stopped and started the service, then refreshed the web browser. To figure out the correct command, I took a look at the startup script: type SYS$STARTUP:HAPROXY$Startup.com Output: $ ! HAPROXY$STARTUP.COM $ !+ $ ! 24-Mar-2022 $ !- $ $ set noon $ $ run/detach - /input=sys$startup:haproxy$run.com/output=sys$manager:haproxy.log - /process_name="HAProxy" - /authorize sys$system:loginout.exe That points to the following file for startup: `sys$startup:haproxy$run.com` and this file for logs `sys$manager:haproxy.log`. The startup file contains the haproxy command I used above: type sys$startup:haproxy$run.com Output: $ set verify $ $ haproxy :== $sys$system:haproxy.exe $ haproxy "-f" "/sys$startup/haproxy.cfg" $ $ exit The log file contaied the following: $ Set NoOn $ VERIFY = F$VERIFY(F$TRNLNM("SYLOGIN_VERIFY")) $ $ haproxy :== $sys$system:haproxy.exe $ haproxy "-f" "/sys$startup/haproxy.cfg" [ALERT] 108/193349 (1060) : parsing [/sys$startup/haproxy.cfg:35] : error opening file for custom error message <404>. [ALERT] 108/193349 (1060) : Error(s) found in configuration file : /sys$startup/haproxy.cfg [ALERT] 108/193349 (1060) : Fatal errors found in configuration. $ $ exit SYSTEM job terminated at 16-APR-2023 20:55:20.95 Accounting information: Buffered I/O count: 581 Peak working set size: 16416 Direct I/O count: 29 Peak virtual size: 249088 Page faults: 736 Mounted volumes: 0 Charged CPU time: 0 00:00:00.26 Elapsed time: 0 00:00:00.28 I tried all different kinds of syntaxes for the error file: - `/sys$sysdisk/haproxy/404.http` - `/dka0/haproxy/404.http` - `/sys$disk/haproxy/404.http` - `dka0:[haproxy]404.http` (didn't work at all) I even defined a few [Logical Names][13], sort of symlinks but then for an entire filesystem, specific to OpenVMS: define haproxy DKA0:[HAPROXY] Then checking that new logical name: dir haproxy Output: Directory DKA0:[HAPROXY] 404.HTTP;1 MAINTENANCE.HTML;1 Total of 2 files. Using `/haproxy/404.http` in the haproxy configuration gave the same error, no matter what logical name I used. I found the following command on stackoverflow, which shows all the file-related stuff an OpenVMS program does. Reminds me of `strace` on Linux. set watch file/class=(all,nodump) It outputs a lot of logs when running a program: haproxy "-f" "/sys$startup/haproxy.cfg" %XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, File protection (13,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read only directory access (13,1,0) %XQP, Thread #0, Directory scan for: HAPROXY.EXE;0, Status: 00000000 %XQP, Thread #0, Access (0,0,0) Status: 00000910 %XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, File protection (16,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read only directory access (16,1,0) %XQP, Thread #0, Directory scan for: HAPROXY.EXE;0, Status: 00000001 %XQP, Thread #0, Alternate access requested: 00000001, Required: 0 %XQP, Thread #0, File protection (11728,39373,0): Access requested: 00000005, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read attributes: Access mode haproxy.exe;1 (11728,39373,0) %XQP, Thread #0, Read attributes: Owner UIC haproxy.exe;1 (11728,39373,0) %XQP, Thread #0, Read attributes: Header 1 accessibility haproxy.exe;1 (11728,39373,0) %XQP, Thread #0, Read attributes: File protection haproxy.exe;1 (11728,39373,0) [...] %XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, File protection (4581,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read only directory access (4581,1,0) %XQP, Thread #0, Directory scan for: MESSAGES.;0, Status: 00000000 %XQP, Thread #0, Directory scan for: MESSAGES.DIR;1, Status: 00000000 %XQP, Thread #0, Lookup (0,0,0) Status: 00000910 [...] %XQP, Thread #0, Directory scan for: HAPROXY.CFG;0, Status: 00000000 %XQP, Thread #0, Lookup (0,0,0) Status: 00000910 %XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, File protection (20,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read only directory access (20,1,0) %XQP, Thread #0, Directory scan for: HAPROXY.CFG;0, Status: 00000001 %XQP, Thread #0, Lookup (11744,2,0) Status: 00000001 %XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, File protection (11744,2,0): Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read attributes: Creation date haproxy.cfg;17 (11744,2,0) %XQP, Thread #0, Read attributes: Revision date haproxy.cfg;17 (11744,2,0) %XQP, Thread #0, Read attributes: Record attributes haproxy.cfg;17 (11744,2,0) %XQP, Thread #0, Read attributes: Owner UIC haproxy.cfg;17 (11744,2,0) %XQP, Thread #0, Read attributes: File protection haproxy.cfg;17 (11744,2,0) %XQP, Thread #0, Read attributes: User file characteristics haproxy.cfg;17 (11744,2,0) %XQP, Thread #0, Read attributes: hardlink count haproxy.cfg;17 (11744,2,0) %XQP, Thread #0, Lookup haproxy.cfg;17 (11744,2,0) Status: 00000001 As you can see, it does a `directory scan` for `haproxy.cfg`. It does not list the directory is searches in however. In the output I can see that it does try to access my error file, `404.http`: %XQP, Thread #0, Control function (11744,2,0) Status: 00000001 %XQP, Thread #0, Final status: 1C000870 %XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, File protection (11566,4,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read only directory access (11566,4,0) %XQP, Thread #0, Directory scan for: 404.HTTP;0, Status: 00000001 %XQP, Thread #0, File protection (11570,2,0): Access requested: 00000001, Status: 00000001, PrvUsd: 00000000 %XQP, Thread #0, Read attributes: Access mode 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Creation date 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Expiration date 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Backup date 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Last access date/time 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Last attribute update date/time 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Data modification date/time 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Revision date 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: ASCII dates 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Access mode 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Journal flags 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: RU active 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Statistics block 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Find ACE by type 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Record attributes 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: User file characteristics 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: File length hint field 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Read attributes: Symlink meta-data 404.HTTP;1 (11570,2,0) %XQP, Thread #0, Access 404.HTTP;1 (11570,2,0) Status: 00000001 So it could access the file (I used the filename `/dka0/haproxy/404.http`). Debugging further got me nowhere, permissions or other rabbit holes. But then I thought of one thing, which are SSL certificates. #### SSL Certificates in HAProxy on OpenVMS I do know of one other way to get HAProxy to read files, which is by using SSL certificated. I generated a standard HAProxy SSL certificate (which is a private key and a cert plus a chain contatenated in PEM format). I copied over that file and placed the following in my config: listen site1 bind :443 ssl crt /dka0/haproxy/mydomain.pem mode http server http1 192.168.1.120:443 maxconn 32 Starting HAProxy with this config file gave me no errors whatsoever, in a browser I could visit the site and get the correct certificate. So then I started thinking, why can it read an SSL certificate but not an HTTP Error file, in the same folder? It wasn't the file path syntax that was incorrect, it must be something else... I went back to [the haproxy manual on errorfile][10] for this version and one thing stood out to me: > For better HTTP compliance, it is recommended that all header lines end with CR-LF and not LF alone. Back [in 2018 when I was involved with the AXPBox Alpha emulator][11] I wrote a few articles on OpenVMS and [one specific article involved line endings][11]. That article links to [the OpenVMS wizard][12] which explains more than I can do here. I tried the following command to convert the line endings: SET FILE/ATTRIBUTE=(RFM=STMLF) 404.http After which I started HAProxy again and to my big surprise, this time, no error message regarding the `errorfile`! It was line endings all along. The error message it logged wasn't `error ACCESSING` the file, it clearly said, `error READING` the file. The path variable was correct all along, it was the file contents that were wrong! I hope you enjoyed this trip down the rabbit hole of debugging in OpenVMS. We looked into the startup script to find the actual executable and the location of the log files. We looked at different ways to define a file path in a config file that expects UNIX style file paths and we debugged file access with a hidden undocumented command. In the end it wasn't an error in the file path syntax, but in the file contents, namely the line endings. [1]: /s/inc/img/Vms-arrow-logo.jpg [2]: /s/tags/openvms.html [3]: /s/blog/OpenVMS_9.2_for_x86_Getting_Started.html [4]: /s/blog/OpenVMS_9.2_for_x86_Getting_Started_part_2.html [5]: /s/blog/OpenVMS_9.2_for_x86_Getting_Started_part_3.html [6]: http://www.haproxy.org/ [7]: https://sp.vmssoftware.com [8]: https://web.archive.org/web/20230419185902/https://forum.vmssoftware.com/viewtopic.php?f=30&t=8622&sid=298dd4b426377beb658b1f240083363e [9]: /s/inc/img/openvms-x86-part4-1.png [10]: https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-errorfile [11]: /s/blog/OpenVMS_Mount_ISO_and_execute_scripts.html#toc_5 [12]: https://web.archive.org/web/20180507190424/http://h41379.www4.hpe.com/wizard/wiz_3707.html [13]: https://web.archive.org/web/20230419200129/https://wiki.vmssoftware.com/Logical_Name --- License: All the text on this website is free as in freedom unless stated otherwise. This means you can use it in any way you want, you can copy it, change it the way you like and republish it, as long as you release the (modified) content under the same license to give others the same freedoms you've got and place my name and a link to this site with the article as source. This site uses Google Analytics for statistics and Google Adwords for advertisements. You are tracked and Google knows everything about you. Use an adblocker like ublock-origin if you don't want it. All the code on this website is licensed under the GNU GPL v3 license unless already licensed under a license which does not allows this form of licensing or if another license is stated on that page / in that software: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see