ZIPPY ORG-AGENDA BUFFER BUILDING
-
-
Preamble
----------------------------------------------------------------------
`org-agenda', if you don't know about the Emacs package, builds views
out of `org-mode' notes. Usually the views will show the current day
or week, showing all notes with timestamps landing within the date
range. But you can also see notes with todo keywords, specific tags,
or other properties if you want. It's pleasantly accommodating to a
variety of workflows.
I use `org-agenda' to show a chronological view into my note taking. I
like being able to look back on an arbitrary period of time, like last
week or last month, to see what I've written. I find the act of review
to build context and confidence in my life.
,----
| Week-agenda (W24):
| Monday 10 June 2024 W24
| Tuesday 11 June 2024
| koreader: 19:42...... Developing
| github_cheatcodes: 20:33...... How to squash many patches
| Wednesday 12 June 2024
| Thursday 13 June 2024
| Friday 14 June 2024
| php: 8:14...... Setup on OpenBSD
| dsc_pc1555: 16:34...... Connecting
| Saturday 15 June 2024
| Sunday 16 June 2024
`----
Building and rebuilding my `org-agenda' is usually very slow. I have
474 notes at the time of this writing. Each of these notes needs to be
opened and parsed to determine its eligibility for inclusion in the
agenda. It takes /minutes/ to build an agenda buffer, which is a huge
deterrent on me actually using the feature.
Many times I've searched the WWW for ways to speed up my `org-agenda'
views. There are some packages that purport to speed things up, but I
found their mechanisms difficult to understand, and their integration
into my existing `org-mode' config excessive. (I like to maintain a
pretty simple set of configuration files for Emacs, about 400 lines.)
So I decided to embark on the task of finding my own way to faster
`org-agenda' views. The remainder of this writing traces my thoughts,
development notes, and outcome.
Development notes
----------------------------------------------------------------------
To start, I ran the Emacs profiler while building a `org-agenda'
buffer build. I observed that the `org-agenda-get-timestamps' function
accounts for 33% of CPU usage and 2682 samples. I figured the high
samples count is probably owed to the many agenda files; again, 474
such notes. So with some confidence I assumed that `org-agenda'
buffers could be built faster if the agenda had fewer files to look
through. Searching the web, I found lots of ad-hoc advice from other
individuals facing a similar problem. I read about hacks to
dynamically populate the `org-agenda-files' variable. Nobody's
specific approach appealed to me, but I got some good ideas. In short,
I decided I'd use `grep' to pre-sift my `org-mode' notes for inclusion
in my agenda.
I was concerned that my approach would get messy. Modifying a variable
crucial to building the agenda could break lots of stuff? But because
the damn thing was so slow and I hardly used it anymore, better to
restore one very-used feature even if it risks breaking some amount of
less-used features.
Next, I looked into the advising functions of Emacs---a very powerful
feature! I had great success using such a function before to fix a bug
with `org-download'. For this task, I imagined a similar approach:
evaluating my function before the `org-agenda' view forms
evaluate. But I'd first need to find the right function to advise. I
experimented with a few that seemed fortuitous.
,----
| (advice-add 'org-agenda-list :before #'roygbyte/org-agenda-list-before)
| (advice-add 'org-agenda-list :around #'roygbyte/org-agenda-list-around)
| (advice-add 'org-agenda-list :filter-args #'roygbyte/org-agenda-list-filter-args)
| (advice-add 'org-agenda-prepare :before #'roygbyte/org-agenda-prepare-before)
`----
I found that advising my function `:before' `org-agenda-list' or
`org-agenda-prepare' successful. At either point, I could grab
information related to the generating agenda buffer. That information
was contained in some of the following variables:
,----
| (print org-agenda-redo-command)
| (print org-agenda-start-day)
| (print org-starting-day)
| (print org-agenda-span)
| (print org-agenda-overriding-arguments)
| (print org-agenda-buffer-name)
| (print org-keys)
`----
I observed the values of these variables in different agenda contexts,
like a week view, custom command, or existing view rebuild. Trial and
error led me to observe the variables that would be helpful for my
operation: `org-agenda-span' and
`org-agenda-overriding-arguments'. (By the way, I found all candidates
for observation by reading through the `org-agenda-list' function, as
well as the `defvar' and `defcustom' variables declared throughout
`org-agenda.el'.) With this, I now had the key parts and could proceed
to writing the code.
I split the logic of the operation into two main parts: identifying
paths to the `org-mode' files of interest for the given agenda; and,
modifying `org-agenda-files' to only contain files associated with the
search query for a given agenda. I began by building first part, which
would in effect be my invocation of `grep' from Emacs.
grep search for relevant files
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
To invoke `grep' from Emacs I use a subprocess. Subprocesses can be
invoked synchronously or asynchronously. I chose the former, since
it's easier to program and better accords with the situation. I built
and tested my `grep' command and its argument in the command line,
then translated it into ELisp. Note that there are many ways to define
a subprocesses' command, and my method is maybe not the best.
,----
| (apply 'call-process "grep" nil t nil "-soE" regex-pattern files-to-search)
`----
I use `apply' to expand the list `files-to-search' into in individual
arguments that are passed to `call-process'. I provide the files as
individual arguments (and not just their enclosing directory) because
it offers more certainty over how `grep' is invoked. Passing a
directory and the `-R' flag would be the alternative to this
approach. But in my trials recursing through a directory opened a host
of problems I decided to avoid.
,----
| (with-temp-buffer
| (setq files-to-search (directory-files
| directory t "\\.org\\'"))
| (apply 'call-process "grep" nil t nil "-soE" regex-pattern files-to-search)
| (goto-char (point-min))
| ;; Clean up buffer
| (while (search-forward ":" nil t)
| (delete-region (- (point) 1) (line-end-position)))
| ;; Translate buffer into list
| (seq-filter (lambda (a)
| (not (string-empty-p a)))
| (split-string (buffer-string) "\n")))
`----
The `grep' subprocess is invoked inside a temporary buffer. When the
subprocess finishes, that buffer contains the results. Unfortunately,
my distros version of `grep' doesn't support only returning matching
filenames, so I have to clean up the buffer. Finally, the buffer is
split by newline into a list and completes by evaluating to that list
of org files matching the initial regular expression.
Ad-hoc modification of org-agenda-files
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The second part of my operation is building the regular expression. In
practice, this function is evaluated first. I only wrote it second
because it seemed the more difficult bit of code
to write. Indeed, it was.
Two values are crucial to building any agenda: start date and
span. The start date corresponds to when the agenda view begins. The
challenge is that `org-agenda' stores these values in different
variables depending on how the agenda is being built. For instance, if
the agenda is being built "fresh", then
`org-agenda-overriding-arguments' will be `nil'. In that case, the
start day is stored in `org-agenda-start-day', unless that variable is
`nil', in which case the start day will be `(org-today)'. However, if
the agenda already exists and is being refreshed or moved to the
previous or next week then the values will all be in
`org-agenda-overriding-arguments'.
I also got tripped up on how `org-mode' represents dates. After some
digging, I learn that `org' represents this value as "the number of
days elapsed since the imaginary Gregorian date Sunday, December 31, 1
BC." I find this by tracing `(org-today)' back to `(date-to-day)'.
Another important detail to building an `org-agenda' view is using the
right start day. I wrote a small function to translate a given day to
the start of the week. For instance: Thursday, October, 3, becomes
Sunday, September 30. This function must also accommodate the
possibility that `org-agenda-start-on-weekday' is set to Monday, which
it could be for strange people who don't start their week on Sunday.
Code
----------------------------------------------------------------------
Below is the finished code. By my eyes, it's not the best looking
Lisp. I still struggle with choosing the best place for line breaks
and indentation. It also seems really long and complicated for such
simple operations. About that: there's a lot of logic required for
ensuring dates are in the correct format, and that edge cases are
accommodated. I don't know if it can be simplified? Maybe. Whatever.
A caveat: my extension doesn't handle non-standard todo keywords or
even tags. I don't use those much, if at all. I provided a minimal
amount of logic for the standard `TODO' keyword, and that's it. I
think extending the function to support other stuff is easy, but I
have no use for it right now.
,----
| (advice-add 'org-agenda-list :before #'roygbyte/org-agenda-list-before)
| (defun roygbyte/org-agenda-list-before (&rest r)
| "Speed up agenda buffer generation by dynamically modifying
| `org-agenda-files`. variable before `org-agenda-list` function
| is evaluated. Performance gains for buffers are largely
| accomplished by using grep to search for org notes containing
| timestamps (or TODOs!) corresponding to the desired time range."
| (cond
| ((or (string-equal org-keys "t")
| (string-equal (car org-agenda-redo-command) "org-todo-list"))
| (setq org-agenda-files
| (delete-dups (roygbyte/files-in-directory-matching-pattern
| (file-truename
Happy helping ☃ here: You tried to output a spurious TAB character. This will break gopher. Please review your scripts. Have a nice day!
| (t (let ((start-day nil)
| (end-day nil)
| (span nil)
| (regex nil))
| (when org-agenda-overriding-arguments
| (setq start-day (roygbyte/day-to-start-of-week
| (nth 1 org-agenda-overriding-arguments)))
| (setq span (let ((span-ambiguous
| (nth 2 org-agenda-overriding-arguments)))
| (if (symbolp span-ambiguous)
| (org-agenda-span-to-ndays span-ambiguous)
| span-ambiguous)))
| (setq end-day (+ start-day span)))
| (when (not start-day)
| (setq start-day
| (cond
| (org-agenda-start-day
| (time-to-days (org-read-date nil t org-agenda-start-day)))
| (org-starting-day org-starting-day)
| (t (roygbyte/day-to-start-of-week (org-today))))))
| (when (not span)
| (setq span (if (numberp org-agenda-span)
| org-agenda-span
| (org-agenda-span-to-ndays org-agenda-span))))
| (when (not end-day)
| (setq end-day (+ start-day span)))
| (setq regex
| (string-join
| (cl-map 'list (lambda (day-number)
| (let ((l (calendar-gregorian-from-absolute day-number)))
| (format "%d-%02d-%02d"
| (nth 2 l) (nth 0 l) (nth 1 l))))
| (number-sequence start-day end-day)) "|"))
| (setq org-agenda-files
| (delete-dups (roygbyte/files-in-directory-matching-pattern
| (file-truename
| (concat org-roam-directory "/pages/")) regex)))))))
|
| (defun roygbyte/day-to-start-of-week (day)
| "Take a DAY, represented by number of days since December 31, 1 BC, and translate
| value so it becomes the start of week."
| (let* ((day-of-week (calendar-day-of-week
| (calendar-gregorian-from-absolute day)))
| (weekday-start org-agenda-start-on-weekday))
| (- day (abs (- weekday-start day-of-week)))))
|
| (defun roygbyte/files-in-directory-matching-pattern (directory regex-pattern)
| "Evaluates to a list of absolute file paths in DIRECTORY whose
| contents match REGEX-PATTERN."
| (with-temp-buffer
| (setq files-to-search (directory-files
| directory t "\\.org\\'"))
| (apply 'call-process
| "grep" nil t nil "-soE" regex-pattern files-to-search)
| (goto-char (point-min))
| (while (search-forward ":" nil t)
| (delete-region (- (point) 1) (line-end-position)))
| (seq-filter (lambda (a)
| (not (string-empty-p a)))
| (split-string (buffer-string) "\n"))))
`----
Footnotes & references
----------------------------------------------------------------------
- `org-roam' was once the cause for a brief crisis in my life. One day,
I rebooted my computer and setup my environment in the usual way, by
running `startx' in my login shell after login. The computer appeared
to "hang". My i3 status bar and other window manager features did not
appear. I was terrified I broke my indispensible device! Well, after a
bit of investigation I uncovered the cause: invokation of `emacs
--daemon' in my Bash RC file, which hanged /forever/ while `org-roam'
attempted to sync files to its database.
-