Categories: emacsconf

RSS - Atom - Subscribe via email

EmacsConf 2023 status update: stuff is happening!

| emacs, emacsconf

EmacsConf 2023 is less than a month away. Speakers have been uploading videos, captioning volunteers have been editing away, and I thiiiiink I've gotten most of the infrastructure dusted off. Exciting!

Here's where we are with regard to talk status:

Graphical view of the schedule Schedule for Saturday Saturday 9:00- 9:10 Saturday opening remarks sat-open 9:10- 9:20 An Org-Mode based text adventure game for learning the basics of Emacs, inside Emacs, written in Emacs Lisp adventure 9:30- 9:50 Authoring and presenting university courses with Emacs and a full libre software stack uni 10:05-10:25 Teaching computer and data science with literate programming tools teaching 10:40-10:50 Who needs Excel? Managing your students qualifications with org-table table 11:05-11:15 Taming things with Org Mode taming 11:30-11:50 one.el: the static site generator for Emacs Lisp Programmers one 1:00- 1:10 Emacs turbo-charges my writing writing 1:25- 1:35 Why Nabokov would use Org-Mode if he were writing today nabokov 1:50- 2:10 Collaborative data processing and documenting using org-babel collab 2:20- 2:40 How I play TTRPGs in Emacs solo 2:55- 3:15 Org-Mode workflow: informal reference tracking ref 3:25- 3:35 (Un)entangling projects and repos unentangling 3:45- 3:55 Emacs development updates devel 4:10- 4:50 Emacs core development: how it works core 5:05- 5:15 Saturday closing remarks sat-close 10:00-10:10 MatplotLLM, iterative natural language data visualization in org-babel matplotllm 10:20-10:40 Improving access to AI-assisted literate programming with voice control voice 10:55-11:15 LLM clients in Emacs, functionality and standardization llm 1:00- 1:20 Improving compiler diagnostics with Overlays overlay 1:35- 1:45 Editor Integrated REPL Driven Development for all languages eval 2:00- 2:40 REPLs in strange places: Lua, LaTeX, LPeg, LPegRex, TikZ repl 2:50- 3:30 Literate Documentation with Emacs and Org Mode doc 3:45- 4:05 How we use Org Mode and TRAMP to organize and run a multi-track conference emacsconf 9 AM 10 AM 11 AM 12 PM 1 PM 2 PM 3 PM 4 PM 5 PM Schedule for Sunday Sunday 9:00- 9:05 Sunday opening remarks sun-open 9:05- 9:25 Top 10 ways Hyperbole amps up Emacs hyperamp 9:40-10:00 Using Koutline for stream of thought journaling koutline 10:10-10:20 Parallel Text Replacement: Does P = NP? parallel 10:35-10:45 Eat and Eat powered Eshell, fast featureful terminal inside Emacs eat 11:00-11:20 The browser in a buffer poltys 11:35-11:55 Speedcubing in Emacs cubing 1:00- 1:40 Emacs MultiMedia System (EMMS) emms 1:55- 2:25 Programming at 200 wpm steno 2:35- 2:45 Mentoring VS-Coders as an Emacsian (or How to show not tell people about the wonders of Emacs) mentor 3:10- 3:50 Emacs saves the Web web 4:05- 4:25 Sharing Emacs is Caring Emacs: Emacs education and why I embraced video sharing 4:40- 4:50 Sunday closing remarks sun-close 10:00-10:20 Bringing joy to Scheme programming scheme 10:35-10:55 GNU Emacs: A World of Possibilities world 11:10-11:20 A modern Emacs look-and-feel without pain flat 11:35-11:55 The Emacsen family, the design of an Emacs and the importance of Lisp emacsen 1:00- 1:20 emacs-gc-stats: Does garbage collection actually slow down Emacs? gc 1:35- 2:15 hyperdrive.el: Peer-to-peer filesystem in Emacs hyperdrive 2:30- 2:40 Writing a language server in OCaml for Emacs, fun, and profit lspocaml 2:55- 3:15 What I learned by writing test cases for GNU Hyperbole test 3:30- 4:10 Windows into Freedom windows 9 AM 10 AM 11 AM 12 PM 1 PM 2 PM 3 PM 4 PM 5 PM

Waiting for 26 talks (~550 minutes) out of 42 total. Talks received so far:

  • TO_ASSIGN (waiting for captioning volunteers) - 8 talk(s), 150 minutes: adventure (05:58), matplotllm (09:34), teaching (19:27), nabokov (09:51), collab (19:16), doc (42:45), scheme (21:01), emacsen (18:28)
  • TO_CAPTION - 2 talk(s), 21 minutes: eval (09:35), mentor (10:44)
  • TO_STREAM - 6 talk(s), 124 minutes: llm (20:26), writing (08:53), ref (15:04), emacsconf (15:05), world (22:20), emms (38:38)

Speakers have been really nice about keeping in touch, so I'm not too stressed about gaps in the schedule. Captioning volunteers have been chugging through the talks and OpenAI Whisper's gotten a bit better at spelling things, so that's terrific too. It's so exciting!

zaeph and bandali will probably host the general track and the development track respectively. They've done it for a number of years now, so it'll probably be fine even if we don't have a dry run all together since they've got limited availability. (And we can take on new volunteers if people want to help read questions!)

My stress level is pretty manageable at this point. I can even spend evenings playing video games with the kiddo and weekends going on little bike adventures, so that's awesome. I'm still a little worried about tech hiccups, but we'll probably be able to figure things out.

Next steps are:

  • keep processing videos and captions
  • make the intro videos available so that speakers can correct my pronunciation of their names
  • smoothen out and document the process for last-minute submissions
  • test everything again

It's happening!

#EmacsConf backstage: coordinating captioning volunteers using a backstage area

| emacs, emacsconf
Figure 1: The backstage area

One of the benefits of volunteering for or speaking at EmacsConf is that you get early access to the talks. We upload the files to a password-protected web server. It's nothing fancy, just a directory that has basic authentication and a static index.html generated by Emacs Lisp.

I organize the talks by TODO status so that people can easily see which talks are ready to be captioned. Since captioning can take a little while, we reserve the talks we want to caption. That way, people don't accidentally work on a talk that someone else is already captioning. When people e-mail me to reserve a talk, I move the talk from TO_ASSIGN to TO_CAPTION and add their name in the :CAPTIONER: property.

Publishing the backstage index is done by an Emacs Lisp function that smooshes the information together and then writes the files directly to the server using TRAMP.

(defun emacsconf-publish-backstage-index (&optional filename)
  (setq filename (or filename (expand-file-name "index.html" emacsconf-backstage-dir)))
  (let ((info (or emacsconf-schedule-draft (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
        (emacsconf-main-extensions (append emacsconf-main-extensions emacsconf-publish-backstage-extensions)))
    (with-temp-file filename
      (let* ((talks
               (lambda (o) (append
                            (list :captions-edited t) o))
               (seq-filter (lambda (o) (plist-get o :speakers))
                           (emacsconf-active-talks (emacsconf-filter-talks info)))))
             (by-status (seq-group-by (lambda (o) (plist-get o :status)) talks))
             (files (directory-files emacsconf-backstage-dir)))
         "<html><head><meta charset=\"UTF-8\"><link rel=\"stylesheet\" href=\"/style.css\" /></head><body>"
         (if (file-exists-p (expand-file-name "include-in-index.html" emacsconf-cache-dir))
             (with-temp-buffer (insert-file-contents (expand-file-name "include-in-index.html" emacsconf-cache-dir)) (buffer-string))
         "<p>Schedule by status: (gray: waiting, light yellow: processing, yellow: to assign, light green: captioning, green: captioned and ready)<br />Updated by and the wiki repository</br />"
         (let* ((emacsconf-schedule-svg-modify-functions '(emacsconf-schedule-svg-color-by-status))
                (img (emacsconf-schedule-svg 800 200 info)))
             (mapc (lambda (node)
                      node 'href
                      (concat "#" (dom-attr node 'data-slug))))
                   (dom-by-tag img 'a))
             (svg-print img)
         (if (eq emacsconf-backstage-phase 'prerec)
             (format "<p>Waiting for %d talks (~%d minutes) out of %d total</p>"
                     (length (assoc-default "WAITING_FOR_PREREC" by-status))
                     (emacsconf-sum :time (assoc-default "WAITING_FOR_PREREC" by-status))
                     (length talks))
          (lambda (status)
            (concat "<li>"
                    (if (string= status "TO_ASSIGN")
                        "TO_ASSIGN (waiting for volunteers)"
                    ": "
                    (mapconcat (lambda (o) (format "<a href=\"#%s\">%s</a>"
                                                   (plist-get o :slug)
                                                   (plist-get o :slug)))
                               (assoc-default status by-status)
                               ", ")
          (pcase emacsconf-backstage-phase
            ('harvest '("TO_ARCHIVE" "TO_REVIEW_QA" "TO_INDEX_QA" "TO_CAPTION_QA")))
         (pcase emacsconf-backstage-phase
             (emacsconf-publish-backstage-processing by-status files)
             (emacsconf-publish-backstage-to-assign by-status files)
             (emacsconf-publish-backstage-to-caption by-status files)
             (emacsconf-publish-backstage-to-stream by-status files)))
            (let ((stages
                   '(("TO_REVIEW_QA" .
                      "Please review the --bbb-webcams.webm file and/or the --bbb-webcams.vtt and tell us ( if a Q&amp;A session can be published or if it needs to be trimmed (lots of silence at the end of the recording, accidentally included sensitive information, etc.).")
                     ("TO_INDEX_QA" .
                      "Please review the --answers.webm and --answers.vtt files to make chapter markers so that people can jump to specific parts of the Q&amp;A session. The <a href=\"\">harvesting page on the wiki</a> has some notes on the process. That way, it's easier for people to see the questions and
answers without needing to listen to everything again. You can see <a href=\"\">asmblox</a> for an example of the Q&amp;A chapter markers.")
                     ("TO_CAPTION_QA" .
                      "Please edit the --answers.vtt for the Q&amp;A talk you're interested in, correcting misrecognized words and cleaning it up so that it's nice to use as closed captions. All talks should now have large-model VTTs to make it easier to edit."))))
               (lambda (stage)
                 (let ((status (car stage)))
                    "<h1>%s: %d talk(s) (%d minutes)</h1>%s<ol class=\"videos\">%s</ol>"
                    (length (assoc-default status by-status))
                    (emacsconf-sum :video-time (assoc-default status by-status))
                    (cdr stage)
                     (lambda (f)
                       (format  "<li><a name=\"%s\"></a><strong><a href=\"%s%s\">%s</a></strong><br />%s (id:%s)<br />%s</li>"
                                (plist-get f :slug)
                                (plist-get f :url)
                                (plist-get f :title)
                                (plist-get f :speakers-with-pronouns)
                                (plist-get f :slug)
                                 (append (list
                                          (unless (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.webm") emacsconf-cache-dir))
                                            "<div>No Q&A video for this talk</div>")
                                           ((file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--answers.webm") emacsconf-cache-dir))
                                            (concat (plist-get f :file-prefix) "--answers.webm"))
                                           ((file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.webm") emacsconf-cache-dir))
                                            (concat (plist-get f :file-prefix) "--bbb-webcams.webm"))
                                           (t t)) ;; omit video
                                          :video-id "-qanda"
                                           (emacsconf-surround "QA note: " (plist-get f :qa-note) "<br />")
                                           (format "Q&A archiving: <a href=\"%s-%s.txt\">IRC: %s-%s</a>"
                                                   (format-time-string "%Y-%m-%d" (plist-get f :start-time))
                                                   (plist-get (emacsconf-get-track f) :channel)
                                                   (format-time-string "%Y-%m-%d" (plist-get f :start-time))
                                                   (plist-get (emacsconf-get-track f) :channel))
                                           (emacsconf-surround ", <a href=\""
                                                               (if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--pad.txt")
                                                                   (concat (plist-get f :file-prefix) "--pad.txt"))
                                                               "\">Etherpad (Markdown)</a>" "")
                                           (emacsconf-surround ", <a href=\"" (plist-get f :bbb-playback) "\">BBB playback</a>" "")
                                           (emacsconf-surround ", <a href=\""
                                                               (if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb.txt")
                                                                   (concat (plist-get f :file-prefix) "--bbb.txt"))
                                                               "\">BBB text chat</a>" "")
                                           (emacsconf-surround ", <a href=\""
                                                               (if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.opus")
                                                                   (concat (plist-get f :file-prefix) "--bbb-webcams.opus"))
                                                               "\">BBB audio only</a>" ""))
                                          :files (emacsconf-publish-talk-files f files))
                     (assoc-default status by-status) "\n"))))
         (if (file-exists-p (expand-file-name "include-in-index-footer.html" emacsconf-cache-dir))
             (with-temp-buffer (insert-file-contents (expand-file-name "include-in-index-footer.html" emacsconf-cache-dir)) (buffer-string))

For example, the section for talks that are waiting for volunteers is handled by the function emacsconf-publish-backstage-to-assign.

(defun emacsconf-publish-backstage-to-assign (by-status files)
  (let ((list (assoc-default "TO_ASSIGN" by-status)))
    (format "<h1>%s talk(s) to be captioned, waiting for volunteers (%d minutes)</h1><p>You can e-mail <a href=\"\"></a> to call dibs on editing the captions for one of these talks. We use OpenAI Whisper to provide auto-generated VTT that you can use as a starting point, but you can also write the captions from scratch if you like. If you're writing them from scratch, you can choose to include timing information, or we can probably figure them out afterwards with a forced alignment tool. More info: <a href=\"\">captioning tips</a></p><ul class=\"videos\">%s</ul>"
            (length list)
            (emacsconf-sum :video-time list)
             (lambda (f)
               (setq f (append
                        (list :extra
                              (if (plist-get f :caption-note) (concat "<div class=\"caption-note\">" (plist-get f :caption-note) "</div>") "")
                              (emacsconf-publish-talk-files f files))))
               (format  "<li><a name=\"%s\"></a><strong><a href=\"%s\">%s</a></strong><br />%s (id:%s)<br />%s</li>"
                        (plist-get f :slug)
                        (plist-get f :absolute-url)
                        (plist-get f :title)
                        (plist-get f :speakers)
                        (plist-get f :slug)
                        (emacsconf-publish-index-card f)))

Each talk has a little card that includes its video and links to files.

emacsconf-publish-index-card: Format an HTML card for TALK, linking the files in EXTENSIONS.
(defun emacsconf-publish-index-card (talk &optional extensions)
  "Format an HTML card for TALK, linking the files in EXTENSIONS."
  (let* ((file-prefix (plist-get talk :file-prefix))
         (video-file (plist-get talk :video-file))
         (video (and file-prefix
                      (or (plist-get talk :video-id)
                          (concat (plist-get talk :slug) "-mainVideo"))
                      video-file talk extensions))))
    ;; Add extra information to the talk
    (setq talk
            :time-info (emacsconf-surround "Duration: " (plist-get talk :video-duration) " minutes" "")
            :video-html (or (plist-get video :video) "")
            :audio-html (or (plist-get video :audio) "")
            :chapter-list (or (plist-get video :chapter-list) "")
            :resources (or (plist-get video :resources) "")
            :extra (or (plist-get talk :extra) "")
            :speaker-info (or (plist-get talk :speakers-with-pronouns) ""))))
     "<div class=\"vid\">${video-html}${audio-html}<div>${extra}</div>${time-info}${resources}${chapter-list}</div>")))

So it's all built on templates and a little bit of code to make the appropriate lists. You can find this code in emacsconf-publish.el.

#EmacsConf backstage: automatically updating talk status from the crontab

| emacs, emacsconf, org

Now that I've figured out how to automatically play EmacsConf talks with crontab, I want to update our approach to using TRAMP and timers to run two tracks semi-automatically.

When a talk starts playing, I would like to:

  • announce it on IRC so that the talk details split up the chat log and make it easier to read
  • publish the media files to (currently password-protected while testing) so that people can view them
  • update the talk wiki page with the media files and the captions
  • update the Youtube and Toobnix videos so that they're public: this is manual for the moment, since I haven't put time into automating it yet

I have code for most of this, and we've done it successfully for the past couple of years. I just need to refamiliarize myself with how to do it and how to set it up for testing during the dry run, and modify it to work with the new crontab-based system.

Setting things up

Last year, I set up the server with the ability to act as the controller, so that it wasn't limited to my laptop. The organizers notebook says I put it in the orga@ user's account, and I probably ran it under a screen session. I've submitted a talk for this year's conference, so I can use a test video for that one. First, I need to update the Ansible configuration for publishing and editing:

ansible-playbook -i inventory.yml prod-playbook.yml --tags publish,edit

I needed to add a version attribute to the git repo checkout in the Ansible playbook, since we'd switched from master to main. I also needed to set emacs_version to 29.1 since I started using seq-keep in my Emacs Lisp functions. For testing, I set emacsconf-publishing-phase to conference.

Act on TODO state changes

org-after-todo-state-change-hook makes it easy to automatically run functions when the TODO state changes. I add this hook that runs a list of functions and passes the talk information so that I don't have to parse the talk info in each function.

emacsconf-org-after-todo-state-change: Run all the hooks in ‘emacsconf-todo-hooks’.
(defun emacsconf-org-after-todo-state-change ()
  "Run all the hooks in `emacsconf-todo-hooks'.
If an `emacsconf-todo-hooks' entry is a list, run it only for the
tracks with the ID in the cdr of that list."
  (let* ((talk (emacsconf-get-talk-info-for-subtree))
         (track (emacsconf-get-track (plist-get talk :track))))
     (lambda (hook-entry)
        ((symbolp hook-entry) (funcall hook-entry talk))
        ((member (plist-get track :id) (cdr hook-entry))
         (funcall (car hook-entry) talk))))

This can be enabled and disabled with the following functions.

emacsconf-add-org-after-todo-state-change-hook: Add FUNC to ‘org-after-todo-stage-change-hook’.
(defun emacsconf-add-org-after-todo-state-change-hook ()
  "Add FUNC to `org-after-todo-stage-change-hook'."
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (add-hook 'org-after-todo-state-change-hook #'emacsconf-org-after-todo-state-change nil t)))

emacsconf-remove-org-after-todo-state-change-hook: Remove FUNC from ‘org-after-todo-stage-change-hook’.
(defun emacsconf-remove-org-after-todo-state-change-hook ()
  "Remove FUNC from `org-after-todo-stage-change-hook'."
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (remove-hook 'org-after-todo-state-change-hook
                 #'emacsconf-org-after-todo-state-change  t)))

Announce on IRC

This is still much the same as last year.

emacsconf-erc-announce-on-change: Announce talk.
(defun emacsconf-erc-announce-on-change (talk)
  "Announce talk."
  (let ((func
         (pcase org-state
           ("PLAYING" #'erc-cmd-NOWPLAYING)
           ("CLOSED_Q" #'erc-cmd-NOWCLOSEDQ)
           ("OPEN_Q" #'erc-cmd-NOWOPENQ)
           ("UNSTREAMED_Q" #'erc-cmd-NOWUNSTREAMEDQ)
           ("TO_ARCHIVE" #'erc-cmd-NOWDONE))))
    (when func
      (funcall func talk))))

Here's a sample command that announces that the talk is now playing.

erc-cmd-NOWPLAYING: Set the channel topics to announce TALK.
(defun erc-cmd-NOWPLAYING (talk)
  "Set the channel topics to announce TALK."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk)
    (setq talk (or (emacsconf-find-talk-info talk) (error "Could not find talk %s" talk))))
  ;; Announce it in the track's channel
  (if (emacsconf-erc-recently-announced (format "---- %s:" (plist-get talk :slug)))
      (message "Recently announced, skipping")
    (when (plist-get talk :track)
      (emacsconf-erc-with-channels (list (concat "#" (plist-get talk :channel)))
          "%s: %s (%s) pad: %s Q&A: %s | %s"
          (plist-get talk :slug)
          (plist-get talk :title)
          (plist-get talk :speakers)
          (plist-get talk :pad-url)
          (plist-get talk :qa-info)
          (car (assoc-default
                (concat "#" (plist-get talk :channel))
        (erc-send-message (format "---- %s: %s - %s ----"
                                  (plist-get talk :slug)
                                  (plist-get talk :title)
                                  (plist-get talk :speakers-with-pronouns)))
         (concat "Add your notes/questions to the pad: " (plist-get talk :pad-url)))
         ((string-match "live" (or (plist-get talk :q-and-a) ""))
          (erc-send-message (concat "Live Q&A: " (plist-get talk :bbb-redirect))))
         ((plist-get talk :irc)
          (erc-send-message (format "or discuss the talk on IRC (nick: %s)"
                                    (plist-get talk :irc)))))))
    ;; Short announcement in #emacsconf
    (emacsconf-erc-with-channels (list emacsconf-erc-hallway emacsconf-erc-org)
      (erc-send-message (format "-- %s track: %s: %s (watch: %s, pad: %s, channel: #%s)"
                                (plist-get talk :track)
                                (plist-get talk :slug)
                                (plist-get talk :title)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url)
                                (plist-get talk :channel))))))

Because the commands change the topic, I need to op the user in all the Emacsconf channels.

(defun erc-cmd-OPALL (&optional nick)
  (emacsconf-erc-with-channels (mapcar 'car emacsconf-topic-templates)
    (if nick
        (erc-cmd-OP nick)

This code is in emacsconf-erc.el.

Publish media files

We used to scramble to upload all the videos in the days or weeks after the conference, since the presentations were live. Since we switched to encouraging speakers to upload videos before the conference, we've been able to release the videos pretty much as soon as the talk starts playing. This code in emacsconf-publish.el takes care of copying the files from the backstage to the public media directory, republishing the index, and republishing the playlist. That way, people who come in late or who want to refer to the video can easily get the full video right away.

emacsconf-publish-media-files-on-change: Publish the files and update the index.
(defun emacsconf-publish-media-files-on-change (talk)
  "Publish the files and update the index."
  (interactive (list (emacsconf-complete-talk-info)))
  (let ((org-state (if (boundp 'org-state) org-state (plist-get talk :status))))
    (if (plist-get talk :public)
        ;; Copy main extension files from backstage to public
        (let ((files (directory-files emacsconf-backstage-dir nil
                                      (concat "^"
                                              (regexp-quote (plist-get talk :file-prefix))
                                              (regexp-opt emacsconf-main-extensions)))))
          (mapc (lambda (file)
                  (when (and
                         (not (file-exists-p (expand-file-name file emacsconf-public-media-directory)))
                         (or (not (string-match "--main.vtt$" file))
                             (plist-get talk :captions-edited)))
                    (copy-file (expand-file-name file emacsconf-backstage-dir)
                               (expand-file-name file emacsconf-public-media-directory) t)))
      ;; Remove files from public
      (let ((files (directory-files emacsconf-public-media-directory nil
                                    (concat "^"
                                            (regexp-quote (plist-get talk :file-prefix)
        (mapc (lambda (file)
                (delete-file (expand-file-name file emacsconf-public-media-directory)))
     (expand-file-name "index.m3u" emacsconf-public-media-directory)
     (concat emacsconf-name " " emacsconf-year)
     (emacsconf-public-talks (emacsconf-get-talk-info)))))

The :public property is automatically added by this function based on the TODO status or the time:

emacsconf-add-talk-status: Add status label and public info.
(defun emacsconf-add-talk-status (o)
  "Add status label and public info."
  (plist-put o :status-label
             (or (assoc-default (plist-get o :status)
                                emacsconf-status-types 'string= "")
                 (plist-get o :status)))
  (when (or
         (member (plist-get o :status)
         (time-less-p (plist-get o :start-time)
    (plist-put o :public t))

Update the wiki page

This function updates the schedule page and the page for the talk. It's also in emacsconf-publish.el.

emacsconf-publish-update-talk: Publish the schedule page and the page for this talk.
(defun emacsconf-publish-update-talk (talk)
  "Publish the schedule page and the page for this talk."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk) (setq talk (emacsconf-resolve-talk talk)))
  (when (functionp 'emacsconf-upcoming-insert-or-update)
    (let ((info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
      (emacsconf-publish-before-page talk info)
      (emacsconf-publish-after-page talk info)
      (emacsconf-publish-schedule info))))

It uses the publishing functions I described in this post on adding talks to the wiki.

This macro commits changes when emacsconf-publish-autocommit-wiki is t, so I need to set that also.

(defmacro emacsconf-publish-with-wiki-change (&rest body)
  (declare (indent 0) (debug t))
      (and (stringp ,(car body)) ,(car body)))))

Make the videos public on YouTube and Toobnix

This is low-priority, but it might be nice to figure out. The easiest way is probably to use open the Youtube/Toobnix URLs on my computer and then use either Tampermonkey or Spookfox to set the talk to public. Someday!

Update the talk status on the server

Last year, I experimented with having the shell scripts automatically update the status of the talk from TO_STREAM to PLAYING and from PLAYING to CLOSED_Q. Since I've moved the talk-running into track-specific crontabs, now I need to sudo back to the orga user and set XDG_RUNTIME_DIR in order to use emacsclient. I can call this with sudo -u orga talk $slug $status in the roles/obs/templates/handle-session script.

Here's the Ansible template for roles/prerec/templates/talk. It uses getent to look up the user ID.

# How to use: talk slug from-status-regexp to-status
# or talk slug to-status


if [ "x$TO_STATUS" == "x" ]; then
#echo "Pulling"
#git pull
echo "Updating status..."
XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR emacsclient --eval "(emacsconf-with-todo-hooks (emacsconf-update-talk-status \"$SLUG\" \"$FROM_STATUS\" \"$TO_STATUS\"))" -a emacs
#echo "Committing and pushing in the background"
#git commit -m "Update task status for $SLUG from $FROM_STATUS to $TO_STATUS"
#git push &

Testing notes

Looks like everything works fine when I run it from the crontab: the talk status is updated, the media files are published, the wiki is updated, and the talks are announced on IRC. Backup plan A is to manually control the talk status using Emacs on the server. Backup plan B is to control the talk status using Emacs on my laptop. Backup plan C is to call the individual functions instead of relying on the todo state change functions. I think it'll all work out, although I'll probably want to do another dry run at some point to make sure. Slowly getting there…

#EmacsConf backstage: autopilot with crontab

| emacs, emacsconf

[2023-10-26 Thu]: updated handle-session and added talk

I figured out multi-track streaming so close to EmacsConf 2022 that there wasn't enough time to get other volunteers used to working with the setup, especially since I was still scrambling to figure out more infrastructure as the conference approached. We decided I'd run both streams myself, which meant I needed to make things as automatic as possible so that I wouldn't go crazy. I wanted a lot of things to happen automatically: playing recorded intros and videos, browsing to the right URLs depending on the type of Q&A, publishing updates to the wiki, and so on.

I used timers and TODO state changes to execute commands via TRAMP, which was pretty cool for the most part. But it turned out TRAMP doesn't like being called when it's already running, like when it's being called from two timers going off at the same time. It gives a "Forbidden reentrant call of TRAMP". We found a couple of quick workarounds: I could reschedule the talks to be a minute apart, or I could cancel the conflicting timer and just start them with the shell scripts.

Last year, we had a shell script that played the intro and the main talk, and other scripts to handle the Q&A by opening BigBlueButton, Etherpad, or the IRC channel. Much of the logic was in Emacs Lisp because it was easy to write it that way. For this year, I wanted to write a script that handled the intro, video, and Q&A portions. This is now in roles/obs/templates/handle-session.

# Handle the intro/talk/Q&A for a session
# Usage: handle-session $SLUG


# Kill background music if playing
if screen -list | grep -q background; then
    screen -S background -X quit

# Update the status
sudo -u  talk $SLUG PLAYING &

# Update the overlay
overlay $SLUG

# Play the intro if it exists. If it doesn't exist, switch to the intro slide and stop processing.

if [[ -f $BASE_DIR/assets/intros/$SLUG.webm ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $BASE_DIR/assets/intros/$SLUG.webm
  firefox --kiosk $BASE_DIR/assets/in-between/$SLUG.png
  exit 0

# Play the video if it exists. If it doesn't exist, switch to the BBB room and stop processing.
if [ "x$TEST_MODE" = "x" ]; then
if [ ! -f "$FILE" ]; then
    # Is there an original file?

if [[ -f $FILE ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $FILE
  /usr/local/bin/bbb $SLUG
  exit 0

sudo -u  talk $SLUG CLOSED_Q &

# Open the appropriate Q&A URL
QA=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-backstage-url"]' < $BASE_DIR/talks.json)
QA_TYPE=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-type"]' < $BASE_DIR/talks.json)
if [ "$QA_TYPE" = "live" ]; then
  /usr/local/bin/bbb $SLUG
elif [ "$QA" != "null" ]; then
  /usr/local/bin/music &
  /usr/bin/firefox $QA
  # i3-msg 'layout splith'

It builds on roles/obs/templates/bbb, roles/obs/templates/overlay, and roles/obs/templates/music. I also have a roles/prerec/templates/talk script that uses emacsclient to update the status of the talk.

I wrote some Tampermonkey scripts to automate joining the web conference and the IRC channel.

Now that we have a script that handles all the different things related to a session, it's easier to schedule the execution of that script. Instead of using Emacs timers and running into that problem with tramp, I want to try using cron. Cron is a standard UNIX and Linux tool for scheduling things to run at certain times. You make a plain text file in a particular format: minute, hour, day of month, month, day of week, and then the command, and then you tell cron to use that file with something like crontab your-file. Since it's plain text, we can generate it with Emacs Lisp and format-time-string, save with TRAMP, and install with ssh. Each track has its own user account for streaming, so each track can have its own file.

emacsconf-stream-format-crontab: Return crontab entries for TALKS.
(defun emacsconf-stream-format-crontab (track talks &optional test-mode)
  "Return crontab entries for TALKS.
Use the display specified in TRACK.
If TEST-MODE is non-nil, load the videos from the test directory."
" (plist-get track :uid))
    (lambda (talk)
      (format "%s /usr/bin/screen -dmS play-%s bash -c \"DISPLAY=%s TEST_MODE=%s /usr/local/bin/handle-session %s | tee -a ~/track.log\"\n"
              ;; cron times are UTC
              (format-time-string "%-M %-H %-d %m *" (plist-get talk :start-time))
              (plist-get talk :slug)
              (plist-get track :vnc-display)
              (if test-mode "1" "")
              (plist-get talk :slug)))
    (emacsconf-filter-talks talks))))

emacsconf-stream-crontabs: Write the streaming users’ crontab files.
(defun emacsconf-stream-crontabs (&optional test-mode info)
  "Write the streaming users' crontab files.
If TEST-MODE is non-nil, use the videos in the test directory.
If INFO is non-nil, use that as the schedule instead."
  (let ((emacsconf-publishing-phase 'conference))
    (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
    (dolist (track emacsconf-tracks)
      (let ((talks (seq-filter (lambda (talk)
                                 (string= (plist-get talk :track)
                                          (plist-get track :name)))
            (crontab (expand-file-name (concat (plist-get track :id) ".crontab")
                                       (concat (plist-get track :tramp) "~"))))
        (with-temp-file crontab
          (when (plist-get track :autopilot)
            (insert (emacsconf-stream-format-crontab track talks test-mode))))
        (emacsconf-stream-track-ssh track (concat "crontab ~/" (plist-get track :id) ".crontab"))))))

I want to test the whole setup before the conference, of course. First, I needed test videos. This generates test videos and subtitles following our naming convention.

(defun emacsconf-stream-generate-test-videos (&optional info)
  "Generate 1-minute test videos for INFO."
  (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
  (let* ((dir (expand-file-name "test" emacsconf-stream-asset-dir))
         (default-directory dir)
         (subed-default-subtitle-length 1000)
         (test-length 60))
    (unless (file-directory-p dir)
      (make-directory dir t))
     (format "ffmpeg -y -f lavfi -i testsrc=duration=%d:size=1280x720:rate=10 -i background-music.opus -shortest %s "
             test-length (expand-file-name "template.webm" dir)))
    (dolist (talk info)
      (with-temp-file (expand-file-name (concat (plist-get talk :file-prefix) "--main.vtt") dir)
        (dotimes (i test-length)
           (* i 1000)
           (1- (* i 1000))
           (format "%s %02d %s"
                   (plist-get talk :slug)
                   (substring "123456789 123456789 123456789 123456789 123456789 123456789 "
                              (1+ (length (format "%s %02d" (plist-get talk :slug) i))))))))
       (expand-file-name "template.webm" dir)
       (expand-file-name (concat (plist-get talk :file-prefix) "--main.webm") dir)

Then I needed to write a crontab based on a different schedule. This code sets up a series of test videos to start about a minute after I run the code, with the dev stream set up to start a minute after the gen stream.

(let* ((offset-seconds 60)
       (start-time (time-add (current-time) offset-seconds))
       (emacsconf-schedule-validation-functions nil)
       (emacsconf-schedule-default-buffer-minutes 1)
       (emacsconf-schedule-default-buffer-minutes-for-live-q-and-a 1)
       (emacsconf-schedule-strategies '(emacsconf-schedule-allocate-buffer-time
       (schedule (emacsconf-schedule-prepare
                      :start ,(format-time-string "%Y-%m-%d %H:%M" start-time)
                      :set-track "General")
                     (sat-open :time 1)
                     (uni :time 1) ; live Q&A
                     (adventure :time 1) ; pad Q&A
                      ,(format-time-string "%Y-%m-%d %H:%M" (time-add start-time 60))
                      :set-track "Development")
                     (repl :time 1) ; IRC
                     (matplotllm :time 1) ; pad
                     (voice :time 1) ; live
  (emacsconf-stream-crontabs t schedule))

That generates gen.crontab and dev.crontab. This is what gen.crontab looks like for testing:

35 11 26 10 * /usr/bin/screen -dmS play-sat-open bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session sat-open | tee -a ~/track.log"
36 11 26 10 * /usr/bin/screen -dmS play-uni bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session uni | tee -a ~/track.log"
38 11 26 10 * /usr/bin/screen -dmS play-adventure bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session adventure | tee -a ~/track.log"

The result: for both tracks, the intro videos play, the test videos play, and web browsers go to the right places for the Q&A.

In case I need to resume manual control:

emacsconf-stream-cancel-crontab: Remove crontab for TRACK.
(defun emacsconf-stream-cancel-crontab (track)
  "Remove crontab for TRACK."
  (interactive (list (emacsconf-complete-track)))
  (plist-put track :autopilot nil)
  (emacsconf-stream-track-ssh track "crontab -r"))

emacsconf-stream-cancel-all-crontabs: Remove crontabs.
(defun emacsconf-stream-cancel-all-crontabs ()
  "Remove crontabs."
  (dolist (track emacsconf-tracks)
    (plist-put track :autopilot nil)
    (emacsconf-stream-track-ssh track "crontab -r")))

Here are some things I learned along the way:

  • I needed to use timedatectl set-timezone America/Toronto to change the server's timezone to America/Toronto so that the crontab would run at the right time.

    In Ansible terms, that's:

    	- name: Set system timezone
    		tags: tz
    			name: ""
    	- name: Restart cron
    		tags: tz
    			name: cron
    			state: restarted
  • I also needed to specify the PATH so that I didn't need to add the absolute paths in all the other shell scripts, XDG_RUNTIME_DIR to get audio working, and DISPLAY so that windows showed up in the right place.

I think this will let me run both tracks for EmacsConf with more ease and less frantic juggling. We'll see!

#EmacsConf backstage: Automatically join BigBlueButton web conferences using Tampermonkey

| emacs, emacsconf

[2023-10-22 Sun] Added IRC script

During EmacsConf, we join BigBlueButton webconferences for live presentations and Q&A sessions so that the speakers and the host can be on stream. I wanted to reduce the number of manual steps needed to join the web conference, since any clicks or keystrokes would need to be done via a VNC connection. I used Tampermonkey to write a script to join BigBlueButton and set things up the way we want to.

I don't have the Tampermonkey installation and setup automated with Ansible yet, but here's what I did for each track:

  1. Install the Tampermonkey extension by going to .
  2. Install the script by clicking on the Tampermonkey extension, choosing Install New Script, and pasting in the following:

       // ==UserScript==
       // @name         Emacsconf BBB setup
       // @namespace
       // @version      0.1
       // @description  Join BBB and set things up
       // @author       You
       // @match*
       // @icon
       // @grant        none
       // ==/UserScript==
           async function() {
               'use strict';
               const NAME = 'emacsconf';
               async function waitUntil(conditionFunc, interval=500, timeout=null) {
                   let initResult = conditionFunc();
                   if (initResult) return initResult;
                   return new Promise((resolve, reject) => {
                       let timeSoFar = 0;
                       let timer = setInterval(() => {
                           let result = conditionFunc();
                           if (result) {
                           timeSoFar += interval;
                           if (timeout && timeSoFar > timeout) {
                       }, interval);
               if (document.querySelector('input.join-form')) {
                   document.querySelector('input.join-form').value = NAME;
               await waitUntil(() => document.querySelector('.icon-bbb-listen')).then((e) => e.closest('button').click());
               await waitUntil(() => document.querySelector('.icon-bbb-user')).then((e) => e.closest('button').click());

    Press Ctrl+s to save.

  3. Add this script to join the IRC channel:

       // ==UserScript==
       // @name         Connect to EmacsConf chat automatically
       // @namespace
       // @version      0.1
       // @description  try to take over the world!
       // @author       You
       // @match*
       // @icon
       // @grant        none
       // ==/UserScript==
       (function() {
           'use strict';
           setTimeout(() => {
               if (document.querySelector('.connect-row')) {
           }, 1000);
  4. Join an BBB meeting. Check the address bar to see if autoplay is disabled (crossed-out autoplay icon). If it is, click on it and change Block audio to Allow audio and video.

Now I can use this Ansible template for a shell script to connect to the BBB session (roles/obs/templates/bbb):

# Open the Big Blue Button room using the backstage link

# Kill the background music if playing
if screen -list | grep -q background; then
    screen -S background -X quit

# Update the overlay
overlay $SLUG
killall -s TERM firefox-esr
firefox$SLUG.html &
sleep 5
xdotool search --class firefox windowactivate --sync
xdotool key Return
xdotool key F11

The overlay is handled by this script (roles/obs/templates/overlay):


SLUG=$(echo "$1" | perl -ne 'if (/emacsconf-[0-9]*-(.*?)--/) { print $1; } else { print; }')

if [[ -f /assets/overlays/$SLUG-other.svg.png ]]; then
    echo "Found other overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-other.svg.png ~/other.png
elif [[ -f /assets/overlays/$SLUG-video.svg.png ]]; then
    echo "Found video overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-video.svg.png ~/other.png
    echo "Could not find /assets/overlays/$SLUG-other.svg.png, please override ~/other.png manually"
    cp /assets/overlays/blank-other.svg.png ~/other.png
if [[ -f /assets/overlays/$SLUG-video.svg.png ]]; then
    echo "Found video overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-video.svg.png ~/video.png
    echo "Could not find /assets/overlays/$SLUG-video.svg.png, override ~/video.png manually"
    cp /assets/overlays/blank-video.svg.png ~/video.png

The result: I can run bbb uni and it'll automatically join the Q&A session for the uni talk in listen-only mode and with the user list hidden.

Getting Mermaid JS and ob-mermaid running on my system - needed to symlink Chromium for Puppeteer

| emacsconf, nodejs, org, emacs

I wanted to use Mermaid to make diagrams, but I ran into this issue when trying to run it:

Error: Could not find Chromium (rev. 1108766). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
 2. your cache path is incorrectly configured (which is: /home/sacha/.cache/puppeteer).
For (2), check out our guide on configuring puppeteer at

It turns out that I needed to do the following:

sudo npm install -g puppeteer mermaid @mermaid-js/mermaid-cli --unsafe-perm
# Cache Chromium for my own user
node /usr/lib/node_modules/puppeteer/install.js --unsafe-perm
sudo npm install -g mermaid @mermaid-js/mermaid-cli
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149 ~/.cache/puppeteer/chrome/linux-1108766
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux64 ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux

(The exact versions might be different for your installation.)

Then I could make a Mermaid file and try it out with mmdc -i input.mmd -o output.svg, and then I could confirm that it works directly from Org with ob-mermaid:

    input ->> res: original.mp4;
    res ->> backstage: reencoded.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: main.vtt;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: normalized.opus;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    backstage ->> media: public files;

#EmacsConf backstage: adding notes to Org logbook drawers from e-mails

| emacs, emacsconf, notmuch

Sometimes I want to work with all the talks associated with an email in my inbox. For example, maybe a speaker said that the draft schedules are fine, and I want to make a note of that in the conference Org file.

First we start with a function that gets the e-mail addresses for a talk. Some speakers have different e-mail addresses for public contact or private contact, and some e-mail us from other addresses.

emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
   " *, *"))

Then we can use that to find the talks for a given e-mail address.

emacsconf-mail-talks: Return a list of talks matching EMAIL.
(defun emacsconf-mail-talks (email)
  "Return a list of talks matching EMAIL."
  (setq email (downcase (mail-strip-quoted-names email)))
   (lambda (o) (member email (emacsconf-mail-get-all-email-addresses o)))

We can loop over that to add a note for the e-mail.

emacsconf-mail-add-to-logbook: Add to logbook for all matching talks from this speaker.
(defun emacsconf-mail-add-to-logbook (email note)
  "Add to logbook for all matching talks from this speaker."
   (let* ((email (mail-strip-quoted-names
                  (plist-get (plist-get (notmuch-show-get-message-properties) :headers)
          (talks (emacsconf-mail-talks email)))
      (read-string (format "Note for %s: "
                           (mapconcat (lambda (o) (plist-get o :slug))
                                      talks", "))))))
     (lambda (talk)
       (emacsconf-add-to-talk-logbook talk note))
     (emacsconf-mail-talks email))))

The actual addition of notes is handled by these functions.

emacsconf-add-to-logbook: Add NOTE as a logbook entry for the current subtree.
(defun emacsconf-add-to-logbook (note)
  "Add NOTE as a logbook entry for the current subtree."
  (move-marker org-log-note-return-to (point))
  (move-marker org-log-note-marker (point))
    (insert note)
    (let ((org-log-note-purpose 'note))

Then we have a function that looks for the heading for a note and then adds a logbook entry to it.

emacsconf-add-to-talk-logbook: Add NOTE as a logbook entry for TALK.
(defun emacsconf-add-to-talk-logbook (talk note)
  "Add NOTE as a logbook entry for TALK."
  (interactive (list (emacsconf-complete-talk) (read-string "Note: ")))
    (emacsconf-with-talk-heading talk
      (emacsconf-add-to-logbook note))))

All together, that makes it easy to use Emacs as a very simple contact relationship management system where I can take notes based on the e-mails that come in.

Figure 1: Logging notes from e-mail

These functions are in emacsconf-mail.el.