Upgrading to Emacs 28.0 for native compilation

Emacs 27.1 was released in August 2020. One of the changes was that it introduced was support for libjansson, a C library for working with JSON, which is significantly faster than json.el. A place where this is particularly beneficial is for LSP performance, as the LSP clients and servers communicate using JSON.

This was my motivation for the upgrade. 2 weeks ago the native compilation branch (led by Andrea Corallo and previously named gccemacs) was merged into master (the development branch for what will later become Emacs 28.1). This project uses libgccjit to perform ahead-of-time compilation of emacs-lisp bytecode (.elc files) to native code (new .eln files), which adds general latency improvements to Emacs across the board.

For some reason this did not affect Emacs when Emacs.app/Contents/MacOS/Emacs was executed from the terminal - only when opened as an application. Whenever a package had a dependency on jka-compr, it would hit a recursive load error:

Error (use-package): evil/:catch: Recursive load: "/Applications/Emacs.app/Contents/Resources/lisp/jka-compr.el.gz", "/Applications/Emacs.app/Contents/Resources/lisp/jka-compr.el.gz", "/Applications/Emacs.app/Contents/Resources/lisp/jka-compr.el.gz", "/Applications/Emacs.app/Contents/Resources/lisp/jka-compr.el.gz", "/Applications/Emacs.app/Contents/Resources/lisp/jka-compr.el.gz", "/Applications/Emacs.app/Contents/Resources/lisp/rect.el.gz", "/Users/mattduck/.emacs.d/eln-cache/28.0.50-6e08c520/evil-common-4cbe422e-ef770841.eln", "/Users/mattduck/.emacs.d/elpa/evil-20210424.1855/evil.elc"

There turned out be various issues reported for this (eg. here). It's caused by load-prefer-newer, which is a variable that controls what happens if Emacs finds multiple versions of the same file (.el, elc, .so). When true, Emacs will load the newest one.

The workaround was to disable load-prefer-newer before loading jka-compr:

(setq load-prefer-newer nil)
(require 'jka-compr)
(setq load-prefer-newer t)

This worked fine, but I'm not sure what the root cause is, or why I can find examples of this error going back a few years but I've never seen it before now.

I was having issues with package-refresh-packages hanging, but they disappeared when I removed Marmalade from package-archives. I'm not sure if this was related to the upgrade or not. Either way I've hardly ever used Marmalade so I just removed it from my package-archives definition.

The signature of the builtin function window--display-buffer changed in 27.1 - it removed the fifth argument DEDICATED. For me this broke my fork of shackle, which uses this function in a couple of places. It's fixed in the upstream repo at https://depp.brause.cc/shackle/, so I just pulled in the fix.

This was a weird one. I set pyvenv-workon globally in my init.el, but I also had had a dir-locals setting for a project that was setting pyvenv-workon to a project-specific virtualenv using a symbol like this:

((python-mode
  (pyvenv-workon . foo\.bar)))

After upgrading, when pyvenv-tracking-mode was enabled, python buffers were extremely slow to respond to input. I pinned the issue down to pyvenv and upgraded pyvenv to 20201227.1623, which didn't help. I eventually realised that pyvenv was constantly switching between my dir-locals virtualenv and my global virtualenv.

The reason for the constant virtualenv switching was that pyvenv-tracking-mode runs a function called pyvenv-track-virtualenv on post-command-hook, and this command was comparing pyvenv-virtual-env-name as a string ("foobar") to the dir-locals declaration for pyvenv-workon as a symbol (foo\.bar). Because the string wasn't equal to the symbol, the virtualenv would keep getting reset every time a command was run in the buffer.

Updating the dir-locals declaration to use a string fixed it. I'm not sure what changed to make this issue start occurring now, as it wasn't anything specifically in my setup or pyvenv.

This error appeared on init:

eldoc error: (wrong-number-of-arguments (lambda nil Return breadcrumbs when on a headline, args for src block header-line,
  calls other documentation functions depending on lang when inside src body. (or (org-eldoc-get-breadcrumb) (org-eldoc-get-src-header) (let ((lang (org-eldoc-get-src-lang))) (cond ((or (string= lang emacs-lisp) (string= lang elisp)) (if (fboundp 'elisp-eldoc-documentation-function) (elisp-eldoc-documentation-function) (let (eldoc-documentation-function) (eldoc-print-current-symbol-info)))) ((or (string= lang c) (string= lang C)) (if (require 'c-eldoc nil t) (progn (c-eldoc-print-current-symbol-info)))) ((string= lang css) (if (require 'css-eldoc nil t) (progn (css-eldoc-function)))) ((string= lang php) (if (require 'php-eldoc nil t) (progn (php-eldoc-function)))) ((or (string= lang go) (string= lang golang)) (if (require 'go-eldoc nil t) (progn (go-eldoc--documentation-function)))) (t (let ((doc-fun (org-eldoc-get-mode-local-documentation-function lang))) (if (functionp doc-fun) (progn (funcall doc-fun))))))))) 1)

It was coming from org-eldoc, which is part of org-plus-contrib. It went away when I upgraded from org-plus-contrib version 20200518 to 20210426 (in the org package repo).

I ran into this error in lua-mode:

lua-mode/:catch: Unknown rx form ‘symbol’ Disable showing Disable logging

It can be fixed by removing the existing lua-mode.elc file.

This "wrong number of arguments" error appeared when loading various packages. I didn't even look into where this was occurring as it disappeared as soon as I upgraded the packages. The upgrades were:

After upgrading LSP, my Python setup stopped working (eg. the language server would return errors that it couldn't find various modules for my python projects). This was fixed by upgrading to the latest versions of python-language-server, pyls-black, pyls-mypy and plys-isort.

During native compilation, this warning is displayed for powerline:

Warning (comp): /Users/mattduck/.emacs.d/elpa/powerline-20200105.2053/powerline.el: Error: List contains a loop ("22", . #0)

This is an outstanding issue for powerline. For now the fix is to not compile it by doing (setq comp-deferred-compilation-deny-list '("powerline")).

This was another strange one. In my init.el I was using the use-package :pin argument to pin org-mode to use the org package archive instead of melpa. I had got to the point where everything would work OK when loading Emacs the first time. But after native-compiling init.el, the next time I opened Emacs it would skip my use-package declaration for org-mode. It turned out that this only happened when I included the :pin argument.

Removing :pin fixed the issue, and didn't have any detrimental effect for me because I don't have multiple org versions installed anymore. I'm curious what this use-package issue is though and whether it's specific to my setup.

I was using :load-path with use-package to load some elisp related to managing windows and buffers, which I keep in a separate repo. The first time I opened Emacs this worked fine, but if I opened Emacs after native-compiling, some of my config would error because it referenced void symbols from this external file.

I'm not sure what this problem was - I just copied the contents of the external file inline to init.el to workaround it. This was acceptable for me as I had been planning to move that code inline anyway.

I tried to call build-emacs-for-macos with the --rsvg flag, which is supposed to provide svg support via librsvg. The log output suggested that it was being compiled as expected, but when I opened Emacs svg wasn't supported. I didn't fix this, and haven't looked into what it is yet.