Enable LuaMetaTeX in ConTeXt tests and examples

[Witiko]

This PR enables LuaMetaTeX in ConTeXt tests and examples after #551.

[Witiko]

I wanted to test the new support for ConTeXt LMTX but I am still seeing some issues, namely the lack of support for the KPathSea library in LMTX, even in TeX Live, see the CI failure from this PR:

make FAIL_FAST=true test
make -C tests
make[1]: Entering directory '/__w/markdown/markdown/tests'
find testfiles/ -type f -name '*.test' -exec ./test.sh  {} +

Creating a Python virtual environment in /__w/markdown/markdown/tests/test-virtualenv.
2025-02-21 10:09:14,197 Running tests for 836 testfiles.
2025-02-21 10:09:14,197 Will fail at first error.
Testfile testfiles/regression/github/issue-508-fancy-lists.test:

  Some commands produced non-zero exit codes:
  - Command context [...] --luatex [...] test.tex [...] exited successfully.
  - Command context [...]          [...] test.tex [...] produced exit code 1.
  
  Some commands produced unexpected outputs:
  - Command context [...] --luatex [...] test.tex [...] produced expected output.
  - Command context [...]          [...] test.tex [...] produced unexpected output with the following diff:
  
    [...]

make[1]: *** [Makefile:13: all] Error 1
make[1]: Leaving directory '/__w/markdown/markdown/tests'
make: *** [Makefile:213: test] Error 2

Looking into the file test.log, I see the following:

lua error       > lua error on line 5 in file ./test.tex:

token call, execute: ...live/2024/texmf-dist/tex/context/base/mkiv/l-sandbox.lua:180: module 'kpse' not found:
	no field package.preload['kpse']
	no file '/usr/local/share/lua/5.5/kpse.lua'
	no file '/usr/local/share/lua/5.5/kpse/init.lua'
	no file '/usr/local/lib/lua/5.5/kpse.lua'
	no file '/usr/local/lib/lua/5.5/kpse/init.lua'
	no file './kpse.lua'
	no file './kpse/init.lua'
stack traceback:
	[C]: in upvalue 'requiem'
	...live/2024/texmf-dist/tex/context/base/mkiv/l-sandbox.lua:180: in function <...live/2024/texmf-dist/tex/context/base/mkiv/l-sandbox.lua:165>
	(...tail calls...)
	[ctxlua]:2: in main chunk
 1     % Load the package.
 2     \startluacode
 3     local kpse = require("kpse")
 4     kpse.set_program_name("luatex")
 5 >>  \stopluacode
 6     \usemodule[t][markdown]
 7     
 8     % Load the support files.
 9     \setupmarkdown [
10       eagerCache = false,
11       import = {
12         witiko/markdown/test = snippet as testsnippet,
13       }
14     ]
15

This is perhaps to be expected: Even if KPathSea is available for LuaMetaTeX in TeX Live, we apparently need to use the predefined object optional.kpse instead of require("kpse") according to luametatex.pdf. Regardless, KPathSea is unlikely to be widely available in ConTeXt standalone and therefore, we shouldn't rely on it.

[Witiko]

@andreiborisov: Can you please test that you can compile the example file examples/context-lmtx.tex using LMTX and version 3.11.0 of the Markdown package for TeX? You will need to remove the following part that loads KPathSea:

\startluacode
local kpse = require("kpse")
kpse.set_program_name("luatex")
\stopluacode

Furthermore, you will also need to copy all dependencies to the directory examples/. As discussed in #402 (comment), this includes at least the following files and packages from CTAN:

1. The expl3-generic.tex plain TeX file from l3kernel
2. The lua-uni-algos Lua package
3. The lt3luabridge plain TeX package
4. The tinyyaml Lua package

Please, let me know how that goes!

If you manage to compile the example document, I will use your feedback to hopefully have the CI running the LMTX tests and compiling the LMTX examples in the next version, to be released by the end of March at the latest. 🤞

[Witiko]

https://github.com/latex3/lua-uni-algos has support for all the normalization algorithms. I've used that once before and everything worked as expected for me.

Incidentally, I am currently in the process in removing the dependency on lua-uni-algos in PR #569 and rewriting all Unicode normalization algorithms in terms of PEG parsers in the file markdown-unicode-data.lua.

[josephwright]

I will though look at what I can do with the data - the major gap seems to be NFD.

https://github.com/latex3/lua-uni-algos has support for all the normalization algorithms. I've used that once before and everything worked as expected for me.

Sure, but that's starting from 'we are working in Lua'. For expl3, as far as possible we want to use the same code paths for all engines - life is otherwise more 'interesting'. So changing the approach for one engine where we don't need to for our main target (LaTeX) is a tricky sell. Hence wanting just the data or as close to that as possible - that means most of the code stays the same.

The NFD part is relatively small - so if push comes to shove I could find a way to pre-digest, or most likely parsing in Lua just for this would work.

[josephwright]

@gucci-on-fleek

Sure, but if expl3 is so slow to load that no one uses it, then effectively 0% is being used. In that case, a 90% complete version that's quick enough for people to actually use seems like a decent trade-off.

I'd say that really the time hit here is pretty small - most realistic documents for me take a lot longer to compile than the start-up time here. That said, I also do aim to support ConTeXt as best I can (despite the fact that in user terms it is smaller than LaTeX) - so I will look again at this, likely after a couple of meetings at the start of September. I am sure we can find an accommodation - even if I can't make things as fast as LaTeX (where we don't have to worry about reading time as that's all mopped up in the format-building process).

[josephwright]

I've put 'Unicode data' on the agenda for today's LaTeX team meeting - I'll report back on ideas.

[Witiko]

In the meantime, I will go ahead with including expl3-generic in the ConTeXt format from our Docker images, as described by @gucci-on-fleek in their helpful post,

Well that's a rather horrible hack, so it's probably not the best idea. But it should be fairly safe inside a dedicated Markdown image, since any one using that for sure wants to use Markdown, and any possible breakage is limited to the container image.

I adapted your hack for ConTeXt MkIV, since we want fast format loading for both LuaTeX (context --luatex) and LuaMetaTeX (context). In fact, the former is currently more important, since we officially support MkIV but will only support LMTX after merging this PR.

In commit 18d8b8c, I added the adapted code to our Dockerfile. This change increased the size of formats/luatex/cont-en.fmt from 12M to 14M (+17%), so I can see why the ConTeXt devs would be wary but I expected way worse.

However, the sad news is that it currently doesn't seem to work. Let me demonstrate:

witiko@witiko-G5-5590:~$ docker run --rm -it texlive/texlive:latest@sha256:9080025bd1762766c229ed7543107aa7799a82fa65b0643c4191305677863859
root@f69c25f40692:/workdir# mkdir -p /usr/local/texlive/texmf-local/tex/context/third/
root@f69c25f40692:/workdir# cat > /usr/local/texlive/texmf-local/tex/context/third/expl3.mkiv <<-'EOF'
> \directlua{function pdf.getcreationdate() end}  % expl3 bug
> \usemodule[expl3-generic]
> EOF
root@f69c25f40692:/workdir# sed -i '1,/^\\setupcurrentlanguage\[/s/^\\setupcurrentlanguage\[/\\loadmarkfile{expl3}\n&/' /usr/local/texlive/2025/texmf-dist/tex/context/base/mkiv/context.mkiv
root@f69c25f40692:/workdir# TEXMFHOME='{/usr/local/texlive/texmf-local, /usr/local/texlive/2025/texmf-dist/tex/latex-dev/}' context --make --luatex
root@f69c25f40692:/workdir# cat > example.tex <<-'EOF'
> \usemodule[expl3-generic]
> \ExplSyntaxOn
> \str_uppercase:n
>   { foo }
> EOF
root@f69c25f40692:/workdir# context --luatex --once example.tex

tex error       > tex error on line 4 in file /workdir/example.tex: ! Use of \??? doesn't match its definition

<argument> \???  
                 ! LaTeX Error: Access to an entry beyond an array's bounds.
l.4   { foo }
             

1     \usemodule[expl3-generic]
2     \ExplSyntaxOn
3     \str_uppercase:n
4 >>    { foo }
5     

mtx-context     | fatal error: return code: 256

The processing is snappy but including expl3 in the format seems to break some parts of it like \str_uppercase:n.

[gucci-on-fleek]

@Witiko

However, the sad news is that it currently doesn't seem to work. Let me demonstrate:

witiko@witiko-G5-5590:~$ docker run --rm -it texlive/texlive:latest@sha256:9080025bd1762766c229ed7543107aa7799a82fa65b0643c4191305677863859
root@f69c25f40692:/workdir# mkdir -p /usr/local/texlive/texmf-local/tex/context/third/
root@f69c25f40692:/workdir# cat > /usr/local/texlive/texmf-local/tex/context/third/expl3.mkiv <<-'EOF'
> \directlua{function pdf.getcreationdate() end}  % expl3 bug
> \usemodule[expl3-generic]
> EOF
root@f69c25f40692:/workdir# sed -i '1,/^\\setupcurrentlanguage\[/s/^\\setupcurrentlanguage\[/\\loadmarkfile{expl3}\n&/' /usr/local/texlive/2025/texmf-dist/tex/context/base/mkiv/context.mkiv
root@f69c25f40692:/workdir# TEXMFHOME='{/usr/local/texlive/texmf-local, /usr/local/texlive/2025/texmf-dist/tex/latex-dev/}' context --make --luatex
root@f69c25f40692:/workdir# cat > example.tex <<-'EOF'
> \usemodule[expl3-generic]
> \ExplSyntaxOn
> \str_uppercase:n
>   { foo }
> EOF
root@f69c25f40692:/workdir# context --luatex --once example.tex

tex error       > tex error on line 4 in file /workdir/example.tex: ! Use of \??? doesn't match its definition

<argument> \???  
                 ! LaTeX Error: Access to an entry beyond an array's bounds.
l.4   { foo }
             

1     \usemodule[expl3-generic]
2     \ExplSyntaxOn
3     \str_uppercase:n
4 >>    { foo }
5     

mtx-context     | fatal error: return code: 256

The processing is snappy but including expl3 in the format seems to break some parts of it like \str_uppercase:n.

Hmm, probably has something to do with the fact that you can't dump Lua state into a format. Looks like this is specifically called out in the expl3 documentation as a potential issue with ConTeXt:

https://github.com/latex3/latex3/blob/1f546c4c9ab6201ca239deca2582f180a1341557/l3kernel/l3luatex.dtx#L669-L672

You should ““just”” need to provide a suitable definition for register_luadata, although I don't immediately know how to do so.

[Witiko]

Thanks for the directions, @gucci-on-fleek!

You should ““just”” need to provide a suitable definition for register_luadata, although I don't immediately know how to do so.

Besides the format file cont-en.fmt, I also see files cont-en.lui and cont-en.luv in the directory /usr/local/texlive/2025/texmf-var/luatex-cache/context/a86c089b384a3076dc514ba966a1fac9/formats/luatex/. These contain Lua state, they seem produced by the file context.mkiv during format initialization, and they are later loaded together with the format file. It seems to me like the same mechanism could be used to dump and later recover expl3's Lua state.

We might be able to achieve this without updating LaTeX3 by extending the hack from commit 18d8b8c with a file expl3.lua that would create a minimal facade for luatexbase before loading expl3.mkiv. However, I am not knowledgable enough about the internals of either LaTeX3 or ConTeXt to do this without setting aside an afternoon just to explore the code and try stuff. Therefore, I don't expect to be able to do this myself before September.

[Witiko]

Incidentally, I am currently in the process in removing the dependency on lua-uni-algos in PR #569 and rewriting all Unicode normalization algorithms in terms of PEG parsers in the file markdown-unicode-data.lua.

After ~4 months, the PR #569 is done and was released earlier today as a part of the Markdown package v3.11.5.

---

The minimal set of remaining tasks for this PR is just the following, as outlined in #557 (comment):

1. Deal with LuaMetaTeX's lack of support for KPathSea in the example file for ConTeXt LMTX from this PR.

This seems easy enough to finish in time for the next release v3.11.6, planned for September.

---

To close #436, the following tasks also need to be finished:

2. Make expl3 part of the ConTeXt MkIV and LMTX format files, as discussed in #557 (comment) and below.
3. Add ConTeXt LMTX to the test suite. (Finishing this point before 2 would likely cause our CI to start timing out.)

If we can figure out how to store expl3's Lua state in a ConTeXt format, as discussed in #557 (comment), then we should also be able to include this change in v3.11.6.

---

To close #402, the following tasks also need to be finished:

4. Write installation instructions for ConTeXt Standalone to the user manual, including making expl3 part of format files.

This is just some copy-editing. We should be able to close #402 together with #436.

[gucci-on-fleek]

@Witiko

The minimal set of remaining tasks for this PR is just the following, as outlined in #557 (comment):

1. Deal with LuaMetaTeX's lack of support for KPathSea in the example file for ConTeXt LMTX from this PR.

resolvers.findfile should mostly be a drop-in replacement; §11.5 of the CLD manual documents how it works.

To close #402, the following tasks also need to be finished:

4. Write installation instructions for ConTeXt Standalone to the user manual, […]

If you upload the Markdown package (and expl3) to https://modules.contextgarden.net/, you can just point people to this Wiki page.

Otherwise, the instructions are:

1. Unpack the .tds.zip file in one of the folders listed by mtxrun --resolve-path TEXMF. $TEXMFMODULES, $TEXMFLOCAL, or $TEXMFHOME are likely the best options.
2. Run mtxrun --generate (and optionally mtxrun --luatex --generate).
3. That should be it.

[…] including making expl3 part of format files.

I don't think that it's a good idea to recommend to users that they should build expl3 into their ConTeXt formats. Inside the Docker images, that's fairly safe to do so, since (1) anyone using the markdown images cares about Markdown first and ConTeXt second, so a faster Markdown runtime is more important than a stable ConTeXt installation; and (2) any changes made inside the Docker image only affect the Docker image, so there's no risk of breaking the rest of the system.

But outside of a container, things are different because (1) most ConTeXt users probably care more about their entire ConTeXt installation than the Markdown package, and (2) if something breaks, the entire ConTeXt installation could get broken, and they can't just pull an older container image to fix it.

And by the way, thank you for the sponsorship!

[Witiko]

resolvers.findfile should mostly be a drop-in replacement [for KPathSea]; §11.5 of the CLD manual documents how it works.

Thanks, that's useful!

If you upload the Markdown package (and expl3) to https://modules.contextgarden.net/, you can just point people to this Wiki page.

I am not sure how reasonable it would be to distribute the Markdown package together with expl3. I'd be more in favor of keeping it as an external dependency that ConTeXt Standalone users have to install manually.

But outside of a container, things are different because (1) most ConTeXt users probably care more about their entire ConTeXt installation than the Markdown package, and (2) if something breaks, the entire ConTeXt installation could get broken, and they can't just pull an older container image to fix it.

I'd put that as a caveat in the instructions and let people decide for themselves.

[Witiko]

And by the way, thank you for the sponsorship!

I appreciate your and @josephwright's help and I am happy to buy you a coffee for your trouble!

[josephwright]

@Witiko We have a plan in hand to speed up loading expl3 in LuaTeX - watch this space!

[josephwright]

Meanwhile: latex3/latex3#1791

[Witiko]

@josephwright: Thanks for the update! It sounds like ConTeXt users will be getting a nice speed boost soon, even without bundling expl3 into the ConTeXt formats. I still plan to look into that later this month, but it feels much less critical now.

Am I reading PR latex3/latex3#1791 correctly that ConTeXt Standalone users will need to install not only expl3 but also lua-uni-algos? That's a bit ironic, given that we dropped the lua-uni-algos dependency in #569 in the previous release of the Markdown package. Of course, the work from #569 is still useful for people who only use the Lua side of the Markdown package and not expl3: plain TeX and OpTeX users, for instance.

[josephwright]

Am I reading PR latex3/latex3#1791 correctly that ConTeXt Standalone users will need to install not only expl3 but also lua-uni-algos? That's a bit ironic, given that we dropped the lua-uni-algos dependency in #569 in the previous release of the Markdown package. Of course, the work from #569 is still useful for people who only use the Lua side of the Markdown package and not expl3: plain TeX and OpTeX users, for instance.

Yes - we need to put the code somewhere and really this is Lua generic support not limited to expl3, so it makes sense.

[Witiko]

It sounds like ConTeXt users will be getting a nice speed boost soon, even without bundling expl3 into the ConTeXt formats.

After the l3kernel update, I am seeing a >3× speed-up in our tests, as discussed in #584. This is very encouraging!

[Witiko]

To close #436, the following tasks also need to be finished: [...]

3. Add ConTeXt LMTX to the test suite. [...]

While trying to get the test suite to work with ConTeXt LMTX, I discovered another issue with LMTX support in expl3, which I reported in latex3/latex3#1797.

[Witiko]

The minimal set of remaining tasks for this PR is just the following, as outlined in #557 (comment):

1. Deal with LuaMetaTeX's lack of support for KPathSea in the example file for ConTeXt LMTX from this PR.

resolvers.findfile should mostly be a drop-in replacement; §11.5 of the CLD manual documents how it works.

The file context-lmtx.tex typesets OK with command make -C examples context-lmtx.tex after some changes.

diff --git a/examples/context.tex b/examples/context.tex
index 34b6b350..6bdabe94 100644
--- a/examples/context.tex
+++ b/examples/context.tex
@@ -4,10 +4,6 @@
 \setupexternalfigures[location={local,global,default}]
 
 % Load the Markdown module.
-\startluacode
-local kpse = require("kpse")
-kpse.set_program_name("luatex")
-\stopluacode
 \usemodule[t][markdown]
 
 % Set options of the Markdown module.
diff --git a/markdown-parser.lua b/usr/local/texlive/texmf-local/tex/luatex/markdown/markdown-parser.lua
index 8211451e..52f5d485 100644
--- a/markdown-parser.lua
+++ b/markdown-parser.lua
@@ -72,14 +72,6 @@ if not modules then modules = { } end
 modules['markdown'] = metadata
 local lpeg = require("lpeg")
 local md5 = require("md5")
-;(function()  -- luacheck: ignore
-  local should_initialize = package.loaded.kpse == nil
-                       or tex.initialize ~= nil
-  kpse = require("kpse")
-  if should_initialize then
-    kpse.set_program_name("luatex")
-  end
-end)()
 local M = {metadata = metadata}
 local walkable_syntax = {
   Block = {
@@ -7217,8 +7209,7 @@ end
 M.extensions.content_blocks = function(language_map)
   local languages_json = (function()
     local base, prev, curr
-    for _, pathname in ipairs{kpse.lookup(language_map,
-                                          {all=true})} do
+    for _, pathname in ipairs(resolvers.findfiles(language_map)) do
       local file = io.open(pathname, "r")
       if not file then goto continue end
       local input = assert(file:read("*a"))
@@ -9165,7 +9156,7 @@ function M.new(options)
   end
   for _, user_extension_filename in ipairs(options.extensions) do
     local user_extension = (function(filename)
-      local pathname = assert(kpse.find_file(filename),
+      local pathname = assert(resolvers.findfile(filename),
         [[Could not locate user-defined syntax extension "]]
         .. filename)
       local input_file = assert(io.open(pathname, "r"),

Here is the typeset result: context-lmtx.pdf.

The changes to the file examples/context.tex can be used as described. A proper fix of the files markdown*.lua should a) include a pcall() around all occurrences of require("kpse"), b) add functions util.findfile() and util.findfiles() that would use either kpse or resolvers, whichever is available, and c) document the dependency on resolvers and add a bibliographic reference to Section 11.5 (Resolvers) of the file cld-mkiv.pdf.