perf: get unpack() out of the NYI list

[MyNameIsTrez]

While writing a transpiler to Lua I hit a 20x performance cliff caused by unpack. I wrote up the full investigation here.

unpack is listed as 2.1 stitch on tarantool's LuaJIT Not Yet Implemented page, which explains the cliff. I confirmed this with -jv.

This PR resolves the cliff by making unpack fully compiled.

The original performance cliff and my workaround

Running the MRE below against the default branch (v2.1-agentzh) 20 times produces only fast runs (~0.025s) or slow runs (~0.3s), never anything in between.

The Dockerfile below reads mre.lua from the current working directory. Every Dockerfile in this PR description is run with docker build -t luajit2-test . && docker run --rm -it luajit2-test.

Fast runs are expected. Slow runs happen when LuaJIT blacklists empty_fn() in response to unpack NYIs. Pass -jv to luajit to see these sporadic blacklisted messages.

Dockerfile that runs mre.lua

FROM alpine:latest

# Install build dependencies
RUN apk add --no-cache git make gcc musl-dev

WORKDIR /workspace

# Clone the default branch and compile
RUN git clone https://github.com/openresty/luajit2.git && \
    cd luajit2 && \
    make -j$(nproc)

# Copy the MRE script from your host's current working directory
COPY mre.lua /workspace/mre.lua

WORKDIR /workspace/luajit2

# Run the benchmark 20 times
CMD ["sh", "-c", "for i in $(seq 20); do src/luajit ../mre.lua; done"]

mre.lua

-- empty_fn takes no args here because many game functions don't
-- take args either; this MRE is meant to mirror that.
-- Note: replacing () with (...) makes the benchmark report fast times,
-- since it avoids the NYI shown below.
local function empty_fn() end

local function run_unpack()
    pcall(empty_fn, unpack({}))
end

-- On my laptop (AMD Ryzen AI 9 HX 370), 80k iterations is the
-- nondeterministic tipping point between always fast (~70k) 
-- and always slow (~90k).
for _ = 1, 80000 do
    run_unpack()
end

local start = os.clock()

-- Because of the previous loop, this unrelated hot-loop is a coin toss
-- between being JIT compiled (fast) or stuck in the interpreter (slow).
for _ = 1, 100000000 do
    empty_fn()
end

print(os.clock() - start)

To work around this, I updated my transpiler to generate a specialized wrapper per argument count instead of forwarding arguments through unpack. Each wrapper indexes the args table directly and is cached, so the code generation cost is paid once while execution stays fully traceable by LuaJIT. This workaround will stay relevant for years, since many programs never update the LuaJIT version they embed.

Workaround mre.lua

local pcall_wrappers = {}

local function get_pcall_wrapper(arg_count)
    if pcall_wrappers[arg_count] then
        return pcall_wrappers[arg_count]
    end

    local arg_list = {}
    for i = 1, arg_count do
        arg_list[i] = "args[" .. i .. "]"
    end

    -- Generate a specialized wrapper to avoid `unpack` (which triggers a LuaJIT NYI).
    -- Example (arg_count=2): return function(fn, args) return pcall(fn, args[1], args[2]) end
    local args_str = #arg_list > 0 and (", " .. table.concat(arg_list, ", ")) or ""
    local code = string.format("return function(fn, args) return pcall(fn%s) end", args_str)

    local wrapper = loadstring(code)()
    pcall_wrappers[arg_count] = wrapper
    return wrapper
end

-- empty_fn takes no args here because many game functions don't
-- take args either; this MRE is meant to mirror that.
-- Note: replacing () with (...) makes the benchmark report fast times,
-- since it avoids the NYI shown below.
local function empty_fn() end

local function run_unpack()
    local args = {}
    local wrapper = get_pcall_wrapper(#args)
    wrapper(empty_fn, args)
end

-- On my laptop (AMD Ryzen AI 9 HX 370), 80k iterations is the
-- nondeterministic tipping point between always fast (~70k) 
-- and always slow (~90k).
for _ = 1, 80000 do
    run_unpack()
end

local start = os.clock()

-- Because of the previous loop, this unrelated hot-loop is a coin toss
-- between being JIT compiled (fast) or stuck in the interpreter (slow).
for _ = 1, 100000000 do
    empty_fn()
end

print(os.clock() - start)

Running the 32 tests I wrote for recff_unpack

Dockerfile that runs t/unpack.t its 32 recff_unpack tests

FROM alpine:latest

# Install build dependencies, perl, perl-utils (for prove), and cpanminus
RUN apk add --no-cache git make gcc musl-dev perl perl-utils perl-app-cpanminus

# Install Perl test dependencies
RUN cpanm --notest IPC::Run3 Test::Base Test::LongString Parallel::ForkManager

WORKDIR /luajit2

# Clone repository and checkout PR #269 into branch 'fix-unpack-nyi'
RUN git clone https://github.com/openresty/luajit2 . && \
    git fetch origin pull/269/head:fix-unpack-nyi && \
    git checkout fix-unpack-nyi

# Build, install to system paths, and run the test
CMD ["sh", "-c", "make -j$(nproc) && make install && prove t/unpack.t"]

It prints this:

t/unpack.t .. ok
All tests successful.
Files=1, Tests=96,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.16 cusr  0.10 csys =  0.29 CPU)
Result: PASS

Alternative: Running the tests without Perl or Docker

If you prefer not to use Perl, prove, or Docker, I wrote a quick Python script that compiles t/unpack.t into a standalone, zero-dependency Lua test harness. It runs the compiler, parses standard output and trace logs, and verifies all 32 edge cases automatically.

convert_tests.py

import re
import sys

def convert_regex_to_lua_pattern(perl_regex):
    """Converts a basic Perl qr// regex to a Lua pattern."""
    # Strip qr/ and /
    core = perl_regex[3:-1]
    # Convert escapes
    core = core.replace(r'\[', '%[')
    core = core.replace(r'\]', '%]')
    core = core.replace(r'\.', '%.')
    core = core.replace(r'\s+', '%s+')
    return core

def parse_and_convert(input_file, output_file):
    with open(input_file, 'r', encoding='utf-8') as f:
        content = f.read()

    # Split the file by test boundaries
    raw_tests = content.split('\n=== TEST ')
    
    lua_tests = []
    
    for raw_test in raw_tests[1:]:  # Skip the preamble
        # Extract the name (first line)
        name, _, body = raw_test.partition('\n')
        name = name.strip()
        
        # Extract the lua, out, and err blocks
        lua_match = re.search(r'--- lua\n(.*?)(?=\n--- jv|\n--- out|\n--- err)', body, re.DOTALL)
        out_match = re.search(r'--- out\n(.*?)(?=\n--- err|\n=== TEST|$)', body, re.DOTALL)
        err_match = re.search(r'--- err(?: eval)?\n(.*?)(?=\n=== TEST|$)', body, re.DOTALL)
        is_eval = '--- err eval' in body
        
        lua_code = lua_match.group(1).strip() if lua_match else ""
        out_code = out_match.group(1).strip() if out_match else ""
        err_code = err_match.group(1).strip() if err_match else ""
        
        if is_eval:
            err_code = convert_regex_to_lua_pattern(err_code)

        lua_tests.append({
            'name': name,
            'lua': lua_code,
            'out': out_code,
            'err': err_code,
            'is_eval': is_eval
        })

    # Generate the Lua test script
    with open(output_file, 'w', encoding='utf-8') as out_f:
        out_f.write("-- Auto-generated standalone test harness for unpack() NYI removal\n")
        out_f.write("local tests = {\n")
        
        for t in lua_tests:
            out_f.write("  {\n")
            out_f.write(f"    name = {repr(t['name'])},\n")
            out_f.write(f"    lua = [=[\n{t['lua']}\n]=],\n")
            out_f.write(f"    out = [=[{t['out']}]=],\n")
            out_f.write(f"    err = [=[{t['err']}]=],\n")
            out_f.write(f"    is_eval = {'true' if t['is_eval'] else 'false'}\n")
            out_f.write("  },\n")
            
        out_f.write("}\n\n")
        out_f.write(LUA_HARNESS)
        
    print(f"Successfully compiled {len(lua_tests)} tests to {output_file}.")

LUA_HARNESS = """
local luajit_cmd = arg[1] or "luajit"
local pass_count = 0
local fail_count = 0

local function trim(s)
    return (s:gsub("^%s*(.-)%s*$", "%1"))
end

print("Running " .. #tests .. " tests using: " .. luajit_cmd .. "\\n")

for i, test in ipairs(tests) do
    -- Write the test code to test.lua (traces expect this filename)
    local f = io.open("test.lua", "w")
    f:write(test.lua)
    f:close()
    
    -- Run LuaJIT with trace logging enabled
    os.execute(luajit_cmd .. " -jv test.lua > test.out 2> test.err")
    
    -- Read outputs
    local f_out = io.open("test.out", "r")
    local actual_out = f_out and f_out:read("*a") or ""
    if f_out then f_out:close() end
    
    local f_err = io.open("test.err", "r")
    local actual_err = f_err and f_err:read("*a") or ""
    if f_err then f_err:close() end
    
    actual_out = trim(actual_out)
    local expected_out = trim(test.out)
    local ok = true
    local reason = ""
    
    -- 1. Check standard output
    if actual_out ~= expected_out then
        ok = false
        reason = "STDOUT mismatch.\\nExpected: " .. expected_out .. "\\nGot: " .. actual_out
    end
    
    -- 2. Check trace logs (stderr)
    if ok and test.err ~= "" then
        if test.is_eval then
            -- Use Lua pattern matching for translated regex
            if not string.match(actual_err, test.err) then
                ok = false
                reason = "STDERR pattern mismatch.\\nExpected pattern: " .. test.err .. "\\nGot: " .. actual_err
            end
        else
            -- Plain text substring match for each expected line
            for expected_line in string.gmatch(test.err, "[^\\r\\n]+") do
                if not string.find(actual_err, expected_line, 1, true) then
                    ok = false
                    reason = "STDERR missing expected trace log.\\nMissing: " .. expected_line .. "\\nGot: " .. actual_err
                    break
                end
            end
        end
    end
    
    if ok then
        print("[PASS] " .. test.name)
        pass_count = pass_count + 1
    else
        print("[FAIL] " .. test.name)
        print("       " .. string.gsub(reason, "\\n", "\\n       "))
        fail_count = fail_count + 1
    end
end

-- Cleanup
os.remove("test.lua")
os.remove("test.out")
os.remove("test.err")

print("\\n---")
print("Passed: " .. pass_count)
print("Failed: " .. fail_count)

if fail_count > 0 then
    os.exit(1)
end
"""

if __name__ == '__main__':
    if len(sys.argv) != 3:
        print("Usage: python convert_tests.py <input.t> <output.lua>")
        sys.exit(1)
    parse_and_convert(sys.argv[1], sys.argv[2])

You can generate the test harness and run it against either the local build or an installed binary:

# Generate the pure Lua test harness
python convert_tests.py t/unpack.t test_unpack.lua

# Option A: Testing the uninstalled local build in the repository
# (LUA_PATH is required so it can find the local jit.* libraries)
LUA_PATH="src/?.lua;;" src/luajit test_unpack.lua src/luajit

# Option B: Testing if you already ran `make install`
luajit test_unpack.lua luajit

Confirming this fixed the original mre.lua

Dockerfile that checks out this PR's branch, to demonstate the original mre.lua now always runs fast

FROM alpine:latest

# Install build dependencies
RUN apk add --no-cache git make gcc musl-dev

WORKDIR /luajit2

# Clone repository and checkout PR #269 into branch 'fix-unpack-nyi'
RUN git clone https://github.com/openresty/luajit2 . && \
    git fetch origin pull/269/head:fix-unpack-nyi && \
    git checkout fix-unpack-nyi

# Compile LuaJIT
RUN make -j$(nproc)

# Copy the MRE script from your host's current working directory
COPY mre.lua ./

# Run the benchmark 20 times
CMD ["sh", "-c", "for i in $(seq 20); do src/luajit mre.lua; done"]

Running unimut to mutation test recff_unpack

Although I brought line and branch coverage to 100%, I couldn't be sure I was covering every edge case, or that there were no redundant sections I could cut.

To address this, I wrote unimut (universal mutator, pip install unimut) for this PR. It is called universal because it lets users register backends for other languages too:

unimut can be run like unimut --file src/lj_ffrecord.c --run 'make -j$(nproc) && prove t/unpack.t'. The Dockerfile below compiles with ASan and UBSan, which brings surviving mutants down from 11 to 9, and adds temporary // unimut on and // unimut off markers around the recff_unpack function this PR adds:

Dockerfile that mutation tests recff_unpack

FROM alpine:latest

# Install build dependencies, perl, perl-utils (for prove), cpanminus, and Python/pip
RUN apk add --no-cache git make gcc musl-dev perl perl-utils perl-app-cpanminus python3 py3-pip

# Install Perl test dependencies
RUN cpanm --notest IPC::Run3 Test::Base Test::LongString Parallel::ForkManager

# Install unimut globally
RUN pip install --break-system-packages unimut

WORKDIR /luajit2

# Clone repository and checkout PR #269 into branch 'fix-unpack-nyi'
RUN git clone https://github.com/openresty/luajit2 . && \
    git fetch origin pull/269/head:fix-unpack-nyi && \
    git checkout fix-unpack-nyi

# Inject unimut markers around the recff_unpack function
RUN perl -0777 -pi -e 's|(/\* unpack\(t, \[i, \[j\]\]\) \*/\nstatic void LJ_FASTCALL recff_unpack)|// unimut on\n$1|' src/lj_ffrecord.c && \
    perl -0777 -pi -e 's|(\nstatic void LJ_FASTCALL recff_tonumber)|\n// unimut off\n$1|' src/lj_ffrecord.c

# Run unimut with AddressSanitizer, UndefinedBehaviorSanitizer flags, and increased timeout
CMD unimut \
    --file src/lj_ffrecord.c \
    --jobs 16 \
    --timeout 120 \
    --run 'make -j$(nproc) PREFIX="$(pwd)/build" \
    TARGET_CFLAGS="-fsanitize=address,undefined -fno-sanitize=alignment,shift -fno-sanitize-recover=undefined -fno-omit-frame-pointer -ftrivial-auto-var-init=pattern -g -DLUAJIT_USE_SYSMALLOC -DLUA_USE_ASSERT -DLUA_USE_APICHECK" \
    TARGET_LDFLAGS="-fsanitize=address,undefined" \
    && make install PREFIX="$(pwd)/build" \
    && PATH="$(pwd)/build/bin:$PATH" prove -I. t/unpack.t'

It prints this concise diff

src/lj_ffrecord.c:376
- if (tref_isk(tri))

src/lj_ffrecord.c:377
- emitir(IRTGI(IR_EQ), tri, lj_ir_kint(J, i));
+ ;

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn == 0) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn < 0) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:397
- for (k = 0; k < n; k++) {
+ for (k = 0; k != n; k++) {

src/lj_ffrecord.c:397
- for (k = 0; k < n; k++) {
+ for (k = 0; k <= n; k++) {

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn <= (0 + 1)) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn <= (0 - 1)) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:397
- for (k = 0; k < n; k++) {
+ for (k = 0; k < (n + 1); k++) {

Survived: 9/146

The 9 surviving mutants are expected. They involve checks against internal LuaJIT implementation details that Lua-level tests can't, or shouldn't, cover.

The CI already fails on the base v2.1-agentzh branch

The Travis CI pipeline fails on the Valgrind job, but the latest commit on the base v2.1-agentzh branch fails with the exact same error. This PR doesn't introduce any new test failures.

[MyNameIsTrez]

Mike Pall merged this into luajit, so luajit2 just needs to synchronize.