[Maven4] Migrate from Maven Archiver to standard jar tool

[desruisseaux]

This is a major refactoring of the Maven JAR Plugin for using the standard jar tool instead of Maven Archiver. The standard tool is available through the java.util.spi.ToolProvider interface, which was introduced in Java 9. Since Maven 4 upgraded its requirement from Java 8 to Java 17, the use of that interface is no longer problematic.

Rational

In early Java days, the jar tool was equivalent to a zip command with a different syntax and a little bit of special processing for the META-INF/MANIFEST.MF file. Because of this quasi-equivalence, it was not difficult to create JAR files ourselves using any library capable to write ZIP files. This is what Maven Archiver does, together with supporting other archive formats. But today, the jar tool became more sophisticated. It now includes options for verifying the consistency of multi-release JAR files, options for updating module-info.class, provides security features specific to Java, etc.. This evolution can be seen in the "Modular JAR files" section of the Maven Plugin documentation, which states that the plugin uses the jar tool for updating the JAR file in a way that the Maven Archiver can't do easily. Therefore, since Maven 4 requires Java 17 and since that Java version gives us an easy access to the jar tool through the java.util.spi.ToolProvider interface, it may be time to abandon our manual creation of a JAR file and rely fully on the jar tool instead.

Benefits

Safer multi-release JAR files

In a multi-release JAR file, blindingly storing the content of META-INF/versions/ directories as if they were ordinary resources is not equivalent to using the jar --release option. The difference is that in the latter case, the jar tool performs some consistency checks. This issue was independently reported by user in #484, which is fixed by this pull request. Note that if this verification is not desired, Maven users can disable it by setting the detectMultiReleaseJar plugin option to false.

Main class managed by the jar tool

In a modular JAR file, it is no longer sufficient to declare the main class in the Main-Class entry of the MANIFEST.MF file. The main class needs to be specified by the --main-class option of the jar tool, which will update module-info.class. For compatibility reason the Maven JAR plugin gets the option value from that manifest entry, but internally the use of the jar tool is mandatory.

Options not yet supported by the plugin

This approach allows to add a <jarArgs> configuration option, similar to the <compilerArgs> in the Maven Compiler Plugin. Such option would allow developers to use new tool arguments before they are supported by the plugin.

Examples of options not yet supported by the plugin are --module-version and --hash-modules. Explicit support for those options could be added in a future plugin version, especially since security is becoming more and more a concern.

Easier debugging and toolchain

This approach makes easy to generate a target/jar.args file when the build fails or when Maven is run in verbose mode. This is similar to the compiler plugin generating a target/javac.args file. This file allows the user to test easily on the command-line, which makes debugging faster. Likewise, the options can also be passed to another tool, which makes easier to resolve #439 as well.

Behavioral changes

The plugin behaviour after the proposed refactoring is different than version 3 in the following aspects:

Removal of default **/package.html excludes

The current plugin version uses an undocumented **/package.html default excludes. This default seems to exist since the initial revision in March 2004, but I saw no explanation for this oddity. This default is not mentioned in the documentation. The removal of this oddity is necessary for allowing the proposed new plugin implementation to specify only some root directories to the jar tool, since the tool can traverse the directory tree itself.

Automatic use of MANIFEST.MF

After this refactor, the plugin automatically uses the META-INF/MANIFEST.MF file found in the classes directory. Before this refactor, the plugin used that file only if explicitly specified in the <manifestFile> archive configuration. The previous policy was discussed in #255. The new policy is a natural consequence of the way that the JAR plugin is reimplemented, and also more useful in the context of multi-module (in Java module sense) projects since each module could contain its own MANIFEST.MF file.

Multi-module projects

The refactoral support multi-module projects, including projects that are both multi-module and multi-release, as discussed in apache/maven-compiler-plugin#998.

Dependencies

This pull request depends on a Maven release which include the following pull requests:

• Fix field accessibility leak in EnhancedCompositeBeanHelper maven#11425
• Accept Java module names as attached artifactId even if they differ from the project's artifactId maven#11549
• Avoid confusion with META-INF/versions when the sub-directories are multi-module maven-compiler-plugin#998

[ascheman]

Thanks for this migration, @desruisseaux — moving JAR creation to the jar ToolProvider with module-source-hierarchy + multi-release support (and the derived per-module POMs) is a big, elegant step.

Reviewing it, I dug into the CI failures, and they share a root theme: the multi-release/module handling in FileCollector/Archive depends on the unspecified Files.walkFileTree directory-iteration order and on jar-tool-vendor tolerances. So it passes on macOS/Temurin and fails on Linux/Zulu — i.e. on CI. I found six distinct manifestations and prepared an atomic fix + tests for each.

They're on aschemaven:feat/from-archiver-to-jar-tool (off your 5886fef) — compare. The full matrix (ubuntu/macos/windows × JDK 17/21 × rc-6) is green with all six, plus deterministic unit tests (ArchiveTest) and a records-jar-validate IT. Each is its own commit, so you can cherry-pick à la carte:

1. jar --validate crashes on records (JDK 17/18) — ToolExecutor runs --validate after create; the JDK 17/18 jar tool throws This feature requires ASM8 on any record (JDK-8282446, fixed in 19). Guard: skip validate when Runtime.version().feature() < 19. — 27cce4a
2. Base-release dir → NPE — the base FileSet is seeded from whichever dir first creates the Archive; on some walk orders that's a versions-modular/<n>/<module> dir (no module-info), so ModuleFinder finds nothing and PomDerivation NPEs. Rebind the base to the version-less dir. — c8a3f66
3. Main-Class on the wrong module — setMainClass removes Main-Class from a manifest shared across modules, so the first module processed consumes it. Per-module manifest copy. — 18581f7
4. Absolute jar entry names — FileSet.add relativizes only the first file; later files stay absolute and the jar tool records absolute .class entries ("names do not match"). Relativize all + repeat -C per file. — 63fe387
5. Empty -C "" for version releases — a versions/<n> dir is added to its own FileSet → relativize(dir,dir)=""; Zulu rejects -C dir "", Temurin tolerates it. Emit .. — e71b7ad
6. Base files leak into a version FileSet — a whole version dir is SKIP_SUBTREE'd, so postVisitDirectory (which resets to the base release) never runs; base files walked afterward land in the version FileSet as ../../../<pkg>. Reset inline. — 0ee85a8

Because these are order/vendor-dependent they only surface on CI (Linux+Zulu); the ArchiveTest unit tests pin the behaviour deterministically so it can't silently regress on any walk order. Happy to open a PR against your branch if that's easier — just say the word. Thanks again, this is really nice work.

[desruisseaux]

Thanks! I cherry-picked the commits with modifications:

1. Rearranged the if and else blocks for avoiding duplicated tests and calls of Runtime.version().feature(). Omitted the part of the log message saying that the JAR was created successfully on JDK 17/18 for consistency with the JDK 19+ branch which emitted no message. Edited the comments and commit message with minor clarifications (the validation issue was not only in the context of module source hierarchy projects) and omission of details about JDK internal in commit (kept in commit message), replaced by links to JDK issue tracker.
2. The commit repairs the problem after it happened (directory handled as the base version when actually it was another version). I think that the root cause of the problem is that the Archive constructor had no version argument. I tried a correction of that root cause instead. We will see with GitHub action if it works.
3. Changes only in documentation and minor adjustments of the tests.
4. Repeating the -C option with the path to the directory in front of every files to include in the JAR file seems a little bit extreme. The report generated by Claude does not explain why the iteration order is an issue. My hypothesis is that the first path must be the shortest one, so that next paths have a common prefix. I'm trying that and we will see what are the GitHub actions result.
5. Same commit but adapted for the changes done in point 4 above.
6. The root cause of the issue can be made clearer, but refactoring in a separated method the part of the postVisitDirectory(…) method that needs to be invoked when a directory is skipped.
7. This integration test for step 1 has been squashed with commit 1.

[rmannibucau]

a post-generation step where the JAR file is read and rewritten with directory entries added using java.util.zip

if this true? if so let's drop toolprovider and get back maven archiver, this is cleaner than a workaround which can be insanelt slower (twice at least) on big jars and try to push upstream the needed features, there is no point to move to a new tool which is twice worse for only a validation most people will not need

[desruisseaux]

if this true?

Not any more. When I wrote this comment, I was not aware of JDK-8276764. The proposal to apply a post-processing step is obsolete now.

[hboutemy]

I just tested building a few simple projects with misc JDKs and this PR:

mvn -V -e -Dversion.maven-jar-plugin=4.0.0-beta-2-SNAPSHOT -DskipTests clean package artifact:3.6.1:describe-build-output ; unzip -p target/*-SNAPSHOT.jar META-INF/MANIFEST.MF

first, I can see the the META-INF/MANIFEST.MF file remains as usual: great

from my test, with JDK 19+, result is locally reproducible: I always get the same output on the same laptop
I did not yet try to compare builds on different machines, I'll check later

with JDK 17 and 18, not reproducible: no problem with order of files, but with timestamp
and also a timestamp is injected in pom.properties

To me, avoid the issue with pom.properties should be easy
and postprocessing jar file to set timestamp should not be hard: like moditect for example https://github.com/moditect/moditect/blob/41784cb65f9a564012212ba94307ad98455d789d/core/src/main/java/org/moditect/commands/AddModuleInfo.java#L115

[laeubi]

So the same sources produce different manifest bytes on different JDK distributions

I wanted to note here that (binary) reproducible builds are not guaranteeing anyways between different JDK versions and vendors as they could provide different byte codes.

Apart from that, supplying the manifest directly would be the safest choice (also for the future) and thats why I think generation of files should be independent from packaging a jar:

• [idea] split the jar goal into two mojos that can be bound to different phases #586

Apart from that, having the validation of MR-jars baked into the jar tool is from my opinion a design flaw and actually what we aim here if I understand correctly, so if the JarOutputStream would have this support this would not only benefit maven but also others as this verification can then be reused see for reference here:

• MR compilation should validate the contract for MR types eclipse-jdt/eclipse.jdt.core#4274

so if anyone is in contact with the JDK team it would be great to bring this to their attention as it would avoid people doing duplicate work an/or workarounds where a programmatic solution would be superior.

[hboutemy]

continuing my base test (no MR or modular build yet, just basic classic jars)

rebuilding on a Mac in parallel of a Linux box with JDK 21 gave me same output: great, we have the output stability that I expect and got until now

So definitively, the JDK 17 and 18 reproducibility issue is what has to be solved for now

We'll see in a second step about modular or MR jars: if someone can point me to a simple project that I can build mself easily, that would be nice, please

one issue I get is twice attach:

[INFO] --- jar:4.0.0-beta-2-SNAPSHOT:jar (default-jar) @ maven-dependency-analyzer ---
[INFO] Building JAR: "target/maven-dependency-analyzer-1.17.2-SNAPSHOT.jar".
[WARNING] artifact 'org.apache.maven.shared:maven-dependency-analyzer:pom:consumer:1.17.2-SNAPSHOT' already attached, replacing previous instance

that is also visible in the artifact:3.6.1:describe-build-output too: and the message that says "replacing" is apparently not replacing but adding another attach
but perhaps it's more a Maven 4.0..0-rc-6 bug: if I drop the artifact:3.6.1:describe-build-output goal to the mvn command, this issue disappears
I'll open a Maven issue for that

[desruisseaux]

I wanted to note here that (binary) reproducible builds are not guaranteeing anyways between different JDK versions and vendors as they could provide different byte codes.

I agree, and this is one reason why I disagree with the importance given to strict (as opposed to semantically) reproducible builds. I think that it can even deserve the real goal, which is security. If some Java tools produce vulnerable code (e.g., Javadoc HTML frame injection vulnerability in 2013), it is difficult to know if a JAR file has been produced by patched tools if we don't have a useful Created-By entry in the MANIFEST.MF file. But we are throwing away this information in the name of strict reproducible builds.

Regarding the split of metadata generation (MANIFEST.MF and Maven files) and packaging in two different phases, I agree with that. But I propose to go step by step:

1. Fix the issue raised in above comments (missing directory entries, Created-By attribute).
2. Get this pull request merged.
3. In a separated pull request, copy (in modified form) in this plugin the Maven Archiver code which is still used. This is the code that generates the MANIFEST.MF attributes.
4. Refactor in two separated goals.

Regarding the JAR file validation, this is one of the reasons for this migration to the jar tool but not the only one. There is also additional options such as --hash-modules. The support of this option is not included in this pull request because it will be a new feature, but it would be the topic of a future pull request.

[desruisseaux]

postprocessing jar file to set timestamp should not be hard

But is is worth the effort? It would be a hack needed for JDK 17 and 18 only. And since the file ordering issue (JDK-8276764) was fixed in JDK 18 only, even if we fix the timestamp the build would still non-reproducible on JDK 17. Therefore, setting the timestamp would be useful for only one specific JDK version: 18.

[jaikiran]

Hello Martin,

And since the file ordering issue (JDK-8276764) was fixed in JDK 18 only, even if we fix the timestamp the build would still non-reproducible on JDK 17.

Bug fixes/enhancements in the JDK are sometimes backported. Those backports are tracked/linked against the original issue. For this specific issue https://bugs.openjdk.org/browse/JDK-8276764 , under the issue description, there's a backports section which shows that this enhancement was backported to JDK 17 (the Resolved/Fixed status on those backports is a sign that the work is complete).

Having said that, my comment is mostly informational and isn't meant as an input to whether or not some post processing should be done for the JAR files.

[seregamorph]

IMO it's critical to support binary reproducibility for all JDKs since minimal 17, otherwise it'll be a regression.

[desruisseaux]

Thanks @jaikiran for the clarification. Then indeed, editing timestamps in a post-processing would work on both JDK 17 and 18.

Whether we should do this post-processing is still uncertain to me. Not doing this post-processing would be a regression only for users wanting binary reproducible builds with the tools of JDK 17 or 18. Users who do not upgrade to a more recent Java version at least for building (even if they target a lower release with --release) may be conservative regarding their Maven upgrade too, in which case the 3.x plugins continue to work as today.

[rmannibucau]

IMO it's critical to support binary reproducibility for all JDKs since minimal 17, otherwise it'll be a regression.

stating that for maven 4 java 17 bytecode is supported using a java 21 or more (to speak "LTS") doesn't sound crazy to me
also agree it is reproducible only for the same JVM baseline (not even minor matching) so on my side it stays not critical since covered technically if well documented

like moditect for example

ZipFileSystem#sync can be worth benching on a big jar (some thousands of classes), ideally it is very small compared to producing the zip (jar) itself but worth evaluating before thinking this is an option IMHO - since we know we can always do it right directly if needed and JDK version is a blocker (once again not for me)

[gnodet]

Review: [Maven4] Migrate from Maven Archiver to standard jar tool

Thank you for this substantial and well-motivated refactoring, @desruisseaux! The migration to the JDK's built-in jar tool via ToolProvider brings genuine technical benefits — safer multi-release JAR validation (#484), proper module hierarchy support, excellent debuggability with target/jar.args, and a foundation to leverage future jar tool improvements.

The overall code quality is high, with thoughtful Javadoc, defensive programming, and careful attention to ordering-sensitive edge cases (the ArchiveTest tests for iteration-order independence are excellent).

Highlights

• jar --release validation catches multi-release API incompatibilities that the old Plexus approach missed entirely
• target/jar.args debug file with platform-aware CLI tips is a first-class debugging feature
• Copy-on-write manifest handling correctly prevents the shared manifest from being consumed by the first module
• Strong integration test coverage for new scenarios (multi-module, multirelease-with-modules, records-jar-validate)

Summary of Findings

Severity	Count	Key Items
Blocking	1	validate() dead-code condition (B1)
Important	5	NPE risk in PomDerivation (I1), resource leak in MetadataFiles.close() (I2), cross-device link failure (I3), NumberFormatException on "-" (I4), silent reproducible build degradation (I5)
Suggestion	6	Unit test gaps for FileCollector/ToolExecutor (S1-S2), visitFileFailed() override (S3), dependency cleanup (S4-S5), missing IT for <jarArgs> (S6)

Upstream Dependencies Note

The PR depends on unmerged PRs (maven#11425, maven#11549, maven-compiler-plugin#998). This is understood from the PR description and not a code issue per se, but the integration timeline should be tracked.

---

This review was generated by an AI agent and may contain inaccuracies. Please verify all suggestions before applying.

[desruisseaux]

@seregamorph, @hboutemy: I'm almost done in fixing the issue raised last week, with the help of @ascheman commits (thanks!). I have made one change which I suspect will be highly controversial. If strictly reproducible builds has been requested, the JDK version in the Created-By attribute is replaced by "Maven JAR plugin on unspecified JDK". The "on unspecified JDK" is my addition. The intend is to remind users that strictly reproducible builds is causing harm here, and encouraging them to switch to semantically reproducible builds if Maven provides a way to run this verification easily. However, I'm not strongly attached to this addition. If there are objections, I will remove it.

[elharo]

I would remove "on unspecified JDK". I might add the actual JDK version since reproducible builds require the same toolchain for now.

For the record, I do not think "semantically reproducible builds" are a thing. Byte-per-byte comparison is the only thing that works.

[desruisseaux]

I might add the actual JDK version since reproducible builds require the same toolchain for now.

Are we requiring toolchain? I thought that we were betting on different JDKs producing the same bytecode when used with the same --release version (which seems a fragile assumption).

For the record, I do not think "semantically reproducible builds" are a thing. Byte-per-byte comparison is the only thing that works.

For the security goal, I have not yet seen a technical argument explaining why semantically reproducible builds would not work. On the contrary, I think that it is more secure than strictly reproducible builds for the reason given above.

In addition of security goal, it seems that there is also a caching goal. It has been argued that Docker images use hash-code for caching JAR files. This is a more difficult case and may be a reason to keep strictly reproducible builds as an option, but not necessarily a reason for makes it the default if semantically reproducible builds can be guaranteed.

[desruisseaux]

Ok, I removed the "on unspecified JDK" part. I think that the risk of controversy is almost certain.

[elharo]

A different compiler version can easily choose to perform different optimizations; e.g. to unroll a loop or not, while remaining fully spec conformant. This produces different byte code, even when compiling for the same Java version. I'm not sure whether determining that two different versions of the byte code for a method produce the same output for the same inputs is equivalent to solving the halting problem, but it's certainly non-trivial and not something I want to rely on for reproducible builds.

[desruisseaux]

A different compiler version can easily choose to perform different optimizations; e.g. to unroll a loop or not, while remaining fully spec conformant.

Yes, this is exactly why I think that strictly reproducible builds currently works only by chance. Doing semantically reproducible builds at the bytecode level would be challenging, but if we start with only the easy part (ignoring timestamp, compression and a few selected metadata such as Created-By), we have something working for the current state of the industry and we prepare the ground for more advanced comparisons in the future if the above-cited differences in bytecode are observed.