Java compiling with reproducible jar

Software runs with binaries. The binaries are the primary objects of interest. If the software run, the user is satisfied.

That is until questions are asked:

Now the sources of the software are in focus. But the question is: Are the sources correct? If a new generation produces abbreviating binaries, the question of correctness cannot be answered with the new generation. The correctness is only a guess or assertion, errors in the saving may have occurred or environmental conditions may not have been fully recorded.

The proven binaries should be used furthermore. It is often not desirable to use new and untested binaries.

An adequate situation is, some bugs should be fixed, based on the given sources. If the given sources produce exact the same binary, than one can assume, that a change of a little detail in the sources changes only the expected behavior. Furthermore some report files (listings, maps) allow to check which details in the binary are changed.

But if the given sources do not produce the same binary, and it is not possible simply to check which changes are done in the binary, then the binary should be tested in all details. Nothing can be assumed to be safe. This is a high effort for time and money.

Another approach for reproducible build is the documentation:

Often the software is released in a less time, not sufficient time for a good documentation. Hence the documentation is in a state before software is ready, maybe with some fixes during the software development process, and the documentation is not the ideal case.

But after delivering the software there may be some time for documentation, before the next order is coming in. But: The software is compiled. If sources are changed in comments it are not the original sources.

If a reproducible build is possible and supported, the software can even be improved with regard to variable identifier, not only comments. The variable identifier may not change the binary code (without debugging informations, the delivery form). This is the challenge especially for embedded control.

This article contemplates only jar files for Java software usual running on PC. But such software is often use to produce C/++ code from graphical programming. Hence it is also in focus for devices and plants which are often programmed with C or C++.

The second approach: Some considerations made for jar files may also true for C/++-compilation.

The goal is: Regenerate a jar file with given sources, which are beside the jar in an software repository. Proofing the correctness of the sources should be able without additional effort. The least premise is only a javac compiler and a jar builder, no more. Any complex system (for example some additional gradle packages) for that approach are an additional effort with maybe newly challenges.

Of course, the reproducible of binaries for different platforms cannot be assigned to embedded software. But if the target platform is similar, or the binaries codes are not immediate machine codes, this principle can be reused.

In Java the binary in jar files are byte code, which determine the running sequences and data, but it is translated to real machine code from the Java Runtime Environment (JRE). The JRE is usage-invariant, but specific for the target platform. The program in Java-Bytecode is usage-relevant, but platform-invariant. That are the two characteristics of the Java technology for this approach.

On MS-Windows traditionally the character set is ISO-8859-1 for German etc. Linux is usual forward-locking, often UTF-8 as universal character set is used. The javac compiler can use any character set with the option

you can set a special encoding, here UTF-8. Without this option the javac uses the "systems encoding" what ever it is. Usage any system settings forces unnecessary differences. One should use always dedicated settings. Elsewhere the results depend on randomness. If the sources is well compiled, but another part is compiled on the PC of the counterpart, which uses for himself another character set by default, it is random.

Hence the encoding should be unconditionally determine by javac invocation. The encoding should be determine by the software development process because compilation with a determined encoding with random encodings in the sources may force problems. Because of most of texts are US-ASCII and a faulty encoding in comments is lesser important, this problem isn’t often in focus.

Another adequate problem, but not true for reproducibility is the usage of the line feed characters. It is simply unnecessary to differ between Windows (using 0d0a), Unix (using only oa) and Mac (using only 0d). One should use an unique linefeed character independent of the Operation System. That is not 0d0a (2 characters necessary); it should be only 0a. All transfer mechanism (DOS2UNIX, Text modus for printf in C, etc.) may be seen as stupid.

The javac compilation on different systems with slightly different versions, but with the same encoding settings (!) deliver the same binary codes for the class files. That is a proper message.

Java is operation system independent (Windows, Linux, Mac, some other platforms). But Java has not the capability of simple command execution scripts.

The UNIX-like shell scripts are available on a MS-Windows platform even if git is used as version management systems. For developer for C/++ often MinGW or Cygwin is familiar. If that tools are present, the

runs shell scripts. sh is sh.exe on MS-Windows, a part of MinGW, present if git is present.

In conclusion all scripts can be written (should be written) as shell scripts executable by a linux shell or by sh.exe.

Inside a shell script java can be invoked both for MS-Windows as for Linux. The only one stupid difference in a Java call is: On windows the ; should be used as path separator, on Linux : is necessary. For that an script variable:

is set. The environment variable OS is set to Windows_NT for all windows versions.

algorithm for reproducible jar

In my Java applications a basic component github:srcJava_vishiaBase is always used. The class org.vishia.util.Zip contains the capability to build a jar file.

Calling command:

This is the command line to build the jar file inside all scripts. The settings of the script variables define what is to do. See next chapter.

The input files for the jar are given with a wildcard path with ‘:’ as separator between the base path and the local path for the jar archive. That are the compiled class files which are stored in …​/binjar, but also some additional resource files inside the jar. The script variable RESOURCEFILES is set for example with

to add all files with the given extension inside the SRCPATH. It should be nice to determine "all files exclude *.java", that capability may be add to the routine FileSystem.addFilesWithBasePath (…​) in future.

Note, there is a space inside the RESOURCEFILES for this example. It is possible to give any number of arguments for file selection for the Zip routine.

Algorithm details:

Next the essential statements are shown:

That is the core routine after commandline parsing. The timestamp is converted via java.text.SimpleDateFormat from a human readable command line argument to the long value. The files are contained as class instance variable, see following:

The operation can create both, a normal zip file or a jar file. If the timestamp is set, the manifest zip entry is written with the shown algorithm. It is the same as in the constructor java.util.jar.JarOutputStream#JarOutputStream(…​, manifest), but the e.setTime(timestamp); is the special extra statement. Hence the Manifest entry gets the given timestamp.

It is an non-complex algorithm from org.vishia.util.FileSystem which detects the : separator char (see command line call example) and fills the listFiles with all found files (the class files).

That are the essential file sorting statements. The TreeMap idxSrc sorts all to their local path names, so the order is defined. For compatibility sort or non sort the sorting files are re-written to the listFiles.

That is the algorithm to build the zip file. Depending on manifest either a JarEntry or a ZipEntry is created. The last statement of this group sets either the given time stamp or that of the file. Because for jar building the calling command line contains

All files gets the given time stamp. The user readable form in the script is set:

This is part of the script inside

This file should be updated only in this line on any new version, it is executed both on a gradle build as for a manual started build from the …​source.zip unpacked files. Hence all files inside the jar have this time stamp, and the jar is the same binary independent on the real build date. The time stamp is determined from the stored sources, not from the random build action.

The rest of the algorithm in org.vishia.util.Zip is standard, see the sources on github or on a possible download from the repository www.vishia.org/…​Java download.

The file _make/makejar.sh for this jar component contains (snippets):

The VERSION can be set outside if desired, it affects only the file names. The TIME can be set outside if desired too, but it affects the content of jar and hence the MD5 code because it determines the time stamps in the jar file.

Both script variables VERSION and TIME can be set outside because this script maybe called for some different jar files. The value set inside should be valid for the main jar file. See next chapters for invoke this script.

The TIME is used as UTC-Time. This is important because daylight saving time switching in local times are disrupting (this effect was detected after re-check the algorithm on 2020-04-04 after the spring daylight saving time switch in Europe).

Outside an variable DEPLOY is given for the output directory and start of this file names.

The cleanup is necessary because all files in binjar are files in the jar Archive.

For different approaches either all files in the given SRCALL should be compiled, or only a few files are the primary files given in FILE1SRC

This is the javac command line. JAVAC_HOME should refer the dedicated path to the JDK. Hence it is possible to use different JDK (maybe Java-8, Java-11 etc) for different compilation activities. It is not a system property which JDK is used, it is a property of this file. Of course the JAVAC_HOME variable should be set outside, and the JDK should be present on the PC.

The CLASSPATH and SRCPATH are set outside because this script is more universal.

This line is explained already above. The script variable JAR_vishiaBase should refer either to a already existing vishiaBase-VERSION.jar or it can refer to the yet compiled class file tree, to build the vishiaBase…​jar file itself.

vishiaZipJar-VERSION.jar and vishiaMinisys-VERSION.jar - some specials

From gradle the manual build shell script is called. The gradle java compile capabilities are not used. The build.gradle file contains the following block to generate a vishiaZipJar-VERSION.jar-file:

Because of the source files are arranged in src/main/java this is the working dir (more exact: _make inside the source tree). It is the same working dir as used for manual build.

The temporary directories inside build and the deployment directory deploy should have the correct number of ../ as relative path from …​src/_make.

The CLASSPATH is xx because the argument should be existing. It is not used here.

The SOURCEPATH is .. relative from _make. SRC_ALL is set, adequate of gradle all java files in this directory are compiled.

The VERSION is gotten from the gradle build script variable. It determines only the name of the deployment only.

The TIME value should only be changed if a new jar content is expected or created. The following rule is recommended:

It is possible that some sources are changed, but a certain jar file is not really changed because the sources for this jar file are not affected. Another situation is, that sources from a used jar file (in -classpath of the javac call) are changed, but the compiled jar file is not affected because the call conditions of that used class files are unchanged. With manual comparison of the generated MD5 code (adequate comparison of binary result of the jar) it is able to recognize whether the jar file should be newly deployed.

The FILE1SRC is the last argument of the javac-call, the files to compile. In this case the jar should contain the org.vishia.zip.Zip as only one class, but it needs some dependent classes. All depending classes are part of the source tree, they are referenced with the SRCPATH. The javac-compiler tool searches the depending classes by itself in the srcpath and in the classpath and compiles all classes which are found in the srcpath, non only the given primary one (FILE1SRC). It means the generated jar file contains some more classes, but less ones.

With the same sources te vishiaMinisys_VERSION.jar is built. The difference is: This mini jar file contains only the algorithm to get a file from internet with MD5 check, adequate to the wget linux command. But wget is not available in any sh.exe environment. Secondly the MD5-check is integrated (other then in wget, TODO SHA-256-check). That is the second jar file from this source set, possible independently able to use.

vishiaBase-VERSION.jar - the jar to the whole component

The third jcc-Block is the compilation and generation of the whole vishiaBase-VERSION.jar with the following task:

The task depends on jcc_zipjar, it means the vishiaZipJar-'version'.jar' is generated already before this task runs. Hence this file can be used as jar file to build jar, set into the environment variable JAR_zipjar

The makejar.sh is the same script as used for vishiaBase.

The TIME is build with the versionNumber which should (need to) be written as valid hh:mm format (def versionNumber='02:03') to build a valid time stamp. So the hh:mm value inside the jar is used as version number. The date should be present the date of the last source.

Here FILE1SRC is not set, instead SRC_ALL is given. Therefore all sources of this path are gathered als sources to compile. That is like the normal approach of gradle - compile all given.

vishiaGui-VERSION.jar - another component with dependencies

The next example shows how a more complex jar file is generate:

This complete example shows the generation of the vishiaGui-VERSION.jar which uses the Eclipse-swt…​jar and the vishiaBase-VERSION.jar with version_vishiaBase as library. Both are set to the CLASSPATH variable because the compiler should know the signatures of called routines. Because the same vishiaGui-VERSION.jar is used as tool to build the jar, its path is stored in the internal gradle (groovy-) variable JAR_vishiaBase. It is used twice. The path to that jar file is to a depending component which is located beside the own working tree. It comes from different git archives: github:testJava_vishiaBase and github:testJava_vishiaGui. The first one depends on the last one. But both are not organized as sub modules inside git, they should be cloned for example one beside the other in the user’s working space. The user should generally decide by itself about the directories on its own hard disk. But the script presumes, they are side by side.

This makejar.sh invocation uses a second SRC_ALL2 because there is a third sub project which contains some more sources, side by side too, which’s sources are given with github:testJava_vishiaRun. But the sources of both components are ziped into the same: vishiaGui-VERSION-source.zip file:

The jar and the source.zip should contain the adequate files. It is possible to use the …​-source.zip in generally to reproducible re-generate the jar.

Instead using gradle it is possible to build the vishiaBase-VERSION.jar only with the given sources. Beside the jar in www.vishia.org/…​Java download there is a vishiaBase-VERSION-source.zip. It does not contain all gradle sources inclusively tests, it does only contain that sources which are part of the jar, but inclusively the _make directory:

The makejar_vishiaBase.sh sets the script variables to build the jar manually:

One of the JAVAC_HOME line is uncommented. The lines show which jdk was used for test.

This both variables are differently set from manual build and from gradle build if this paths are more complex. In this case they are simple, but used for all three following makejar.sh calls.

It is the first block to build vishiaZipJar-VERSION.jar. Because of the version should be known in this context it is set to the VERSIONZIPJAR-variable here and to the VERSION which determines the jar file name.

The user should decide where the temporary build is stored. The /tmp directory is set with the TMP variable content of the Operation System in MS-Windows. In my case it is a RAM-disk which safes the SSD write cycles and is faster. It is not necessary to use $TMP instead in the script on Windows.

The DEPLOY determines where the output files are written to and how is the start of name. The rest of the name is determined in the core script.

Adequate to the approach in gradle, the yet even compiled class files are used to build the jar. It is okay if the sources are ok. If the sources are faulty, then the generation is aborted.

This lines should be unchanged. Note the chmod command, it is for Linux.

The following block organizes the build of the other jar files. Because the vishiaJar-VERSION.jar is even generated, it is used at once:

To generate the jar file with dependencies, see `vishiaGui-VERSION-source.zip' from www.vishia.org/…​Java download. The procedure is the following:

Alternatively the path can be changed in the makejar_vishiaGui.sh script. Then the generation is proper.

It is possible that the generated jar has the same content though the depending jars have different versions. It should be evaluated.