Note: The strategy for the src tree was changed in 2022/07, this is the new description using src/cmpX/…​ instead src/main/…​

date: 2022-08-03 / 2023-08-17 / 2024-01-20:

1. Summary for use

Overview

Table 1. basic maven file tree

tree

src

The first level of sources in a working area or "sandbox" is always src

src/cmpnX

The second level is for a specific component (instead main in maven), for the sources of the application itself. See chapter How to separate components

src/cmpnX/java

The third level describes the kind of sources, see chapter Distinction of kind of sources in the 3th level

src/cmpnX/cpp

The third level for C/++ sources.

src/cmpnX/make

The third level for make scripts.

src/cmpnX/docx

Documents for the component as part of the component, here office document files.

src/cmpnX/asciidoc

Documents for the component as part of the component, here ASCIIdoc-sources.

src/cmpnX_test/cpp/…​

Source for test of the component can be separated (instead, adequate test in maven)

src/cmpnX_test/docx

Documents for the component’s test

src/cmpnX_test/IDE_MSC

The third level IDE-files, but see also:

IDE/cmpnX_test/MSC

Parallel to source the working area for the IDE, see see chapter Directories for the IDE beside src

libs

This can be used as an directory, where external Libraries are loaded which can be found in the internet. Such an directory is sometimes able to found, and it is sensible. But in maven external libraries are often also stored in a systems directory. See Libs and tools on the source tree

src/load_tools

See Libs and tools on the source tree. This directory contains only the not files which organzizes loading of tools from internet. Not the loaded files.

tools

This directory contains tools for working, which are usual only simple jar files , batch files or shell scripts. Some of this files are delivered with a given version of an application, but some other can be reloaded from an internet archive too. See chapter Libs and tools on the source tree

IDE

This directory contains in sub dirs files for projects for an IDE = Integrated Development Environment. See especially chapter Directories for the IDE beside src

build

A directory where build outputs where written. This directory should be cleaned if a 'build all' should be done. It can refer via symbolic link also to a temporary location maybe especially on a RAM disk. See chapter build beside src

2. The idea of a common structure of files for working directories

The following idea is gotten from maven:

https://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html

It is not recommended from me to use maven or gradle (wikipedia), this is another decision as the decision for this file tree. But the idea of maven file tree structure is proper to use, also outside maven or gradle using.

See also discussion in https://softwareengineering.stackexchange.com/questions/81899/how-should-i-organize-my-source-tree

The basic for this idea of a well defined file tree is "convention for configuration". That was one of the important step from the older tool ANT from appache.org towards to maven. In ANT, or sometimes in simple make systems, or in software at all, there is no rule which files are stored where.

But the idea of "convention for configuration" is not the reason for the here presented approach. configuration is not so hard to create. The more important reason is, having a well order of files, especially also on using of several components. Primary the maven file tree divides in the sources for the application itself (main) in test sources (test) and also in docs . Furthermore it defines where built file are stored (build), and some libs. That is practical. Here some more additional ideas in this direction are presented, more consequently as in maven or gradle.

The question how to deal with components (see next chapter How to separate components) is also a primary approach for this file tree. It is solved other than in maven or gradle.

2.1. Sources in the working tree

The basic idea can be explained here as following, also considerate the idea how to deal with components, other than in maven:

Table 2. basic maven file tree

tree

src

The first level of sources in a working area or "sandbox" is always src

src/main

The second level is main for the sources of the application itself. This is originally maven.

src/test

The second level is test for test sources, only used for test, not for the product itself, also as in maven

src/main/java

The third level describes the kind of sources, for maven/gradle often Java sources.

src/test/java

Same strategy for src/test, kind of sources

src/doc/docx

Documents should be parallel to test and src.

2.2. Sources of different components in the working tree

Now, abbreviating from the maven idea, instead the anonymous main the name of the component is written. This enables to have some component sources parallel. See also next chapter How to separate components

Table 3. file tree for more as one component

tree

src/cmpnX

The 'main' sources of one component.

src/cmpnX_test

Also the test of a component is handled as a component in the adequate kind.

src/cmpnYZ

The 'main' sources of another component.

You can mount a components source directory via a symbolic link. Then you can have the same component with the same currently (!) source version parallel in more as one working trees, or you can participate in one working tree from the current version anywhere other. This is often a proper approach.

As you see in the table above, the repository for a version system (here .git) has its root in this component tree, the only sources of this only component are part of this version management. See also chapter separated git repository for each component

Table 4. version management roots in the file tree for more as one component

tree

.git

The root of version management not for the sources, but for this working tree. See chapter The git repository for a whole application, clone the components repository

src/cmpnX/.git

Sources for each component should have its own version management, because they are elaborately and reused in more applications. Other than often seen where the directory root for the git tree is only parallel to the src directory, here the git is related to the component. The .git can be the directory for the repository itself, or also a link to a directory where the repository is. See ../../Git/html/GitUsage.html#gitdir

src/cmpnX/script_UsedCmpn

The question is, from where get the other components, if they are not part of the version management on the root. The question should answered related to the components. Each component may need specific other ones. But the components should be placed all in the here shown file tree structure. The components similar used should have all the same version. See chapter load depending components (clone git, mklink)

src/cmpnYZ/.git src/cmpnYZ/script_UsedCmpn

Same for another component

See also chapter Git repository for the whole src tree. The relation between more as one git is made manually, not with the git-intrinsic multi-git-management. The reason is: There is more flexibility. A sub-git (a necessary module) can be created with calling the clone_UsedCmpn.sh script. But also, the proper file tree of the component can be linked, with a always existing git, or also as copy from a zip file or what ever. The decision for that can be done by the user in a specific way specific for this working tree. The scripts support it, they are not executed automatically.

Note: the .git can be the directory for the repository itself, or also a link to a directory where the repository is. See ../../Git/html/GitUsage.html#gitdir

2.3. Distinction of kind of sources in the 3th level

Some more entries for th 3th level:

Table 5. file tree 3th level

tree

src/*/java/…​packages…​

The 3th level follows the maven approach, the sources are sorted to the programming language. For java the package path starts here.

src/*/resources

The third level with another kind of sources, here so named resource files for Java.

src/*/cpp

As also for the C and C++ language, both marked with cpp. This is familiar existing. Sub directories below cpp should be used to structure files, and used in #include "pathTo/header.h". It means the include path should refer only src/cmpnX/cpp.

src/*/asciidoc

This contains the documentation for the component related to the sources. Especially for https://asciidoctor.org/ or https://asciidoc.org/ the source of documentation should be close to the sources of the programs, because it can be simple included with relative links.

src/*/docs

You can also introduce src/*/docs for example word documents. But you should regard that are not textual files, the versioning is not so proper. It means it may be better stored outside of a git and zip file tree, in a document management directory.

src/*/smlk

What is smlk for a kind of sources? - Yes, it is Simulink, also often used.

src/*/mySpecificLang

Either a language have a usual known mnemonic, or you find a shorten. This system of language in the first level is favored for the gradle tree.

src/*/IDE

That contains files for the IDE = Intergrated Development Environment. But it is not the space for the IDE itself. See chapter Directories for the IDE beside src

src/*/script_UsedCmpn

Here some information can be found, which components are necessary as dependent. See chapter load depending components (clone git, mklink)

All in all, the really sources are all located in src. It is often only less, 1..10 MByte. The src subdir must not contain temporary files and also not Megabyte of content which can also be found on dedicated locations in the internet. But the links to this locations should be part of the sources.

Only content in src should be versioned.

It should be taken into consideration that the whole src tree may be also stored and exchange, often as zip file. This is a second approach parallel to a versions history. Both have its understandment. The versions history is usual important to trace the development, whereby free bundled sources files of a dedicated version are proper for delivering and comparison of main versions.

Table 6. backup parallel to src tree

tree

src_back

This directory can be used to store some zip content of the whole src or specific components. Use always a time stamp in the writing style yyyy-mm-dd and a unique name of such zips. Storing zips is not against versioning, it completes the idea of versioning.

2.4. Other parts than sources in the working tree

Furthermore, some common tools and also libs for organization should be placed in the working tree, as also temporaries:

Table 7. file tree common tools and libs

tree

libs

This can be used as an directory, where external Libraries are loaded which can be found in the internet. Such an directory is sometimes able to found, and it is sensible. But in maven external libraries are often also stored in a systems directory. See Libs and tools on the source tree

src/load_tools

See Libs and tools on the source tree. This directory contains only the not files which organzizes loading of tools from internet. Not the loaded files.

tools

This directory contains tools for working, which are usual only simple jar files , batch files or shell scripts. Some of this files are delivered with a given version of an application, but some other can be reloaded from an internet archive too. See chapter Libs and tools on the source tree

IDE

This directory contains in sub dirs files for projects for an IDE = Integrated Development Environment. See especially chapter Directories for the IDE beside src

build

A directory where build outputs where written. This directory should be cleaned if a 'build all' should be done. It can refer via symbolic link also to a temporary location maybe especially on a RAM disk. See chapter build beside src

3. How to separate components

3.1. comparison with Java package path

From the view of maven or gradle with Java using orientation, the topic of components is clarified by Java itself:

Java knows a strong package structure, valid over the whole world. Any source file is well sorted in in this package structure. No conflicts exists, and all Java developer regard this approach.

What is it for an approach:

The package-path starts usual and accepted from all with the URL internet page address of the responsible company or with a commonly accepted name.

So, the Java core sources have the package path java/lang/…​, java/util/…​ Special Java sources start for example with sun/…​ or com/sun if they were created in the past from the Sun company. Other ones start with oracle or more consequently with com/oracle because they are written from a company which can be found in the internet with oracle.com. Sources in the package path org/w3c are from www.w3c.org etc. pp My own sources starts with org/vishia/.

So, in a Java source tree, all components can be mixed without conflicts. Of course the further entries in the package path are well sorted, in responsibility to the company which determines the start of the path. The company should be handle consequently, and confusions with other companies are excluded. There is no company which starts its own sources for example with com/oracle or (I hope so) with org/vishia.

3.2. What is a component

This is sometimes not well defined.

A software or some other technical things consist of modules. A module is an own described and testable unit which can be used in different kinds. A source file or some associated source files with maybe dependencies one another build a module.

An common understandment is, that a component is a greater unit than modules.

A component is either an assembly of modules that form an independent greater unit - or special in software it is an assembly of modules which are delivered together. All modules in a component have the same version and they are tuned together.

This is a sensible definition. Following this, a component has one repository (for example in git), with its versions.

Of course sub components can be defined. Each (sub) component has its repository, and a component consting of sub components has a repository with child dependencies.

For the view of Java: A jar file is a deliviring form of a component maybe consisting of sub components. You should find all the sources of one jar file in one repository. A jar file and its sources has a defined version.

From the view of C/++ developing: Some C or C++ sources which are commit together should be seen as a component. This is a version, with a responsible, and maybe in the delivering form of a library with header files, or even as source pool.

For other languages or also hardware description files it is adequate.

3.3. separated git repository for each component

That is an interesting question.

As also shown in the chapter Sources of different components in the working tree, the .git is related to components. The components should be distinguish in the 2^th level in the 'src\componentX' tree.

src/AComponent
     +-.git
     +-java
     |  +-companies/package/path
     +-resources
src/BComponent
     +-.git
     +-cpp
     |  +-internal/src/tree
     |  +-include ... not recommended
     +-asciidoc/...
     +-some_more/...
     +-lib

For the include directory see also remarks in chapter Separate source and include directories or arrange header beside sources

The sources of the components are independent of the maven file tree idea and they are also independent of any names (which may not have been reconciled).

This is the best way for separation. This may be also valid for Java, though Java sources are already well separated by the package tree.

For example usual in my Java sources I have the structure:

src
 +--java_vishiaBase/
 |   +-.git
 |   +-java
 |      +-org/vishia/...
 +--java_vishiaGui/
 |   +-.git
 |   +-java
 |      +-org/vishia/...

Both components have an additional directory between java/ and the Java package tree. That is not mentioned for maven. Maven suggest src/main/java/org/…​ without a component sub directory. Why?

Maven has another concept for components. Usual it uses complete jar files, which have the component structure as jar file, and this jar files are not part of the sources in your own source tree. Often their are stored in a temporary folder (C:\users\myName\p2\pool\plugins\org.apache…​jar) and updated automatically from internet on demand. This is the maven approach, get all over the internet from the world. But this approach is not proper for all. In particular, the questions "which version is used", "are all impacts considered" and some more are not necessarily sufficiently clarified with such an approach. The second drawback is an overly unexplained dependency on the Internet. But exactly the approach "everything can be found in the world" is the core approach of maven. Maven is oriented to large software packages - no limitations, hard disks have enough space. This one is not recommended by me! And also often not proper for embedded control. But the maven file tree is a proper idea.

3.4. Git repository for the whole src tree

The .git in the component’s directory is for the component’s version. Whereas a .git in the whole src or working tree is for the application or test environment.

MyWorkingTreeForWhatEver
 +-.git
 +-src
 |  +-cpmnX
 |  |  +-.git
 |  +-cpmnY
 |  |  +-/.git
 +-IDE/...
 +-tools
 +-build
 +-someScriptfiles.sh.bat

or adequate

MyWorkingTreeForWhatEver
 +-src
 |  +-.git
 |  +-cpmnX
 |  |  +-.git
 |  +-cpmnY
 |  |  +-/.git
 +-IDE/...
 +-tools
 +-build
 +-someScriptfiles.sh.bat

The second form emphasizes the idea that only sources should be versioned. The content of build is by the way temporary, should never be versioned. But from where comes the tools, the content of IDE and some scripts. The answer is following:

  • The tools and also libs should be loaded from a trusted source in the internet, by the way. It is a question of versioning and also save memory in all application sources to do so. The scripts to load the tools and libs are part of the component’s sources, see Sources of different components in the working tree. There is mentioned a src/cmpnX/script_UsedCmpn/load_UsedLibs.sh script. Also in the chapter Other parts than sources in the working tree there is mentioned a src/load_tools/* directory with a bill of material list and some scripts.

The repository (or other version tool) beside the sources should contain all stuff which is necessary for the application or test environment additonal to the component’s sources,

and

should refer to load files of the components, at least the primary ones. Because the components contains the src/cmpnX/script_UsedCmpn/*, further dependencies are clarified with it.

Note that you can have a .git directory inside that sources:

src/component/.git/TheRepository_itself

or you can have a .git file which refers the repository on another location:

src/component/.git
.git contains:
gitdir: path/to/repository/.git

The second one it preferred by me, because two reason:

  • On manual archiving (create a zip) you have not all stuff of the repositories in the zip

  • You have a mirror location which make it easier to compare, change, have experience.

3.5. Using the same component sources in different working trees

Sometimes one working tree is for one application, the other one is for elaborately tests, and a third working tree or Sandbox is for another application.

Then sometimes it should be clarified that all application should use exactly the same sources (test with the same sources should be clear, another application should be and proper behavior to the first one).

There are three ways to get it:

  • a) All components should be written in only one working tree. This is possible, the files should not be confusing because of the proper inner file tree. But the disadvantage: If you want to clarify, which sources are used, or deliver the sources with a zip file, it is too much. All applications are in one. Maybe also the overview about the used files is lost.

  • b) Each component (or specific determined components) should have its own working tree. That is also proper if the maintainer of the working trees are different persons, on different computers. Then, the content should be tuned, should be the same.

    • For that either all working trees uses the same version form a central repository (in network),

    • or the working trees are tuned by comparison of the file content.

  • …​ A diff viewer for files in a directory tree works usual fast. The presumption is: The directories should be present either in the same network, or, also possible, a interchanged data storage is used. ("Please give me your files on a stick, I compare it").

    This opportunity is also able to use for temporary or for experience different versions.

  • c) Each component (or specific determined components) have its own working tree, but the directories of the used sub components are symbolic links (known from UNIX, but also able to use in Windows with mklink /J linkname path/to/src).

    Then, similar as a) changing of the source in one working tree (for one application) offers the changed sources immediately for the other application. Often this effect is desired. One is tested, changed, then without additional effort the other application or test application is tested with this same changes.

    The ability c) helps to sort the files (better than a), and gives an opportunity to switch some components to the b) approach, and back to c), only by changing the link, instead copy or locally checkout another version, all what is necessary.

    It means this is often the best approach.

Example:

D:/software/workingTreeA/src/compn_A/cpp/filex.cpp
                               +
                               +-----------+
                                           +
N:/networkdrive/software/workingTreeB/src/compn_A/cpp/src/filex.cpp

compnA should either a copy, compared, or a linked directory. Changing in D:/software/workingTreeA/ can force an immediately change in the other N:/networkdrive/software/workingTreeB without effort. For example another person can immediately test, or you start an remote test. Note, for a network drive on windows you should use mklink /D, a simple Junction does not work. But: A Junction inside the same hard disk is proper seen from another PC in network.

  • mklink /J name targetdirpath creates a so named Junction, which works only for a targetdirpath on the same hard disk but it is seen proper in the network.

  • mklink /D name targetdirpath creates a symbolic link, also able to any other drive such as also a network drive. This is the adequate to a symbolic link in UNIX. This command needs administrator rights to execute.

  • mklink /H name targetfilepath creates a file link, so named 'hard link' similar as UNIX, whereby the targetfilepath need to be on the same drive.

See also chapter load depending components (clone git, mklink) and chapter mklink to other locations

Testing approach using git test abilities:

Of course, such scenarios are also offered in a Git test environment. But then all changes should first be committed without testing for the test, not the best approach in all cases where minor changes should be checked for compatibility.

The core of this question is outside of the source tree organization question, but some details are related.

3.6. How to get interfaces to a binary given library

There are two systems, the system of C and C++, and the system of all other languages.

3.6.1. Common solution: interface as part of the binary

Lets present this in Java. The class file (the compilation result) contains the signature of public data and all operations of the appropriate class. If a user file is compiled and uses a precompiled class in any library, there is a

import package.path.to.UsedClass;

The compiler sees this statement, has a search path to all libraries, searches the appropriate pre compiled class and knows how fields and operations should be used.

If the class is part of the sources to compile, the compiler knows all sources to compile as a whole (other than in C/++ where only one compilation unit is handled in one step). It means the compiler (javac) can check whether this class is part of the source tree. This is also supported from the fact that Java has a strong rule: A public class must organize in a file.java with exact this class name, and all files are assembled due to the package path. This enables the javac compiler to immediately check whether the class is presented as source file. If it is found, the first pass of the compilation is done for this used class, while the compilation of the using class is continued. It is really a powerful simple organization based on the file and package path rules. Only if the appropriate source file is not found, it will be searched in all given libraries.

More as that. You can have a used class in one version while compiling, but present the same class in another version on runtime. Both is admissible if the other version of the class is compatible. This is detected before run time, while a class is loaded and compiled to the machine code. The linkage pass is done in Java before start of a program (so named JIT, just in time compiler).

The interface concept in Java increases this flexibility even more. To compile time only the interface should be present as file. The implementation (class file) can vary.

3.6.2. C / C++

The system in C/++ is more simple. The header file, should be present as source while compilation, describes the interface to the implementation. Here the wording 'interface' is meant commonly, independent of the interface concept in the language. A library must be translated with the same header file, or at least with a header file with the appropriate content, and (!be attention!) with the same compilation guides (compiler flags, pragmas). In C language nothing is checked in linkage time (for example different packing strategies of non bounded data elements, or also type of arguments of functions). The result may be a insufficient non obvious behavior, or better for the user, a consequently crash. But this is not good for the developer or the software maintainer.

C++ checks more, which results then in sometimes obscure (for newcomer) linkage errors. But better for the whole software.

Thats why (a little conclusion) I use Java instead C/++ for non embedded applications.

3.6.3. Separate source and include directories or arrange header beside sources

The separation in source and include sub directories is very familiar. But the sources and their header files belong together. Do not separate what is related! The idea to separate header files in a include directory comes prior from UNIX whereby the sources are not present, instead delivered as library (lib) and the include describes the source given interface to the libraries. Secondly the include and the lib is used in different compilation phases, the first for compilation, the second for linking. Whether the include is real proper to the lib, this is only clarified by a common version, or perhaps by chance and sluggishness, not else. Elsewhere it may be better to arrange the include to the lib. But in the old time the whole unix system with its C compiler was delivered from one source in a proper way. Also, the open source idea was invented only in the 80^th with the "GNU NOT UNIX" slogan.

For C/++ development for embedded applications the libraries have a lesser meaning. Usual the whole software is compiled as a whole, also for variation of some implementation conditions (optimizing rules, association to RAM or ROM, maybe some conditional compilation etc.). My point is, you should not separate header and implementation files. Store it side beside in the same source directory. Also if a component is delivered by a library (to save compile time), the sources are a point of interest and should be also delivered with that pre compiled library, beside the header.

4. Libs and tools on the source tree

4.1. Depending parts loaded from internet

A common approach also used for such systems as gradle and maven is: Some necessary libraries and tool files are loaded on a central position on the PC. This is the home directory of the user ('C:\Users\name…​' in Windows). Then there are accessible. For example on Eclipse usage you find:

C:\users\myName\p2\pool\plugins\....

One of the idea is, the libs and tool files have a unique version name, the same as worldwide in the internet, in the adequate (maven…​) repositories. The local existence on the PC is only a mirror for fast and internet free access.

There is a principle discussion: Should a file with a versioned name to be used, or should the file have the simple name, and it is gotten from a version archive?

  • a) The first case, versioned name, enables to have more version parallel and central. This is the maven approach. An application should use the file with the versioned name. For the maven build system this is clarified.

  • But if another non-maven application should also use this file, it should know the versioned name.

  • b) The second case, unique name as archive member, requests that the de facto versioned file is stored in a locally relation to the using application. It can be seen as disadvantage, that the same file in different versions should be stored in different folders, and maybe sometime the same version is also stored more as one time in different folders, which needs space on hard disk.

  • But the advantage for the localized usage is: It accesses the file with the simple known name.

Maven/Gradle has decided to a). But this has also some more disadvantages:

  • An application is not closed on a working space, it needs also some parts in other directories. This is not obvious. For example if an application is started with another user name, it does not find the libs.

  • If you deliver a version outside of internet usage as so named "copy deployment", a simple bundle of files for example as zip, it is not complete.

  • You have often not an overview which libs and tools are used, which are present, are they currently or used …​.

  • The amount of libraries and tool files seems to be near unlimited. Too much. Loaded libs and tools from older versions remain loaded on hard disk.

Of course the pure maven approach includes the possibility of cleaning all, and reload only new necessary files from internet. It is a internet driven approach.

The approach b) means, that an application is closed containing on a directory tree, with all necessities.

  • The files can be loaded one time from internet, then there are locally findable. Of course cleaning and reloading is possible, but often not necessary.

  • Disadvantage: If you have more applications which uses exact the same version, this files are existing more as one time on the hard disk.

  • But for related tools you can have also a symbolic linked directory ( in Windows often a so named Junction mklink /J NAME to/dir is possible.

  • It may be seen as advantage, that an update of the libs and tools with a new version is effective immediately without any effort. The application uses the same file with the same name, in the newer version. If a new version is tested, and it is compatible, this is not an disadvantage, more, it is adequate the DevOp apporach (Development and Operation).

  • If it seems to be that the newer version may be faulty, or only a comparison with the older version should be done, it is only necessary to intermediate store the other version, maybe with temporary broken symbolic link only for the test. It is very simple to do this change.

To support a closed directory tree the approach b) is better and here recommended.

4.2. libs and tools content only load one time, associated to the working tree

Hence in my work on a working file tree I prefer a directories libs and also tools beside src and all other.

  • libs should contain loaded libraries, for Java approach this are especially .jar files. For C/++ usage this can be also pre-compiled libraries.

  • tools contains also often .jar files, but not to use for compilation. There are small tools to work.

    This tools directory does not contain the elaborately files for example for IDEs (Integrated Development Environments) in sizes of Gigabyte. It should only contain small tools in less MByte size and maybe only some shell scripts of batches.

To avoid using too much space, there is a possibility using links, see also Chapter Copy the IDE project files from src/cmpnX/IDE_Platform to IDE/cmpnX/Platform where links are also used and explained.

The tool files are usual able to found with proper versions in internet archives. Also the sources should be found beside this archives in internet, as best case using a Reproducible build approach, see vishia.org/Java/html/source+build/reproducibleJar.html.

So on delivering a source tree in this form for example in an zip archive this files need not be a part of them. They can be loaded from internet, only one time on creation or unpack of this source file tree. After them they are stable present, associated to the application, without conflicts to other application and independent of the internet.

To load this archives from internet a small tools/minisys_vishia.jar is used as part in the git archive as only one common. It contains the necessary GetWebfile class.

Wget as known linux cmd is not available unfortunately in a standard MinGW installation, neither it is anyway a standard on any Linux System. Hence it is provided with the minisys_vishia.jar for all systems where Java runs. But minisys_vishia.jar does more.

The GetWebfile works with a bom, a bill of material, see articel in german: Jeff Luszcz "Risiken bei Open-Source-Software: Warum eine Bill-of-Materials sinnvoll ist"

java -cp tools/vishiaMinisys.jar ...
  org.vishia.minisys.GetWebfile ...
  @tools/bomVishiaJava.txt tools/

(…​ is for line continue).

4.3. Using a bom (bill of material) for which tool files in which version

See also vishia.org/Java/html/source+build/src_Archive.html#loadTools

Following is an older description.

The bomVishiaJava.txt contains the re-check of the vishiaMinisys.jar, and check and download of vishiaBase.jar and vishiaGui.jar. The bom contains MD5 checksums. With it the already existing vishiaMinisys.jar is checked whether the checksum is okay. It it is not so, a warning is outputted. The other files are loaded and checked (whether the download is correct). If there are existing (on repeated call), the MD5 checksum is build and compared. The MD5 checksum is noted in this archive. Hence it is not possible (with the safety of MD5) to violate the files all on server, downlaod process and on the own PC.

The next importance is: It is documented which files are used from where. Other systems loads some downloaded stuff in a home directory (C:\Users... on Windows), not simple obviously which and from where. And the third importance is: The sources of this jar files are stored beside the jar file at the server. The jar files can be build reproducible (see https://www.vishia.org/Java/html5/source+build/reproducibleJar.html).

  • The tools/vishiaBase.jar is a Java executable archive (class files) with about 1.2 MByte, which contains especially the JZtxtcmd script interpreter. That is used to generate the test scripts and for Reflection generation (further usage of sources). It is a necessary component. This file is downloaded from a given URL in internet. If necessary you can find the sources to this jar file beside the jar file in the same remote directory. With the sources you can step debugging the tools for example using the Eclipse IDE https://www.eclipse.org.

  • The tools/vishiaGui.jar as Java archive contains the ability to execute the SimSelect GUI which is used in src/test/ZmakeGcc/All_Test/test_Selection.jzT.cmd to build and executed specific test cases. It also contains some other classes for example for the 'inspector' or the 'file commander'

5. build beside src

In the maven or gradle approach beside src there should be a build:

 +-src
 |  +-main
 |  +-test .....
 +-build

The build is the destination folder for all built results, also for the end-used executable. The executable (the last result of built) can be copied from there to a delivery directory.

The content of build should be seen anyway as temporary. The build process can be repeated any time and should be repeated in a 'clean all & rebuild' approach.

It may be recommended to use a RAM disk for this build organized with a symbolic link with the advantage that writing and the build process runs faster and the hard disk is not used for too much temporary stuff.

If you want to get a zip archive from the sources, you should not include the build, zip only the src directory.

You shold remove all temporary stuff with one batch file:

Script: +clean_build.bat as pattern
echo off
REM cleans build directory

echo called: %0

REM remove first the content of build because it may be a link.
REM the next deletes only files, do not delete directories ...
if exist build del /S/Q/F build\* > NUL

REM remove all
if exist build rmdir /S /Q build

REM now remove build, maybe as linked folder.
REM Note: rmdir does not remove files in a linked folder. ...?
if exist build rmdir build

if not "%1"=="nopause" pause
exit /b

Similar as shown in chapter Temporary location for build and output directories for IDEs it can be created as link.

Script: +clean_mklinkBuild.bat as pattern
echo off
echo called: %0
REM cleans and creates a directory in %TMP% for temporary stuff.
REM recommended: Use a RAM disk for such if you have enough RAM istalled (0.5..1 GByte is usual really sufficient for the RAM disk)
REM a build directory will be created as symbolic link (JUNCTION in MS-Windows) to this temporary folder.
REM The used temporary for the link build
set TD=%TMP%\WorkboxSpecificDir

call .\+clean_build.bat nopause

REM TMP should be set in windows, it may refer a RAM disk
REM only emergency if TMP is not set:
if not "%TMP%"=="" goto :tmpOk
  REM Windows-batch-bug: set inside a if ...(...) does not work!
  echo set TMP=c:\tmp
  set TMP=c:\tmp
  mkdir c:\tmp
:tmpOk
echo TMP=%tmp%

REM clean content if build is not existing, and link
if not exist build (
  REM Note: rmdir /S/Q cleans all, del /S/Q/F does not clean the directory tree
  if exist %TD% rmdir /S/Q %TD%
  mkdir %TD%
  mklink /J build %TD%
  echo build new created > build/readme.txt
)

if not "%1"=="nopause" pause
exit /b

6. Directories for the IDE beside src

This idea is not the basis for the Maven or Gradle approach. Maven or Gradle can be seen also as a build system outside of an IDE

IDE = Integrated Development Environment such as Eclipse, Visual Studio or a specific IDE for embedded software.

Firstly, the IDE project files can be seen as part of the sources. Then it should be stored below src/component/IDE. But that has an disadvantage:

Often beside the project files of the IDE the temporary directories are created. If they are inside src... and you make a fast zip or backup with the sources, the difference is: You may have only kByte or less MByte for the sources, or otherwise 100.. MByte with the sources and the temporaries of the IDE. That’s worse.

Unfortunately, common IDEs have the disadvantage of storing their generates alongside their IDE project files. It will be very more better if the build or bin directory are able to set inside the IDE properties to any desired proper other location, maybe a RAM disk, but anyway outside sources. Usual it is not so. The Microsoft Visual Studio does not only store generates in the adequate debug and release directory, additional a large database file is immediately created next to the project file.

Separation of the IDE solves this problem. The IDE beside src should not need to be copied in a backup, and it can be anytime removed (cleaned).

Filetree: IDE beside src
 +-src
 |  +-cmpnX
 |  +-cmpnX_test
 |     +-IDE_MSC        IDE files for test on platform MS Visual Studio
 +-IDE
    +-cmpnX
       +-MSC            use the same 3th level for the IDE files.
       +-Platform_B

As you see in the example tree you can have more as one IDE files for different platforms, for test, and maybe also different applications.

The IDE project files should refer the source files always starting with a back path (../../../src/…​.) but that is usual not a problem. All (known by me) IDEs support relative paths going outside to sources.

6.1. Copy the IDE project files from src/cmpnX/IDE_Platform to IDE/cmpnX/Platform

As described in the chapter above the IDE files should be not part of the src tree, they should be separated in ÌDE beside the src to avoid trash in the src tree.

But: Some of the IDE files should be stored in the software version, without additional effort. How to do?

There is a solution using hard links. What is a hard link?

A hard link is known in the UNIX world since ~1970. It means the same file content is available from different directories. In UNIX (or LINUX) you can create a hard link to an existing file with the cmd

ln path/to/existingfile linkedfile

Then the same file content is available also in path/to/existingfile as in linkedfile from view of the current directory.

Such hard linked files are a little bit worse understandable by normal users. Hence this system was not available in the first Windows versions and not in DOS: But currently, windows supports hard links:

mklink /H linkedfile path/to/existingfile

It is the same as in UNIX/Linux but of course with other order of arguments :-(

Look on the following example: In a specific project the files for the IDE are stored as:

Filetree: Files as original in src for the IDE
src/ExmplOrthi
     +-+spawn_IDE_MSC.bat
     +-IDE_MSC6
        +-+clean.bat
        +-+clean_mklink_build.bat
        +-ExmplOrthi.sln
        +-ExmplOrthi.vcxproj
        +-ExmplOrthi.vcxproj.filters
        +-readme_doNotUse_thisLocation.txtThe script

The batch file +spawn_IDE_MSC.bat copies the files in IDE_MSC as hard link

Script: +spawn_IDE_MSC.bat as pattern
echo off
REM NAME_CMPN name of the software component, DIR-name after src/
REM PLATFORM short designation of the platform, 3th part of IDE/%NAME_CMPN%/%PLATFORM%
REM NAME_PJ name of the IDE project
set NAME_CMPN=ExmplOrthi
set PLATFORM=MSC
set NAME_PJ=ExmplOrthi
echo cleans and creates a directory IDE\%NAME_CMPN%\%PLATFORM% beside the src tree
echo as Workspace for the %PLATFORM% Project and Test and Simulation.
echo All Project files are hard linked to the src tree beside.
echo but be carefully, some editors cut the link.
echo compare the project files in IDE\%NAME_CMPN%\%PLATFORM%
echo with the original sources in src\%NAME_CMPN%\IDE_%PLATFORM%
echo for vesioning and save back.
echo if the files are proper, you can remove this just created location anytime.
cd /d %~d0%~p0\..\..
echo currdir=CD
if not exist src\%NAME_CMPN%\+spawn_IDE_%PLATFORM%.bat (
  cd
  echo ERROR faulty path, should find: src\%NAME_CMPN%\+spawn_IDE_%PLATFORM%.bat
  pause
  exit /B
)
REM file tree structure see https://vishia.org/SwEng/html/srcFileTree.html
if not exist IDE mkdir IDE
cd IDE
if not exist %NAME_CMPN% mkdir %NAME_CMPN%
cd %NAME_CMPN%
if exist %PLATFORM% (
  echo WARNING exists: %CD%\%PLATFORM%
  echo will be deleted, press abort ctrl-C to exit
  pause
  rmdir /S/Q %PLATFORM%
)
mkdir %PLATFORM%
cd %PLATFORM%
echo creates a so named hard link, the files are the same as in src\%NAME_CMPN%\IDE_%PLATFORM%
echo but be carefully, compare the files, some editors cut the link
mklink /H ExmplOrthi.sln ..\..\..\src\%NAME_CMPN%\IDE_%PLATFORM%\ExmplOrthi.sln
mklink /H ExmplOrthi.vcxproj ..\..\..\src\%NAME_CMPN%\IDE_%PLATFORM%\ExmplOrthi.vcxproj
mklink /H ExmplOrthi.vcxproj.filters ..\..\..\src\%NAME_CMPN%\IDE_%PLATFORM%\ExmplOrthi.vcxproj.filters
mklink /H +clean_mkLink_build.bat ..\..\..\src\%NAME_CMPN%\IDE_%PLATFORM%\+clean_mkLink_build.bat
mklink /H +clean.bat ..\..\..\src\%NAME_CMPN%\IDE_%PLATFORM%\+clean.bat
dir
pause

The result is a new created file tree on IDE/MSP430/TimeIntr beside `scr`with

Filetree: appearance in the IDE directory
 Verzeichnis von D:\vishia\fbg\source.wrk\IDE\ExmplOrthi\MSC

2024-01-20  20:36    <DIR>          .
2024-01-20  20:36    <DIR>          ..
2021-10-01  21:54               528 +clean.bat
2024-01-20  19:04               418 +clean_mkLink_build.bat
2024-01-20  20:36    <JUNCTION>     .vs [t:\tmp\FBcl-test\MSC15_ExmplOrthi\.vs]
2024-01-20  20:36    <JUNCTION>     Debug [t:\tmp\FBcl-test\MSC15_ExmplOrthi\Debug]
2024-01-20  19:00             1.631 ExmplOrthi.sln
2024-01-20  20:36           225.280 ExmplOrthi.VC.db
2024-01-20  20:01            19.607 ExmplOrthi.vcxproj
2024-01-20  20:01            22.447 ExmplOrthi.vcxproj.filters

Note: If you need a sub directory as IDE files, use a "JUNCTION" (a directory symbolic link)

The .vs and Debug are created from calling +clean_mklinkDebug.bat what is another topic too, see next chapter. The other files are hard links to the versions files in src/EcmplOrthi/IDE_MSC. If you change the project files, it is automatically changed also in the source directory. But all temporary stuff is locally in IDE/…​..

There is a pitfall: If an editor reads the given file, but on save it removes the file and creates a new one with the same name, the hard link relation is broken. You should know your tools. Unfortunately working with hard links may not be accepted by all tools, because it is not so familiar in Windows (known in UNIX since ~1970). The solution for this problem is: Compare the file content from IDE beside src and src/cmpnCY/IDE_Platform/…​ if you have expected changes and then commit the stuff in the src/cmpnX directory.

6.2. Temporary location for build and output directories for IDEs

The content of +clean.bat in the chapter above is:

Script: +clean.bat as pattern
echo removes %~d0%~p0\Debug etc (*.db, *.sdf, *.user, .vs, x64 etc)

if exist %~d0%~p0\debug rmdir /S/Q %~d0%~p0\debug
if exist %~d0%~p0\dbgC1 rmdir /S/Q %~d0%~p0\dbgC1
if exist %~d0%~p0\x64   rmdir /S/Q %~d0%~p0\x64
if exist %~d0%~p0\x32   rmdir /S/Q %~d0%~p0\x32
if exist %~d0%~p0\Win32 rmdir /S/Q %~d0%~p0\Win32
if exist %~d0%~p0\.vs   rmdir /S/Q %~d0%~p0\.vs
::if exist %~d0%~p0\*.user  del /S/Q %~d0%~p0\*.user
if exist %~d0%~p0\*.db    del /S/Q %~d0%~p0\*.db
if exist %~d0%~p0\*.sdf   del /S/Q %~d0%~p0\*.sdf

it removes the temporary stuff, and supports a 'clean' approach.

The +clean_mklink_build.bat creates a symbolic link, a 'junction' in windows for the Debug and Release temporary directories to a %TMP% location. If you have this %TMP% localized in a RAM disk, you save time and burden for your hard disk. For a RAM disk you can use for example http://memory.dataram.com/products-and-services/software/ramdisk. It is recommended if you have enough RAM in your system (>= 8 GByte). Using 1 or 2 GByte as RAM disk is usually enough and sufficient.

Script: +clean_mklink_build.bat as pattern and example
echo off
call +clean.bat
set DBG=%TMP%\FBcl-test\MSC15_ExmplOrthi
if exist %DBG% rmdir /S/Q %DBG%
mkdir %DBG%
mkdir %DBG%\Debug
mkdir %DBG%\Release
mkdir %DBG%\x32
mkdir %DBG%\x64
mkdir %DBG%\.vs
if not exist Debug mklink /J Debug %DBG%\Debug
if not exist Release mklink /J Debug %DBG%\Release
if not exist x64 mklink /J x64 %DBG%\x64
if not exist x32 mklink /J x32 %DBG%\x32
if not exist .vs mklink /J .vs %DBG%\.vs
pause

7. The git repository for a whole application, clone the components repository

If you follow the approaches for components in chapter Using the same component sources in different working trees then you have proper versions for all your components. But the versioning of the whole application is missing.

You should not use the approach a) in chapter Using the same component sources in different working trees, not all applications and tests in one working tree. It may be confusing.

If you have a working tree for one component, maybe with test in the src/test tree, you can have a git repository for the whole tree. But this should not contain all files! It should be a parent git (or other version system) which refers to the version of the sub repository.

I’m not favoring a git sub directory tree, it is too much git-oriented and unflexible. For example if you yet only want to have a copy, or sometimes use other version systems than git, or whatever else, it is more flexible to deal with separated repositories. You have the responsibility to adjust all manually, of course, but this is done anyway if you decide about the versions. Please follow the idea in the sub chapter:

Solutions are shown in the next chapter.

8. load depending components (clone git, mklink)

For any component the depending components necessary to work with should be known. Either the presence of a jar file is sufficient, this is also for compiling java, used as library. Then the jar file should present in tools, see chapter Libs and tools on the source tree.

If you need the sources of the depending component, then some files in

src/cmpnXY/script_UsedCmpn/

should be existing for each component or even one for all in some variants from where to get. This script files are part of the sources.

8.1. mklink to other locations

It may be sensible to have other components in exact the same sources as in other environments. Then you work only on one source pool. This is your decision. Sometimes it is better to separate sources, because changing on one component should not be side effect to others. But sometimes it may be also sensible to use just the same sources, test it in one environment, and uses the changes immediately for others (and test it also in this situation). Note that you can anytime break this dependency and load another version of sources, if you see problems, search for errors.

The strategy should be: Some far components can be consistent in the same kind in some hard disk contents, for some developer. For example in my development I store all stuff under D:\vishia, but that is not important. But as sub directory there is a proven structure:

Filetree: Example of proven structure of some development parts
D:\vishia
    +- Java     Java sources, tools
    +- emC      "embedded multiplatform {cpp}, sources, tools
    +- FPGA     FPGA stuff
    +- fbg      Graphic programming with Function Blocks
    +- etc.

All this main directories uses usual a dedicated

Filetree: Example of proven structure of some development parts
D:\vishia
    +- Java     Java sources, tools
    |   +- tools
    |   +- cmpnJava_vishia
    |       +- src
    |           +- srcJava_vishiaBase
    |           +- ....
    +- emC
        +- cmpn_emC
            +- src
                +- src_emC
                +- src_emC_Appl
    ...etc.

This is well structured. It can be structured in a similar or better the same kind on another people which uses this same sources.

Then search the component and mklink /J is possible:

The following script is part of the component vishiaJ2Vhdl which is a translator for Java written FPGA designs to VHDL. This component contains the sources for the translation. But it has dependencies.

Script: src\vishiaJ2Vhdl\script_UsedCmpn\mklink_UsedCmpn.bat
@echo off
REM this file should exists in adapted form two levels to root, parallel to this component dir vishiaJ2Vhdl,
REM it is yourWorkingDir/src/mklink_SrcJava_vishiaFpga.bat
REM If it is started from the original given, then either the copy is done, or the adapted script is started.
echo called: %0
REM goto working dir
cd /D %~d0%~p0\..\..\..

echo create non existing links (JUNCTION) due to %CD%/%~n0.bat
if not exist src\vishiaJ2Vhdl (
  echo faulty location of %0
  pause
  exit /b
)

REM create a symbolic link to a directory where the src/vishiaFpga is originally stored, adapt this path if necessary.

if not exist tools mklink /J tools ..\..\Java\tools

cd src
REM create a symbolic link to a directory where the src/vishiaFpga is originally stored, adapt this path if necessary.
if exist srcJava_vishiaBase goto :srcJava_vishiaBase_ok
  if exist ..\..\..\Java\cmpnJava_vishia\src\srcJava_vishiaBase mklink /J srcJava_vishiaBase ..\..\..\Java\cmpnJava_vishia\src\srcJava_vishiaBase
  else echo srcJava_vishiaBase not able to find
:srcJava_vishiaBase_ok

REM The vishiaFpga should be stored here as original
if not exist vishiaFpga mklink /J vishiaFpga XXX-path-if-not-exist

REM The exmpl_vishiaJ2Vhdl_BlinkingLed is necessary here for generate the documentation (link to example code snippets):
if not exist exmpl_vishiaJ2Vhdl_BlinkingLed mklink /J exmpl_vishiaJ2Vhdl_BlinkingLed ..\..\Example_J2Vhdl_BlinkingLed\src\exmpl_vishiaJ2Vhdl_BlinkingLed

dir
if not "%1" == "NOPAUSE" pause

Inside the script it is tested whether a component is located in a relative location a little bit more far on the hard disk. If the strategy is: Using this component from the original position, one should make a directory link to it. Then the component is presented in the src tree (as link) where it is expected.

8.2. Clone git repository from an internet location

This is the other variant, the far variant, work with worldwide versioned sources.

You can/may/should have one file for each component which clones the repository from a remote location. The script example comes also from vishiaJ2Vhdl as the second variant there. It clones the srcJava_vishiaBase. The script can be used in the same form also for other components (copy it, copy the content). It starts from the directory two levels after src but changes to src\

Script: src\vishiaJ2Vhdl\script_UsedCmpn\mklink_UsedCmpn.bat
echo this shell script gets the srcJava_vishiaBase core sources
VERSION_TAG="2023-10-12"
cd `dirname $0`/../..  ##script directory as current, cd to src/...
echo clone srcJava_vishiaBase tag $VERSION_TAG from github if necessary ...
if ! test -d srcJava_vishiaBase; then
  if test $VERSION_TAG ==""; then
    git clone https://github.com/JzHartmut/srcJava_vishiaBase.git
  else
    echo clone the srcJava_vishiaBase with tag "$VERSION_TAG" as 'detached head':
    echo git clone https://github.com/JzHartmut/srcJava_vishiaBase.git -b $VERSION_TAG
    git clone https://github.com/JzHartmut/srcJava_vishiaBase.git -b $VERSION_TAG
  fi
  cd srcJava_vishiaBase
  pwd
  echo touch all files with the timestamp in .filelist:
  #this file is part of TestJava_vishiaBase, hence the .../libs exists:
  java -cp ../../tools/vishiaMinisys.jar org.vishia.util.FileList T -l:.filelist -d:.
else
  echo srcJava_vishiaBase already exists, not cloned
fi

You can use the clone with the given version or comment the VERSION_TAG to get the last version. If you want to have the clone of the last master commit, for currently developing work, you can uncomment and comment the adequate lines.

For commonly usage you can choose a proper directory name, appropriate to the component name in your working tree, independent of the name in the remote git repository. But sometimes the name of the current git repository is also proper for the component’s name.

The clone write the working files in the given dstdir. Hence this file should be arranged beside the component’s directory. This is proper because, instead the component you have the +gitclone…​sh files in a cleaned src tree. This may be the delivering form of an application without all components.

Now you can either immediately execute the +gitclone…​sh file, which writes both the repository and the files in the component’s directory. But this is against the recommendation in chapter separated git repository for each component. It is possible, of course.

The other variant is also simple: clone firstly, then move the .git repository to its specific other location, and write a .git file instead. Then checkout the files on the git-Revisionarchive-location to get a mirror.

The org.vishia.util.FileList restores the timestamp of all files, which is not done by a git checkout command respectively by the git clone. See also restoreTimestampsInSrc.html

8.3. Get the sources as zip and unpack it

From github or other central git repositories you can get one version as zip. Unpack it on the proper position. Be careful with the start directory.

The other way to get a zip is from the deploy archive of jars, where the …​source.zip are beside.

See vishia.org/Java/html/source+build/src_Archive.html#source_zip

8.4. Check the sources with reproducible build

Before you use sources you should clarify whether the compilation is identically with used jar file content.

See vishia.org/Java/html/source+build/src_Archive.html#rebuild