Zmake Generation

Topic:.ZmakeGen.

Topic:.ZmakeGen.substantiation.

Last changed: 2015-03

A maker has three tasks:

• Organisation of the process of make, calling of compiler, any other translator, linker in the correct order.

• Checking of necessity of translation: comparison of timestamps of all sources (indirect depending ones too!) and the destination.

• Decision what is necessary to make: A make file can contain a lot of tasks for some targets. If a determined target should be maked, only that targets are necessary to translate which are need for the goal target.

The last of the three points is one of the basic idea of make, but it is not used often. Make files are written for a specified target and only the really necessary translations are contained there.

The second of the three points are a problem often: The basic idea for make created with C and UNIX in the 1970-th was: Changed files have a new timestamp. If a file was copied from an older archive, the copy of the old file gets the current time. So the maker detects that a translation is necessary in comparison to the time stamp of the now older destination file.

The property that copied files get a new timestamp is a Unix/Linux property. In Windows copied files get the timestamp from its source, the old one. It is a philosphy what is better. Some people (inclusive me) means, keep the old timestamp while copying is the better one. The timestamp is an interesting property of a file. If I see a file with timestamp 2003-... or 1997-... I know it is an older one. Often I know the context of history of the file viewing its timestamp. Files can be sorted in a file viewer by timestamps. If copied older files lost its time stamp information I don't see what's happen. I should look into the files, is there a version information? Searching with eyes, not sorting automatically, effort. Of course a file version management system can show all that information, but ...

Most of maker tools does not work properly if older files are copied with its older timestamp. They don't detect the necessity of translation. Therefore 'make all' is the decision often in practice. But 'make all' runs a long time. The second goal of make is absent. A maker is degraded only to a simple organisator of execution.

There is a second problem of the second of the three points: Detection of necessity of translation because a changed indirect depending source: Usual the dependency of the destination from sources are written in a make file in form:

destination: source1 source2 source3

The make script file should name the dependency explicitely. But in praxis C-sources includes headers, the headers includes other headers, there are dependent. But nobody corrects the dependency line in the makefile, because it is sophisticated often.

If a make process 'forgets' the compilation of a C-file because the not known dependency to a changed header, an obscure behavior can be occured which is difficult to detect and debug (data addresses are shifted, wrong data, addresses of virtual methods are shifted, it calls the wrong methods).

Because the non-detected dependencies the decision is again: 'make all'.

A third problem is given if the sources of the make process are generated (second sources). That is because any superior development tool is used mostly for graphic programming. The generated source have a new timestamp though they do not may have a change content. In actual fact the translation is not necessary, but the fact cannot be detected by a maker tool. Therefore all is maked with high execution time effort.

It means, the second goal of a maker unprofitable. A standard maker is degraded to a tool which is a simple execution control script only with a special sophisticated syntax.

Topic:.ZmakeGen.zmakeGoal.

Zmake in its new form is only a simple execution script written in JZcmd syntax.

But the second goal of make: checking the necessity of translation by timestamp check with dependency check can be done inside the translation execution routines with special convenient sub tools. The simplest one is: Comparison of time stamp of 2 files for conditionally execution:

if(FileSystem.checkNewless(fileSrc, fileDst, 1) > 0 ){
  execute...
}

With 1 as third parameter the fileDst will be removed to force translation by checking the existence of the file in a later executed step:

if( not fileDst.exists()) {
  makeit...
}

Both problems, the complex dependency check and the detection of changed or non-changed generated (second) sources is resolved by the tool CheckDeps_C which can invoked inside the JZcmd zmake script.

The approach with Zmake is: Development without Zmake in an IDE (Visual Studio, Eclipse) with a test environment and debug possibilities; and deployment for a target system using Zmake. Compiler and linker errors are shown immediately from the compiler output. Because the sources are tested before, only basicly compiler errors are able to expect. For example because a faulty include path. The C(++) sources for test on PC with the IDE and to deploy for the target are the same. The deployment-compilation via Zmake is more simple. It can be more simple adapted to target compiler conditions.

Because of Zmake generates the ready-to-use batch file for compilation you can better see what the compiler is intent to do on invocation (which sources, for any source, which options). It is better to understand in comparison to the often cryptic code of a standard make file.

Topic:.ZmakeGen.ZmakeUserScript.

Last changed: 2015-03

The idea of separation of ZmakeGenCtrlScript from the ZmakeUserScript may be a good but not necessary decision. It is possible to have only one script. The ZmakeGenCtrlScript is included.

ZmakeUserScript: ===================JZcmd============+=======>ZmakeExecutionScript:
Keep the users make                   ^              |        Generate the ready to use
control file simple.                  |              |        make file to execute
                             ZmakeGenCtrlScript:     |
                             Use a second file,      +=======>Execution direct
                             which controls details           (Java methods or cmd line)
                             of the make process.

Topic:.ZmakeGen.TmakeUserScript.

Last changed: 2015-03

For an example see examples_JZcmd:zmake-compile/exampleCcompile.jz.bat

The script starts with call of JZcmd. In this case it is assumed that a Windows-batch-file or unix shell script jzcmd can be found in the PATH. The jzcmd script gets the name of the file itself as first parameter.

jzcmd examplemakefile.jz.bat -t:textout.txt

Alternatively a jzcmd.bat can be located with relative path in the environment. For content of jzcmd see Topic:.JZcmd.invoke.cmd.:

call ..\..\myTools\jzcmd.bat examplemakefile.jz.bat -t:textout.txt

In this case the tool base for JZcmd is found in the local context.

pause
exit /B

That are Windows cmd statements to prevent closing the command line window.

==JZcmd==

After that marker the JZcmd script starts.

include $ZBNFJAX_HOME/zmake/genXMI_exec.jzcmd;
include ../../myTools/myZmake.jzcmd;

It is possible to include scripts, which are ready to use for the Zmake process. Some scripts are given in the zbnfjax environment. Some scripts may be user specific. The includes scripts will be parsed and translated in the time of execution of this script.

currdir = scriptdir;

Usual the current directory used for absolute file paths is set to the directory where this scriptfile is stored.

currdir = <:><&scriptdir>/../..<.>;

It is possible and recommended that the current directory is set to a parent location especially if the make script is stored in a sub directory of the local working area in the file system. It is recommended to use relative paths.

Fileset mySources =
( srcpath/file1.ext
, srcpath2/*.c
);

A Fileset determines some files which are used for input. There may be more as one Fileset for some operations.

String paramXyz= "-flags -options";

Some parameter for some stuff.

Filepath objfiles = "test/obj/*.obj";

It is a path and the extension for obj-files for compilation, for example.

The script can define all zmake targets in its main routine. Of course it can be written in a sub routine too, which is called from main().

main(){
  call openDependencyCheck(
    cfgfile = Filepath "cfgCheckDep.cfg"
  , depfile = Filepath "dependencies.txt");

  call makeExample();

  call closeDependencyCheck();
  <+><.+close>;                    ##closes the text output
  cmd cmd /C _gen.bat;
}

In the main the openDependencyCheck-sub routine is called. A configuration file with include paths and the dependency-file is given as arguments. The dependency-file is created firstly or it is used for check of timestamps and dependencies see CheckDeps_C

The main routine can containt the zmake statements immediately. It is a better style to call a subroutine.

The closeDependencyCheck() writes the current situation of timestamps of all files and dependencies after processing all checkDeps.processSrcfile(...) in the zmake generation script.

At last in this pattern the generated batch-file is executed in the JZcmd script. Alternatively it can be executed in the batch part of this file or it can be invoked in any other environment.

The zmake lines are written in a sub routine:

##One of the zmake user routines.
sub makeExample(){
  zmake dstpath/file.ext := makeroutine(&mySources, paramx=paramXyz);
}

zmake is the invocation of a zmake target. The makeroutine should be found in the included scripts. There can be more as one zmake call, some other JZcmd statements, sub routines etc.

Syntax for zmake see Topic:.JZcmd.syntax.zmake.

See examples ../examples_JZcmd/Java2Xmi/readme.html, ../examples_JZcmd/Cheader2Xmi/readme.html, examples_JZcmd/zmake-compile/readme.html

That is all for the zmake user script. It may be more complex if the task is more complex.

Topic:.ZmakeGen.ZmakeGenScript.

Last changed: 2015-03

The ZmakeUserScript can contain the initializing of Checkdeps, in the pattern above the call openDependencyCheck(...) This subroutine and the close routine is programmed here. A variable checkDeps is defined on script level in the include script.

Obj checkDeps = new org.vishia.checkDeps_C.CheckDependencyFile(console, 1);

sub openDependencyCheck(Filepath cfgfile = "cfgCheckDep.cfg", Filepath depfile = "dependencies.txt"){
  checkDeps.readCfgData(cfgfile.absfile(), currdir);
  checkDeps.readDependencies(depfile.absfile());
}

sub closeDependencyCheck(){
  checkDeps.writeDependencies();
  checkDeps.close();
}

The included JZcmd script should contain the requested zmake routines.

sub makeroutine(Obj target, String paramx = "default"){

The target is an instance of javadoc-vishiaBase/org/vishia/cmd/ZmakeTarget which contains the input FileSet (maybe more as one) and the output Filepath which is given in the zmake ...-line. The paramx is an example for a second parameter of this routine with a default value which can be given in the zmake line.

Usual some file should be handled. The next line gathers all given files of the target's input in a List<File>;

  List inputsExpanded = target.allInputFilesExpanded();

Sometime anything should be done with each input file per file. Therefore a for statement of JZcmd can be used. First the showing of the filename in the stdout is programmed.

  for(input:inputsExpanded) {
    <+out><&input.localfile()><.+n>   ##show in cmd line window
    //some statements for the file

If the input file should produced an output file only if it is changed the checker for dependency is used. It works for C/C++-sources. That routine deletes the object file. Therefore it can be ask whether a compilation is necessary by testing the existence of the object file:

    checkDeps.processSrcfile(input.absfile(), input.localfile(), objDirExt.absdir(), ".obj");

The routine nextNr is a special feature of JZcmd and produces an unique label for a windows batch file. The objname builds a proper String which is used more as one in the next statements:

    String nextNr1 = nextNr;   ##provides an incremented number
    String objname = <:><&objDirExt.absdirW()>\<&input.name()><.>;

The next statements are written in the JZcmd output text file, not executed. They are executed after the text output is generated in a system's batch invocation. It is an example for MS-Windows. Adequate shell scripts for UNIX may be written.

    <+>
      REM <&objDirExt.absdir()>
      if exist /out:<&target.output.absdirW()>\obj\<&input.name()>.obj goto :<&nextNr1>
      cl.exe /c /TP <&incPath> <&input.absfileW()> /Fo<&objname><&objDirExt.ext()> 1><&objname>.log 2><&objname>.err
      :<&nextNr1>
    <.+>

In the batch file the existence of the object file is tested. The compiler, in this example from MS Visual studio, is invoked conditionally.

The separation of execution with the JZcmd script and with a generated script file is a possibility, not a necessity. It have some maybe advantages and features:

• It may be required that a generation process should be done by an invocation of a batch file, a maker etc. by an superior environment, for example for graphical programming. Therefore such an script is produced by JZcmd.

• The test of dependency and newless by invocation of checkDeps.processSrcfile(...) can be done in the batch file too, whereby the Java virtual machine is loaded in the batch script or maybe from a maker because it is given as Java class. The class javadoc-Zbnf/org/vishia/checkDeps_C/CheckDeps supports dependency check for a whole source tree or a list of given files. It the dependeny check is necessary in another script it can be used.

• In the showed from of this pattern the generated script generates all if the objects are removed anyway. It may be removed with a del command. Then it is a make all. For this the Java Virtual Machine is not necessary. The generated script can be used without necessity of Java if it may be necessary.

A compiler can be invoked in the JZcmd script immediately. The invocation of compile creates an own Process on the system. One should write for example:

   String msg = cmd cl.exe /c /TP &incPath &input.absfileW() <:>/Fo<&objname><.> objDirExt.ext();
   if(msg >= "error"){
     <+>
       Compilation error for <&input.localfile()>
       <&msg>
     <.+>
   }

That form has the advantage, that an evaluation of stdout or any other messages can be done with the capablity of JZcmd. That's some more as in batch files.