Some videos, also linked in the text:

An example to download and test:

Dr. Hartmut Schorrig, www.vishia.org, 2021-06-21

1. Motivation

Often tests are controlled by scripts, or the test environment (test bench) is any source containing the test conditions. For different tests more as one scripts are necessary which should be select to use.

But the scripts are equal in wide range, only specific parts are adapted to different test cases. Hence it is better to generate the yet necessary test script from a template with place holder and replace the place holder with the correct values.

The ../../JZtxtcmd/html/JZtxtcmd.html is used to generate the scripts. A GUI environment build with the vishia Gral concept is used for selection of parameter of generation.

This concept is flexible to use for all approaches where scripts should be generated to control tests or also parameter etc. The adaption is based on script files.

The tool presented here needs two specific jar files (Java binaries) with ~ 3 MByte and the standard SWT Java binary for graphic, and some textual script files, no more. It is able to integrate with given solutions, both in Java environments and as a command application (stand alone executable).

2. Usage - overview

2.1. Principle: Text template with placeholder

Supposed, a script is necessary to control the test. This script may be:

  • A main program or function in a source which is a test bench for a module in C/++ language.

  • A header file which contains some compiler switch definitions.

  • For Simulink: A math script or text file with parameter

  • FPGA-simulation: A VHDL file which is the test bench

  • A XML file or yaml file which contains parameter of test

  • A flow written in Phyton which controls a test

  • …​ somewhat else, a textual script

For a simple explanation a simple test file with parameters and a XML file is presented here:

#Simple test script with parameter
title = "t1_v1"
value1 = 123.45
value2 = "text"

Adequates in XML:

<?xml version="1.0" encoding="ISO-8859-1"?>
<Test:root xmlns:Test="https://myUrl/TestDefs" Test:title="t1_v1">
  <Test:Values Test:value1="123.45" Test:value2="text" />
</Test:root>

The content of this files are an example, it is user specific.

You need a file which contains exactly this texts but with place holder for the values. The place holder are written in the form: <&path.variable>. But additionally a frame for the text is necessary. So the template for the text file for this example is (file testfile_text.jzTc):

==JZtxtcmd==

##
##Template for the whole testfile_text.
##arguments values and texts comes from the selected lines of the Test stimuli
##
subtext testfile_text(String title, Map values, Map texts)
<::>
#Simple test script with parameter
title = "<&title>"
value1 = <&values.value1>
value2 = "<&texts.text>"
<.>  ##

The area between <::> and the closing <.> is the whole original text with the placeholder. The text is organized in JZtxtcmd as 'subtext' syntax, with calling arguments, see ../../JZtxtcmd/html/JZtxtcmd.html#Topic.JZtxtcmd.script.sub.. The template script can also contain comments written as ##line end comment which are not part of the generated result. A simple # is not a comment because often the # is need to generate comment lines. Of courses that requirement or feature is regarded in JZtxtcmd.

It is also possible in XML (file testfile_xml.jzTc):

==JZtxtcmd==

##
##Template for the whole testfile_xml.
##arguments values and texts comes from the selected lines of the Test stimuli
##
subtext testfile_xml(String title, Map values, Map texts)
<::>
<?xml version="1.0" encoding="ISO-8859-1"?>
<Test:root xmlns:Test="https://myUrl/TestDefs" Test:title="<&title>">
  <Test:Values Test:value1="<&values.value1>" Test:value2="<&texts.text>" />
</Test:root>
<.> ##

JZtxtcmd is proper also and especially for generation of results which uses elaborately writing with < …​ > for example XML, but also some other script languages e.g. the Mathworks-specific "tlc" scripts ("target language compiler" control script). The distinction to XML style is: The placeholder start with <& which is never used in XML as necessary direct text. Some control constructs start with <: also not used in XML. So that sequences can be used immediately in the template. But nevertheless this character sequences can be written as <:<&> etc. to produce it as output, see ../../JZtxtcmd/html/JZtxtcmd.html#Topic:.JZtxtcmd.text.transcript..

2.2. Tables with the values for placeholder, test variants

From where come the values?

You should define tables, (see file stimuliTables.jzTc in the appended example):

==JZtxtcmd==

List values @name =
[ { name="v1", descr="test-var 1", value1="123.45", value2="536.5" }
, { name="v2", descr="test-var 2", value1="345.67", value2="5" }
, { name="v3", descr="test-var 3", value1="987.65", value2="1000" }
];

List texts @name =
[ { name="t1", descr="text1", text="text" }
, { name="t2", descr="text2", text="other text" }
];

The List is type of ../../Java/docuSrcJava_vishiaBase/org/vishia/cmd/JZtxtcmdExecuter.ListMap.html. It can also be accessed as Map with a key. The variable which’s content builds the key is designated after the @ in the list’s head.

Any line of this tables is a java.util.Map container with the name of the variables as key. The line should contain a variable named descr for the GUI tool. Furthermore any variables are admissible holding values. But each line of a table should contain the same named variables.

The tables build groups of selectable test cases, which can be used for manual selection, and also for test generation.

2.3. The GUI for test case selection

The GUI with this given tables looks like:

TestStimulator 5Tables

The tables in the table script are shown and are selectable. One can select a specific test case with this tables and press [gen selection] to get the generated files for this case.

It is possible to assemble some test cases and press [gen test cases] to generate all files for all tests, or to control execution of test.

2.4. Integration in a given test environment

For now the tool creates only the textual files. If you want to use the tool independent of your test system, you can use the generated files in the test system.

But it is also possible to interact with the test system. This is done with the capabilities of the JZtxtcmd script language, or also with batch files. You can generate a script which invokes executables by itself or sends and receives messages for example via socket communication. This depends only on your design of the scripts.

The evaluation of test results is not a functionality of the Stimuli Selector. But the scripts can help for evaluation.

3. Example as zip file

This documentation refers an example given as ../download/StimuliSelect_Example.zip able to get as link from here. The example contains

TestStimulator
 +-libs
 |  +-bomVishiaJava.txt
 |  +-vishiaMinisys.jar
 |  +-+load.bat
 |
 +-stimuli.jzT.cmd
 +-stimuliTables.jzTc
 +-testfile_text.jzTc
 +-testfile_xml.jzTc
 +-nextStimuli.bat

The libs directory should contain the necessary jar files. This is:

2021-06-17  22:47               822 bomVishiaJava.txt
2021-01-04  09:35         2.445.585 org.eclipse.swt.win32.win32.x86_64.jar
2021-06-13  23:21           218.624 socketcmd.exe
2021-06-17  22:41         1.279.231 vishiaBase.jar
2021-06-18  09:41         1.121.551 vishiaGui.jar
2020-12-26  23:29            79.436 vishiaMinisys.jar

But this jar files and the executable are not contained in the zip file itself, instead they are gotten from its repository in internet ../../Java/Download/versionArchive. The file bomVishiaJava.txt contains all information about the repository path, file name and a MD5 check sum. The vishiaMinisys.jar contains the program to load (it is less, runs also in Linux) and +load.bat invokes the loading or checks the files with MD5 if they are loaded. You can also use a newer version if necessary, only the bomVishiaJava.txt should be renewed. That is possible also manually. You can open the repository by yourself and look to newer versions, or have a info about.

The sources of the jar files are also contained in the repository beside the jar, including a compilation file. The compilation is designed as 'reproducible build' ../../Java/html/source+build/reproducibleJar.html

The amount of Megabyte is less. The tool can be integrated in any other Java environment. Yet Java-8 is used, but compilation to a newer version should not a problem with the given sources.

The other files are explained in the following test. The example in a matter of principle, not an useable example. But it should be a proper template for own usage.

4. The GUI for test case selection

This GUI is a Java program using Eclips-SWT as graphic driver. It is invoked with (file stimuli.jzT.cmd):

REM starts as windows command (batch) file:
if not exist genScripts mkdir genScripts
set LIBSPATH=./
set CP=%LIBSPATH%/libs/vishiaGui.jar;%LIBSPATH%/libs/vishiaBase.jar
REM comment out swt.win32_x86_32.jar or swt.win32_x86_64.jar
::set CP=%CP%;%LIBSPATH%/libs/org.eclipse.swt.win32_x86_32.jar
set CP=%CP%;%LIBSPATH%/libs/org.eclipse.swt.win32_x86_64.jar
REM hint: use java if something is wrong, to see outputs,
REM       use javaw for normal usage without back cmd window
set JAVAW=java
echo dir=%CD%
REM call the GUI. This file %0 is used as argument for SimSelector.
REM It contains all control after the JZtxtcmd label
echo on
%JAVAW% -cp %CP% org.vishia.stimuliSelector.StimuliSelector %0 -size:C
echo off
pause
exit /b

The class creating the GUI is the ../../Java/docuSrcJava_vishiaGui/org/vishia/stimuliSelector/StimuliSelector.html. The first argument is this file itself (%0 is the full path in the Windows batch), used as JZtxtcmd script for organization of the GUI.

The GUI with this given tables looks like:

TestStimulator 5Tables

The same example file stimuli.jzT.cmd contains also the generation sub routine, the routine for the [gen selection] button and the association to the tables:

==JZtxtcmd==

currdir=<:><&scriptdir><.>;

include stimuliTables.jzTc;
include testfile_text.jzTc;
include testfile_xml.jzTc;

The designation ==JZtxtcmd== on start of a line marks the content as JZtxtcmd script inside this given *.cmd file. The exit /b as last statement above ends the usage as cmd. The other files are included here.

sub btnGenSelection ( Map line1, Map line2, Map line3, Map line4, Map line5, Map line6) {
  <+out><&scriptdir>/<&scriptfile>: btnGenSelection (
     <&line1.name>, <&line2.name>) ..... <.+n>;
  call genTestfiles(values=line1, texts=line2);
}

This is the JZtxtcmd sub routine which is searched and used by the GUI for the [gen selection] button. The arguments are the selected lines in the possible 6 tables. The sub routine calls the generation routine, the commonly defined arguments are associated to the application specific names of the generation routine. This routine is in the same form for all usages

The next sub routine should be adapted to the necessary files for the test. This is the example which generates the above shown files testfile_text.txt and testfile_xml.xml:

##
##This is the generation routine for one test case,
##either for manual [gen selection] or used for [gen test cases]
##
sub genTestfiles(String name = "testfile", Map values, Map texts) {

  String title = <:><&texts.name>_<&values.name><.>;     ## build the title
  mkdir genScripts;
  String sfText = <:>genScripts/<&name>_text.txt<.>;
  Openfile fText = sfText;
    <+fText><:call:testfile_text : title=title, values=values, texts=texts><.+>
  fText.close();
  <+out>gen: <&sfText><.+n>

  String sfXml = <:>genScripts/<&name>_xml.xml<.>;
  Openfile fXml = sfXml;
    <+fXml><:call:testfile_xml : title=title, values=values, texts=texts><.+>
  fXml.close();
  <+out>gen: <&sfXml><.+n>
}

First a title for the test is built with the short given names in the lines, which are also the keys for selection. This title is used as argument for the templates.

The two expected files (example) are generated. Openfile opens the named file and offers the java.io.Writer. <+fText>…​ writes to the file. <:call:…​> invokes execution of the subtext which contains the template text with place holder. That’s all for this example.

The GUI should know which tables should be used to show and select. This is contained in the included file stimuliTables.jzTc. Two of the tables are shown already in the introducing chapter Tables with the values for placeholder, test variants.

The StimuliSelector.jzT.cmd contains which are the name of tables to show. In the so named class in JZtxtcmd the association to the tables is named. The GUI java program searches this class and the tdata.. variables.

##
##This class defines which tables should be used in the StimuliSelector GUI
##
class ToGui
{
  List tdata1 = values;
  List tdata2 = texts;
  List tdata3 = var_A;
  List tdata4 = var_B;
  List tdata5 = var_C;
}

With 6 tables test cases with 6 coordinates to modify tests (builds combinations) are possible. The GUI can be enhanced for example to use 12 coordinates or combination variants, using 4 x 3 tables, obviously on a normal monitor, or more. But in practical usage 6 coordinates seems to be enough.

5. Manual selected and automatic tests

Firstly this GUI and the tables support manual selected test cases in several combinations. This example is held simple, but some combinations should be imaginable also for this example.

With 6 combinations of for example 10 entries per table a lot of combinations are possible (10^6 = 1000000, only for example).

5.1. Manually selection of test cases and generate

It means, before establish automatic test cases, reasonable combinations should be found. Manual tests with expertise of the results can help to find the combinations.

You should think about what is to test, think about which combinations are proper, and get experience with some combinations. It is gathering experience with manual handling.

Hence you can select one combination with the tables, press the button [gen selection], maybe improve the test files, study the behavior of your system in test, and also complete the tables and the characteristics of the tables.

5.2. Test case selection

The idea is: Assembling some test cases on demand and study manually the behavior. Then gather the test cases in a expression:

TestStimulator Sel1

If you have a test selected in the tables and you click [add sel] button then you get a select expression in the text box right of [gen test cases]:

1=v2; 2=t1; 3=A1; 4=B1; 5=C1;

This expression contains table numbers and associated keys of the lines of test cases.

In the example secondly the shown case v3 was selected in the table, and then pressed [add sel] again. Then this case is also added. If you select other lines in the other tables and click [add sel] this lines are added too. You can check some test results manually pressing [gen selection], then [add sel]. But:

1=v2, v3 ; 2=t1, t2; 3=A1, A2, A3; 4=B1, B3; 5=C1, C2, C3;

This expression means, this two cases from table 1 are combined with the two cases from table 2 and the other selections in the tables. It describes 2*2*3*2*3 = 72 test cases, the combination of all. If the tables are longer, it is sometimes too much. Hence there are more possibilities:

1=v2, v3 ; 2=t1, t2; 3=A1; 4=B1; 5=C1;
: 1=v1 ; 2=t1, t2; 3=A1, A2; 4=B1; 5=C3;

That are two independent combinations, separated with the : colon. It is 2*2 + 2*2 = only 8 test cases. You may have the expectation that you want to test the v1 combination only with the given ones, for example only with C3 and the other cases from table 1 especially with C1 only. - to reduce the amount of combination for a fast overview test.

You can entry this expression manually, but also write the colon, :, set the cursor after it, select in table and press [add sel]. The expression builder detects that you are in a separated part of the expression. Because it is empty, it takes all lines of the table.

The test case select expression allows skillful combinations, some selections with some others. See the next expression:

1=v2, v3; 4=B1, B3; + 1=v1; 4=B2
& 2=t1,t2; 3=A1; + 2=t2; 3=A3;
& 5=C1

The first line dedicates 4 cases in combination v2, v3, B1, B3 and a combination v1, B2. The + means a 'add' of a case.

The & means 'and with', a selection in other tables to build a combination. Here the 5 combination above with table 1 and 4 are combined with the second line, they are two combinations with t1 and t2 and A1 and a third one with t2 and A3, In the manual or considered tests this combinations may be proper.

Last not least this is combined & 'and with' the C1 of the 5th table.

The expression can be written in one line or more, white space is the principle

In result the following tests are combined:

test case: v2_t1_A1_B1_C1
test case: v3_t1_A1_B1_C1
test case: v2_t1_A1_B3_C1
test case: v3_t1_A1_B3_C1
test case: v1_t1_A1_B2_C1
test case: v2_t2_A1_B1_C1
test case: v3_t2_A1_B1_C1
test case: v2_t2_A1_B3_C1
test case: v3_t2_A1_B3_C1
test case: v1_t2_A1_B2_C1
test case: v2_t2_A3_B1_C1
test case: v1_t2_A3_B1_C1
test case: v2_t2_A3_B3_C1
test case: v3_t2_A3_B3_C1
test case: v1_t2_A3_B2_C1

To fill such more complex expression you have the following possibilities:

  • Write manually, you see or know the key names, it is not complicated.

  • Set the cursor in a proper part of the select expression and double click a line in a table, this line will be added on the correct position.

  • Set the cursor to the desired parth, select a case in tables and press [add sel]. Then only the lines will be added of that tables which are already part of the expression part. This helps to select in groups 'and with …​' after the ampersand &.

  • Mark lines in tables and press [add sel]]. To mark lines you must hold 'shift' and press 'arrow down'. Mark with mouse is yet not possible (2021-06), may be possible later depending of the capability of the "GRAL" Graphic Adaption Layer. Then all marked lines are written in the part. Tables without marking are not considerate. You can also mark only one line to add only this line. But this is the same as double click a line.

5.3. Syntax of the test case expression

The syntax for the select expression is:

select::= { <selAnd> ? : }.    (1)
selAnd::= { <selAdd> ? & }.    (2)
selAdd::=  { <selLines> ? + }. (3)
selLines::= { <#table> = {<$?selItem> ? , } [;] }. (4)

It is written as ZBNF expression. It is similar BNF or EBNF. {…​? .. } means a repetion with separator after the ?. <#?table> is a number with the meaning (semantic) "table". <$?selItem> is an identifier (letters, digits and underlyer) with the given semantic meaning. […​] is optional, hence the semicolon is optional. But it should be written. The ZBNF syntax has the advantage that semantic information are given in the formal syntax string ("Zemantic BNF").

Look on the example:

1=v2, v3; 4=B1, B3; + 1=v1; 4=B2
& 2=t1,t2; 3=A1; + 2=t2; 3=A3;
& 5=C1
  • 4) Should come to know from inner to outer. You see this expression in the image:

    1=v2, v3; 4=B1, B3;
  • It means that the table line with key v1 and v3 from table 1 are selected to test, in combination with B1 and B3 from the table 4.

  • 3) It is possible to have more independent combinations, separated with + - means add selection combination. In the image you see a second combination which should use to test:

    + 1=v1; 4=B2
  • 2) The & means 'and with'. It is a selection with other tables which are combined with the given combinations. Of course you may have here also more combinations separated with +. The example

    & 2=t1,t2; 3=A1; + 2=t2; 3=A3;

uses the + also here. It means 3 cases: t1_A1, T2_A1 and T2_A3.

  • 1) The colon separates independent combinations.

6. Test case generation

The button [gen test cases] calls the sub routine

##
##This routine is the button routine for the [gen testcases] button.
##
sub btnGenTestcases(String select) {
  ##....
}

The content of this routine depends on the requests of the test system. They are different possibilities, see chapter Interaction of this Stimuli Selector with a test system. But the anyway used functionality is: evaluation of the select expression.

6.1. select String evaluation and choice of lines in the tables with the values

The example contains three variants of using the selection. You have:

##
##This routine is the button routine for the [gen testcases] button.
##
sub btnGenTestcases ( String select) {
  call btnGenTestcases_A(select=select);
  ##call btnGenTestcases_A(select=select);   ##generate all files with different names
  ##call btnGenTestcases_M(select=select);   ##use socket messages
  ##call btnGenTestcases_F(select=select);   ##use file semaphores
}

Change the called routine to …​_A as shown then the following is called:

##
##This routine is the button routine for the [gen testcases] button
##for generation all files.
##
sub btnGenTestcases_A ( String select) {
  Obj testcs = java org.vishia.testutil.TestConditionCombi.prepareTestCases( select, 5);
  for(testcase: testcs ) {
    String name = <:><:for:var:testcase><&var.sel><:hasNext>_<.hasNext><.for><.>;
    <+out>test case: <&name><.+n>
    Obj lineValues = values.get(testcase[0].sel);
    Obj lineTexts = texts.get(testcase[1].sel); ## generates the files for this case:
    call genTestfiles(name = name, values = lineValues, texts = lineTexts);
  }
}

This routine generates for all test cases a set of files with different names. The name for the generated test file(s) is build from the table content. Calling the same routine as in [gen selection] the files are generated.

This routine can be seen as principle pattern:

  • First the Java routine ../../Java/docuSrcJava_vishiaBase/org/vishia/testutil/TestConditionCombi.html is called. It is part of the vishiaBase.jar. This routine analyzes the select string and outputs a simple List, here testcs. Each item of the list (testcase in the for loop) is also a list. Each item of this nested list (testcase[0] etc. or var in the for-loop to build the name) contains the table number in nr and the key value of the line in sel.

In the example you see the access to the line of the table values for the test case using the key testcase[0].sel, adequate lineTexts = texts.get(testcase[1].sel) for the other line of table.

Alternatively you can generate also a file as organization frame for all tests with content controlled by the test cases.

6.2. Example generate a test frame only, and generate the current test scripts inside the test tool

The next example comes from a Simulink usage, not from this example. See https://www.vishia.org/smlk/html/SmlkTimeSignals/SmlkTimeSignals.html.

sub btnGenTestcases(String select) {
  String name="genTestCases";
  ##....
  <+out><&scriptdir>/<&scriptfile>: genTestcases(<&select>, <&name>) .....<.+n>
  String ffcases = <:><&dirMdl>/+genstimuli/<&name>.m<.>;
  Openfile fcases = ffcases;
  <+fcases>
  <:>
==
==%%Fills the array of test cases
==clear testcases;
==ix = 1;
==<.><.+>
  Obj testcases = java org.vishia.testutil.TestConditionCombi.prepareTestCases(select, 6);
  Num ixcase = 1;
  for(testcase: testcases) {
   String name = <:><:for:var:testcase><&var.sel><:hasNext>_<.hasNext><.for><.>;
   <+out>test case: <&name><.+n>
   <+fcases><: >
   <:>
===testcases{ix, 1} = '<:for:var:testcase> -u:<&var.sel><.for>'; ix = ix +1; <.><.+>
    ixcase = ixcase + 1;
  }
  <+fcases>
  <:>
==
==for ix = 1 : size(testcases)
==  disp('====================== Prepare Simuli file =========================');
==  args = testcases{ix};  %curly braces
==    %The next script generates the stimuli file.m with the given args using this script too
==    cmd = strcat('test\Smlk\+inspcStimuli\+stimuli\createStimuli.jzT.cmd', args);
==    disp(cmd);
==    system(cmd);                            %generates genStimuli.m
==    disp('====================== START Simulation ==========================');
==    sim('+inspcStimuli/TestInspcStimuliMdl.mdl');
==
==end
  <.><.+>
  fcases.close();

Result of this generation is a Matlab script file. You see the content with place holder after the == on start of lines. The generated Matlab script file organizes while running (if the test is started) the creation of a Matlab array. This array contains only a String which is used as arguments for test script generation (line ===testcase{ix, 1}=…​). For example it is generated as Matlab script:

clear testcases;
ix = 1;
testcases{ix, 1} = ' -u:fast -u:mid1 -u:ramp -u:Filter_1 -u:x -u:x'; ix = ix +1;
testcases{ix, 1} = ' -u:fast -u:midup -u:ramp -u:Filter_1 -u:x -u:x'; ix = ix +1;

This is done for all test cases as result of preparing with TestConditionCombi.prepareTestCases(select, 6);

After that action the generated Matlab script contains a for loop for all elements of the Matlab array. The for loop is executed in the generated Matlab script. It does the following:

  • Invocation of the test script generation for the current case, using the content in the Matlab array. For that the script createStimuli.jzT.cmd is called from Matlab, in the Matlab script. The execution command is system(cmd). This prepares the scripts for the current test.

  • After them via sim('Modelname') the simulation with this scripts is started.

The createStimuli.jzT.cmd generates the current used test script via JZtxtcmd (Java call) as command called inside the Matlab script. This file contains:

echo off
REM call of this file generates one stimuli file with the given settings.
REM Settings are cmd line arguments written -u:<&line1.name> ...etc for JZtxtcmd

REM Note: Invocation from its own directory or from the simulink root dir.
REM cd to the simulink root dir if necessary
if exist mdlStimuli_SelectionTool.jzT.cmd cd ..\..\..\..

REM The JZtxtcmd start file is not this, it is mdlStimuli_SelectionTool.jzT.cmd,
REM because that file includes some more and also this.
REM Argument writing style is -u:KEY to designate as user argument for the main ()
java -cp ../libs/vishiaBase.jar org.vishia.jztxtcmd.JZtxtcmd ...
  ... test/Smlk/+inspcStimuli/+stimuli/createStimuli.jzT.cmd %1 %2 %3 %4 %5 %6

exit /B



==JZtxtcmd==

 include Stimuli_template.m.jztc;       ##the template for the simulink stimuli file
 include SetInspcArgs_template.jzTc;    ##the template for Inspc stimuli
 include Stimuli_SelectionTool.jzT.cmd; ##contains the tables for selection

 currdir = <:><&scriptdir>/../../../..<.>; ##it is the root of the Smlk working tree



 ##
 ##This main routine is used to generate the stimuli file.m from the Simulink simulation m-file
 ## the arguments are the keys sorted to the tables.
 ##
 main ( ) {
   <+out>stimuli_scripts/createStimuli.jzT.cmd: main(<&$1>, <&$2>, <&$3>, <&$4>, <&$5> )<.+n>
   call genStimuli(key1 = &$1, key2 = &$2, key3 = &$3, key4 = &$4, key5 = &$5 );
 }

The first lines presents a batch file (cmd file). Java is called. It ends on exit /B.

The next lines after ==JZtxtcmd== contains a main() routine which is invoked firstly. This main routine gets the keys which are comes from the Matlab array as 'user arguments' written as -u:key generated via -u:<&var.sel> in the script above.

It is a nesting of some scripts, which allows using the capability of test organization in Matlab (with an array for arguments and possibility of a system(cmd) call, and then start the simulation via sim(model).

The details how the genStimuli(…​) works are not presented here, see link above. It is a normal file generation using the template with placeholder approach.

7. Interaction of this Stimuli Selector with a test system

If the test system is also programmed in Java, i.e. during test execution a continuing Java program controls the tests, then it should be possible to integrate this Stimuli Selector directly into this test system.

For that the test solution does only need this here used two jar files, it can call the adequate Java classes in a proper specific way.

But often a test system is programmed using any other script language, or the effort to integrate this Stimuli Selection tool into the test system seems to be too high. Another approach is: The Stimuli Selector should possibly run in an extra process, as a separate window, or even on another computer in the network. This is especially the case if the test system runs on an embedded platform (with network capability).

Therefore, the question of interaction between the two should be clarified in more detail. There are some approaches:

  • a) With the select String some files are generated which are used from the tests afterwards.

 select expression [gen test cases]
 ----------------------+-----------
                       |
                       +-> generates files
                           ------------+--
                                       |
                                       +-...-> execute the test independent
  • a1) It is possible to generate all test files.

  • a2) It is possible too to generate only one file for control the test. The test files with data can be generated out of the StimuliSelector via calling the given generation scripts via JZtxtcmd invocation. The invocation of JZtxtcmd execution is an invocation of Java usual via command line. Often test tools can call command line statements, then it is possible. The time for execution (initialization of Java, translation of classes, etc.) is not so high, it is usually much less than 1 second, less compared to the test times.

 select expression [gen test cases]
 ------------------+---------------
                   |
                   +-> generates the test control files
                       ------+-------------------------
                             |
    while test execution:   uses this control file
    Using JZtxtcmd           |
    but independent of StimuliSel: -> generates the test files
                                      ----+-------------------
                                          |
                                          +-...-> execute the test
  • b) The following shown approach works with interaction between a test system and the Stimuli Selector GUI. The test system should have the capability

    • either of calling a cmd for the operation system in its execution loop.

    • or it should able to check the existence of files, and a file transfer and also copy or rename should be possible.

    • or the test system should able to exchange messages preferred via socket (Ethernet) communication or maybe also serial.

One or more of this features are often available.

The Stimuli Selector GUI can deal with files in the network, can communicate via ethernet but not yet via serial (UART). But this is possible to enhance.

The StimuliSelect_Example shows two approaches, handle with files and socket. Wherby the test system does not handle with sockets by itself but uses the SocketCmd.exe to do so.

7.1. Possibility to interact via socket - or serial

Socket communication is an elegant approach to data exchange. A simple string message can be sent and received with a UDP telegram. The queue to store messages is already given with the 'Telegram Stack' on driver level.

However, if the queue is to work properly, the socket communication should be initialized once at startup (open socket) and remain open. Otherwise, the partner can send a message and only after that the communication will be opened. Then the message is lost. This can occur if the test system does not have the socket communication approach natively, but should execute it via a command line call. If the socket cannot remain open for the entire time, it should be timed.

The socketCmd.exe given in the example can be used if the test system has no native socket support but can call system commands. The problem for remain opening is solved in the following way:

  loop in testsystem             Stimuli Select GUI
   |                              +-open socket after [gen test cases]
   |                              loop in gen test cases thread:
   +---> socketCmd.exe               +-wait for receive
            +->open Socket           |
            +->transmit Cmd ========>| accept receive,
            +-wait for receive       +-prepare test files
            | accept receive <====== +-transmit Cmd
            +->close socket          +->go back in loop
   +<--- finish socketCmd.exe
   +- execute test
 <-+go back in loop

The Stimuli Selector GUI transmits only a command after receiving, and the socketcmd.exe transmits firstly, then wait for receiving. It means the socket is open and waiting for receive in the moment if the Stimuli Selector GUI transmits. That is proper.

If the test system is started firstly while the socket on the Stimuli Selector GUI is not opened, because the [gen test cases] is inactive, then the transmition of a cmd by socketcmd.exe is ignored, but the receiving after them recognizes the missing destination from the transmisson and returns error 10054. This is a standard behavior for socket communication. In this situation the socketcmd.exe returns with errorlevel=254 which can be evaluated for 'wait for GUI'.

The socketcmd.exe is programmed in C/++ with emC sources (available via Github) inside the project IDE/IDE/VS15_emCapplications/emCapplications.sln. It offers:

  • Transmit a message (a cmd) via UDP to the destination

  • Wait for receiving a message, test some simple messages and return the detection via error number. This is a simple approach to use it in command line scripts.

  • Wait for a time on errors

  • Only act as 'delay' for this test approaches.

On GUI side it is lesser complicated because Java can deal natively with sockets. The socket is only open if the thread for [gen test cases] does run. It may be also a decision to open the thread for the whole running of the GUI.

For the Gui the class ../../Java/docuSrcJava_vishiaRun//org/vishia/communication/SocketCmd_InterProcessComm.html offers the socket communication opportunities:

  • constructor called with IPv4 address and port for own and partner opens the socket

  • get the status and error status, important if the address parameters are faulty

  • tx(String) sends a String content in an UDP telegram

  • String waitRx() waits for a String content received on the own port (independent from where).

Using in the StimuliSelect_Example

The socketCmd.exe is stored in libs/ but gotten from its vishia-repository.

The file StimuliSelector.jzT.cmd contains

##
##This routine is the button routine for the [gen testcases] button.
##
sub btnGenTestcases ( String select) {
  call btnGenTestcases_M(select=select);
  ##call btnGenTestcases_A(select=select);   ##generate all files with different names
  ##call btnGenTestcases_M(select=select);   ##use socket messages
  ##call btnGenTestcases_F(select=select);   ##use file semaphores
}

You can change the called routine, change M against A or F to use the other approches. Here M is described.

##
##This routine is the button routine for the [gen testcases] button.
##Here it starts another thread which generates in loop step by step
##  after receiving a "next" command from UDP communication (using socketCmd.exe)
##If this routine is invoked secondly (press button secondly) and the thread is active
##  then the UDP socket connection is closed to abort the generation thread.
##
sub btnGenTestcases_M ( String select) {
  if(jztc.envar.soRx) {    ##hint: special variable inside Java wrapper.
    <+out>...abort genTestCases: <.+n>
    jztc.envar.soRx.tx("abort");
    jztc.envar.soRx.close();
    jztc.envar.soRx = null;
  }
  else {
    <+out>generate test cases: .... <.+n>
    Thread execThread = {         ## This thread generates one test case in each for loop
      call genTestCaseThread_M(select=select);
    }
    ##do not use: execThread.join(0);
    ##because the wrapper routine should be immediately finished,
    ##it is called in the GUI thread!
  }
}

The button routine quests an internal variable soRx. It is null if no communication is pending, then genTestCaseThread(…​) is called.

If it is pressed again during test cases are generating respectively the thread is running and waiting for interaction, the socket communication is closed. Following the receive routine for communication is aborted and the thread is finished. This is important, because for interaction it is not clarified what the partner does. Anytime the control over the whole process should be given.

In this routine a thread is created in JZtxtcmd: ../../JZtxtcmd/html/JZtxtcmd.html#Topic:.JZtxtcmd.thread. which calls getTestCastThread_M(…​).

The tread routine is used also for immediately invocations for nightly tests, see chapter Automatic execution of tests:

##
## genTestCases either in the thread or in automatically call
##
sub genTestCaseThread_M(String select) {
  String sIpOwn="UDP:127.0.0.1:45040";
  String sIpDst="UDP:127.0.0.1:45041";
  jztc.envar.soRx = java new org.vishia.communication.SocketCmd_InterProcessComm
                                                      (sIpOwn, sIpDst);
  Bool contFor = true;                       ## possibility to abort the generation
  Obj testcs = java org.vishia.testutil.TestConditionCombi.prepareTestCases( select, 5);
  Bool rxHasError = false;
  for(testcase: testcs && contFor ) {

In this first part the communication via socket is set up with the shown socket communication class. The address string can be better defined at the start of the script to be able to change it if needed. The port number is manually determined in coordination with the interaction partner. Here a local communication ("127.0.0.1" is the 'loop back' address) is used, instead any network card with known IP-V4 can be addressed too.

In expecting of a proper communication the generation of test files is started via calling prepareTestCases(…​), see chapter above. The files for the first test cases are generated:

In the loop soRx waits for a messae from the test system. But before an error handling is done.

    rxHasError = jztc.envar.soRx.hasError();   ## first time may be open error,
    if(rxHasError) {                           ## faulty socket etc.
      <+out>ERROR socket receive on <&sIpOwn>: <&jztc.envar.soRx.getState()><.+n>
      contFor = false;
    } else {
      ##                                       ## waits for a cmd received via socket:
      String next = jztc.envar.soRx.waitRx();  ## from the test system.
      <+out>rx from Test System: <&next><.+n>
      contFor = bool(next >= "step");          ## repeats, generate next if "step" is received
      if(contFor) {
        ##                                     ## prepare one test file

The quest soRx.hasError() is especially for the first invocation to detect an open error for the communication. Then the thread is aborted with a message, cannot work. This is common when perhaps an incorrect network has been selected, the port is in use, etc. The last one is especially given if the tool is started twice!

If no error is given, then the Stimuli Selector waits for a request from the test system via jztc.envar.soRx.waitRx();.

The test system may running in a loop (waiting for the Stimuli Selector) or not. It is emulated by the following batch file TestSystem_Msgs.bat:

echo off
set SOADDR_OWN=127.0.0.1:45041
set SOADDR_GUI=127.0.0.1:45040
REM see usage:
libs\socketcmd.exe
:loop
echo first wait for the start message via %SOADDR_OWN% from Stimuli Selection
echo ... expected: test abort or finish from socket
::echo on
:ask
libs\socketcmd.exe -own:%SOADDR_OWN% -dst:%SOADDR_GUI% -to:2000 -cmd step -rx test abort finish
::pause
::echo %errorlevel%
if errorlevel 255 goto :loop
if errorlevel 254 goto :GuinotReady
if errorlevel 4 goto :unknown
if errorlevel 3 goto :finish
if errorlevel 2 goto :abort
if errorlevel 1 goto :test

If this batch was started without running [gen test cases], it loops via errorlevel 254. To prevent a high CPU load because of spinning the socketcmd.exe waits here with 2000 ms timeout. It means if an error occurs it waits 2 seconds (with operation system thread capabilities) before return.

The thread in the Stimuli Selector GUI is continued in case of receiving step with:

        String name = <:><:for:var:testcase><&var.sel><:hasNext>_<.hasNext><.for><.>;
        <+out>test case: <&name><.+n>
        if(jztc.envar.stimuliSelector) {  ##Hint: stimuliSelector is not given outside GUI
          jztc.envar.stimuliSelector.btnGenTestcases.setBackColor
                            (jztc.envar.colorGenTestcaseActive, 0);
          jztc.envar.stimuliSelector.btnGenTestcases.setText("abort generate");
        }
        Obj lineValues = values.get(testcase[0].sel);
        Obj lineTexts = texts.get(testcase[1].sel); ## generates the files for this case:
        call genTestfiles(name = "testfile", values = lineValues, texts = lineTexts);
        ##
        java java.lang.Thread.sleep(1000);
        jztc.envar.soRx.tx("test");            ## starts the test with msg to Test System
        ##
        if(jztc.envar.stimuliSelector) {
          jztc.envar.stimuliSelector.btnGenTestcases.setBackColor
                            (jztc.envar.colorGenTestcaseWaitRx, 0);
          jztc.envar.stimuliSelector.btnGenTestcases.setText("abort wait rx");
        }
      }
    }
  }

It changes the color and text of the buttons to see in the GUI what’s happen. But this is only done if the variable stimuliSelector is available in jztc.envar. This is because the same routine is also used for Automatic execution of tests without the GUI.

Then the files for the testcase are generated calling exactly the same routine as in the button [gen selection].

After a seconds-sleep for this example only to show the green color of the button, but may be in praxis for timing conditions of the test system, the message test is sent to the test system. After them the color of the button is changed again, and it continues in the loop which starts with waiting to step from the test system, see above.

If test is received from the test system, socketcmd.exe returns with errorlevel 1 (the first -rx token) and continues on:

:test
echo ....test is running 3 seconds
type genScripts\testfile_text.txt
libs\socketcmd.exe -to:3000
REM if test is finished, send step to StimuliSelector for the next loop.
REM it is possible because the StimuliSelector has a message queue
echo test finished, sends "step" via socket:
goto :loop

For this example the test is only emulated by type of the content of one generated file and a wait. Because MS-Windows has no delay capabilities the socketcmd.exe is here used only to delay, an intrinsic proper capability.

A serial communication is similar. But a serial driver is not yet contained in the Stimuli Selector GUI. A simple way to do this is: Program in C/++, run it all the time, and exchange data via sockets with this special program and the Stimuli Selector GUI.

7.2. Possibility to interact via files

This is another possibility and socket and query files should combined sometimes.

In a test system it is often possible to query the existence of files and to remain in a loop if they are not present. This seems to be the best approach for the test files itself. But there is a pitfall. When the test file is started to be written, it already exists. But it is not ready for use. The test system should wait until the file is really ready. But this may not be queried in a unique way. Hence: Using the test files itself is a lesser good decision.

Files can be used as semaphore. If there are existing, it is a unique state. They can be deleted, then they are not existing, even correct.

Files can be addressed in a network. It is possible to realize a network communication with these semaphore files.

Another important idea is: Files should not be stupidly created or deleted. They should be renamed. The advantage is: Renaming only changes the directory entry in the file system. There is no necessity to allocate new space on the media.

But there is also a pitfall: Sometimes, if a process was aborted, the files are messed up. To prevent exactly this, the following operation is written in the vishiaJava_Base.jar component:

With this functionality the principle of file-semaphores are used in the Stimuli Selection GUI (JZtxtcmd) in the following way:

FileSystem.renameCreate(File: "genScripts", "*.msg", "idle.msg", 1);

It cleans a situation from pre-usage. The existence of any semaphore file may be possible. Expected is: genScripts/idle.msg exists from pre-usage or by first usage it is not existing. After execution, this is present and no other genScripts/.msg file exists. Note: You can use also the ‘’ as part of line, for example msg*_FromXtoY.txt. All files with the wildcard will be cleaned up.

FileSystem.renameCreate(File: "genScripts", "*.msg", "test.msg", 1);

A similar line. Because the situation is cleaned, the idle.msg is renamed to test.msg It is better to use this invocation instead a simple rename command, because - the situation may be unknwon again. Any other tool or person can touch on the file system.

The other side, the test system: Have a look on a test.do script used on a VHDL test system (https://www.aldec.com/en/products/fpga_simulation/active-hdl):

@label start
echo wait for Stimuli Selector GUI to execute a test
@label waitmsg
@if [exist -file "..\src\test\Lattice_pj\Test_All_SpeA\genScripts\abort.msg"]
  @goto finish
endif
@if not [exist -file "..\src\test\Lattice_pj\Test_All_SpeA\genScripts\test.msg"]
  @goto waitmsg
endif
##test.msg detected, now rename because it was seen.
!ren "..\src\test\Lattice_pj\Test_All_SpeA\genScripts\test.msg" idle.msg

This script part waits for the file test.msg but also for finish.msg. It runs in a "goto"-loop elsewhere, it is a "spinning". It test.msg is detected, it can be usual rename without failure in the currently running situation. Writing !ren is necessary here for this script language, ! to start a system command.

But in continuing this script the back message works with an event via socket:

acom -O3 -e 100 -work work -2002  .../Main_Test_SpeA_tb.vhd
asim Main_Test_SpeA_tb behavioral
run 38000 ns
endsim
##And now send a message to the StimuliSelector to produce the next file
!..\libs\socketcmd.exe -own:127.0.0.1:0xaff1 -dst:127.0.0.1:0xaff0 -cmd step
goto start

In this loop in the script also an evaluation of test results can be done.

Behavior in the StimuliSelect_Exampl

The file StimuliSelector.jzT.cmd contains

##
##This routine is the button routine for the [gen testcases] button.
##
sub btnGenTestcases ( String select) {
  call btnGenTestcases_F(select=select);
  ##call btnGenTestcases_A(select=select);   ##generate all files with different names
  ##call btnGenTestcases_M(select=select);   ##use socket messages
  ##call btnGenTestcases_F(select=select);   ##use file semaphores
}

You can change the called routine, change M against A or F to use the other approches. Here F is described. That routine starts with:

##
##This routine is the button routine for the [gen testcases] button.
##Here it starts another thread which generates in loop step by step
##If this routine is invoked secondly (press button secondly) and the thread is active
##  then the spRx variable is set to null to abort the generation thread.
##
sub btnGenTestcases_F ( String select) {
  if(jztc.envar.soRx) {    ##hint: special variable inside Java wrapper.
    <+out>...abort genTestCases: <.+n>
    FileSystem.renameCreate(File: "genScripts", "*.msg", "abort.msg", 1);
    jztc.envar.soRx = null;
  }
  else {
    <+out>generate test cases: .... <.+n>
    Thread execThread = {         ## This thread generates one test case in each for loop
      call genTestCaseThread_F(select=select);
    }
  }
}

It is similar to btnGenTestcases_M(…​) as in chapter Possibility to interact via socket but soRx is here a simple Boolean variable which is removed.

The genTestCaseThread(…​) creates a thread in JZtxtcmd: ../../JZtxtcmd/html/JZtxtcmd.html#Topic:.JZtxtcmd.thread

The tread routine is used also for immediately invocations for nightly tests, see chapter Automatic execution of tests:

##
##The genTestCases thread.
##Hint: The sub routine is the wrapper arround the thread.
##      The sub routine itself is finished immediately, necessary because it is calling
##      in the GUI thread.
##
sub genTestCaseThread_F(String select) {
  jztc.envar.soRx = java new java.lang.Boolean(1);
  FileSystem.renameCreate(File: "genScripts", "*.msg", "idle.msg", 1);
  Bool contFor = true;                       ## possibility to abort the generation
  Obj testcases = java org.vishia.testutil.TestConditionCombi.prepareTestCases
                                                              (select, 5);
  Bool rxHasError = false;
  for(testcase: testcases && contFor ) {

The start of the routine cleans up a possible mess situation with the file semaphores. It deletes all *.msg files and renames only one to idle.msg.

In expecting of a running test thread the generation of test files is started via calling prepareTestCases(…​), see chapter above. The files for the first test cases are generated:

In the loop soRx waits for a messae from the test system. But before an error handling is done.

    String name = <:><:for:var:testcase><&var.sel><:hasNext>_<.hasNext><.for><.>;
    <+out>test case: <&name><.+n>
    if(jztc.envar.stimuliSelector) {  ##Hint: stimuliSelector is not given outside GUI
      jztc.envar.stimuliSelector.btnGenTestcases.setBackColor(jztc.envar.colorGenTestcaseActive, 0);
      jztc.envar.stimuliSelector.btnGenTestcases.setText("abort generate");
    }
    Obj lineValues = values.get(testcase[0].sel);
    Obj lineTexts = texts.get(testcase[1].sel); ## generates the files for this case:
    call genTestfiles(name = "testfile", values = lineValues, texts = lineTexts);

    ##
    java java.lang.Thread.sleep(1000);
    FileSystem.renameCreate(File: "genScripts", "*.msg", "test.msg", 1);

The test files are created for the first test case, independent of the running test system. After them the idle.msg is renamed to test.msg. But if the situation may not clarified in any case, instead rename also the renameCreate(…​) routine is used. It is a java routine with less additional effort for this additional features.

The test system may be started before or after them. It waits in any case of the file test.msg:

:loop
echo first wait for presence of the semaphore file from Stimuli Selection
echo ... expected: genScripts\test.msg, ...\abort.msg or ...\finish.msg
:waitstart
if exist genScripts\test.msg goto :test
if exist genScripts\abort.msg goto :abort
if exist genScripts\finish.msg goto :finish
libs\socketcmd.exe -to:2000
goto :waitstart

The socketcmd.exe is here used only for a thread-wait behavior. It goes in a loop till the file is existent.

If the test.msg is recognized, it executes the test:

:test
echo
echo The test needs a moment, here a ping is executed which needs some time.
echo ....test is running 3 seconds
type genScripts\testfile_text.txt
libs\socketcmd.exe -to:3000
echo test finished, rename step.msg:
ren genScripts\test.msg step.msg

goto :loop

The test is here also emulated by a wait (via socketcmd.exe after type. Ater the test the test.msg is renamed to step.msg. This is rcognized by the Stimuli Selector GUI:

    <+out>Thread waits for file step.msg:<.+n>
    while( NOT File:"genScripts/step.msg".exists() && jztc.envar.soRx) {
      java java.lang.Thread.sleep(1000);     ## wait for renaming to step.msg or abort
    }
    <+out>file-semaphore detected: step.msg::<.+n>
    contFor = jztc.envar.soRx;              ## repeats, generate next if "step" is detected
  }

Here in JZtxtcmd also the existence of a file is checked, with Java capabiities. The File:"name" creates a java.io.File istance, which*s exists() operaiton is invoked. Additional the jztc.envar,soRx is quest. It is a boolean variable with true, which is set to null on abort, see above. The following sleep(…​) prevents too high CPU load on spinning.

The generation loop is continues if soRx is still true.

  <+out>Thread finished<.+n>
  FileSystem.renameCreate(File: "genScripts", "*.msg", "finish.msg", 1);
  if(jztc.envar.stimuliSelector) {
    jztc.envar.stimuliSelector.btnGenTestcases.setBackColor(jztc.envar.colorGenTestcaseInactive, 0);
  }

After finish the loop in the Stimuli Selector GUI the file is renamed to finish.msg which is checked by the test system, see above.

7.3. Evaluation of test data

This can be included either in the test system or in the GUI. How to evaluate test results - it depends deeply on the test system. Often results are stored in files, the content of the files should be evaluated, and over all results an average or overview should be built, and a report of faulties. This can be done of course immediately in the test system, but also integrated in the Stimuli Selector GUI. The benefit of the Stimuli Selector GUI is: It is programmed in Java (from JZtxtcmd Java routines can be called simple). And - some things can be done - Adaptions - immediately in JZtxtcmd without necessity of Java compiling.

The example does not have such capabilities, look at concrete applications.

8. Automatic execution of tests

This is the last and important capability.

The Stimuli Selector GUI has firstly the approach to find out test cases manually and generate the test files.

The second is, check some combinations, as shown in the chapters above, find out proper combinations for nightly tests.

But the nightly test itself should be started and executed without manual handling. It should be started via a command line invocation, from a timing service, in a superior execution loop etc.

It is very simple. Look in the example. Here in AutomaticTestExecution.jzt.cmd:

set LIBSPATH=./
echo on
java -cp %LIBSPATH%/libs/vishiaBase.jar org.vishia.jztxtcmd.JZtxtcmd %0
echo off
pause
exit /b


==JZtxtcmd==

include StimuliSelector.jzT.cmd;

currdir=<:><&scriptdir><.>;


main() {

  Obj jztc.envar.soRx = null;

  call genTestCaseThread(select = "1=v1; 2=t1; 3=A1, A2, A3, A4; 4=B1; 5=C1; " );

This is a simple invocation of JZtxtcmd via java, can be included also in another Java programming. This file is used also as JZtxtcmd script, contains a main() and invokes call genTestCaseThread(select = "…​."); with any select String. The select String can be copied from the select text box of the Stimuli Selector GUI after successfully tests. If the select String contains more lines, it is able to write in a good visual form:

  call genTestCaseThread(select = <:>1=v2, v3; 2=t1, t2; 3=A1; 4=B1; 5=C1;
                                     : 1=v1; 2=t1, t2; 3=A1, A2; 4=B1; 5=C3;<.> );

  call genTestCaseThread(select = <:>1=v2, v3; 4=B1, B3; + 1=v1; 4=B2
                                     & 2=t1,t2; 3=A1; + 2=t2; 3=A3;
                                     & 5=C1<.> );
}

That are three test combination executed one after another.

The vishiaGui.jar and the swt…​jar is not necessary because graphic capabilities are not used. Because the variable soRx is provided in the GUI, it is supplemented here by a definition in the script. (The variable need to provide in the GUI for GUI appoaches because there content should be preserved though a new script version is loaded).

Threre can be more as one such files, organized to the user’s approach. The execution can depent on test results (execute finer tests if results are proper) etc etc. But that is a feature of test organization outside of the Stimuli Selector GUI.