hmtoggle_plus1Introduction

The term Regression Testing refers to running a suite of tests that are designed to show that, after improvements to a system, no new bugs have been introduced.  Generally this is done to see how the behavior of a new release of a SUT compares with that of a prior release of the SUT. MxVDev has good support for automation of this process.

In the context of MxVDev, a Regression Test Suite is a selection of the available Scenarios that can automatically be executed in sequence. When you create such a sequence, it is saved in a Regression Command File.  The sequence may be all of the Scenarios for the project, or it may be a subset. You can include sets of Scenarios from several projects.

Notes:

•The Regression Test creates a copy of the ScenariosAndTestCases folder. This is so tests form a set with their results. For this reason, all Scenarios and TestCases included in a Regression Test must be in their project's ScenariosAndTestCases folder or a subfolder under it.

•If you get a Path Too Long Error, see Troubleshooting–Path Too Long Error.
hmtoggle_plus1Regression Sets

You can have one or more Regression Sets.  Each Regression Set can use a different MxVDev Project.  It is often the case that there can be several Projects that access some of the same Scenarios and TestCases, or that one System has several MxVDev test projects.  It is useful to be able to test more than one Project in a Regression Command File.  Each Regression Set can include one or more Scenarios from the chosen Project.  See the section below on Creating Regression Sets.

Note: If you wish to select scenarios using Tags, you must create the Tags before starting the Regression Test Wizard.
hmtoggle_plus1Create/Edit a Regression Command File

You create and edit Regression Command Files (.mxreg) using the Regression Test Wizard. The wizard provides a wide variety of options for organizing the report:

•To start the wizard, select Tools‑>Regression Test from the main menu.  

•Alternatively, you can use the Project Explorer:

oTo create a new Regression Command File, right-click on Regression and select New Command File.

oTo edit an existing Regression Command File, right-click on the existing Regression Command File and select Edit.

RCF

Use the Add, Edit, and Remove buttons to manage the Scenario Sets. Click on the Set line to select it and activate all the buttons.

There are several steps in the wizard which capture information about how you would like the Regression Test to execute.  These are described below.

hmtoggle_plus1Creating Regression Sets

Click the Add or Edit button to continue to the Regression Test Wizard.

Regression Test1

•Title - This text is displayed in the header at the top of the report.

•Project Path - Click 3dots_button to browse for the Project File.

•Test Info - Use this panel to document the test.  This information is included in the Regression Test Report.

Click Next to proceed through the wizard.

RegressionTest2

•Result Location.  The results of a Regression Test run are by default stored in the Regression folder of the project.  A full copy of all the tests run is made so that you can see exactly which tests were run and what the results were at a later date. You can let MxVDev automatically create the Regression Result Set sub-folder for the results or you can enter the name manually.

oAuto Date - A Regression Result Set folder is created under the Regression folder for the project.  The folder is named based on the current PC date and time.  There is an option (found under Tools->Options) to limit the number of Regression Result Set folders that are preserved on your computer.  When the number is exceeded, the oldest Regression Result Set will be deleted.  To prevent a Regression Result Set from being automatically deleted, just rename the folder.

oManual - Enter the name of an folder where you want to store the Regression Result Set or click 3dots_button to browse.

•Report Options - Use this panel to select which sections to include in the Regression Test Report and how it is formatted.

oInclude Summaries - If your report is for a single scenario, turn off these options as the results will be trivial.

oShow TestCase Usage - Select to include the TestCase Usage Summary in the report. This section shows which Scenarios use each TestCase.

oShow Failure Details - Select to include the Failure Details section in the report. Deselect for a shorter report.

oIn Frame - If you check the In Frame box, the report is presented in frames for easy navigation, however the Back button on your browser does not work in frames mode.

oInclude mxout folders - Select to include a listing of the contents of mxout folders in the report.

oShow Tolerance Table - Tolerance tables may be omitted to make the report shorter. The default is Show.

oShow Transitions Table - Transition tables may be omitted to make the report shorter. The default is Show.

oGenerate JUnit file - See Jenkins Integration.

Click Next to proceed through the wizard.

Regression Test3

Use these panels to provide more information to include in the report:

•Distribution List

•Test Equipment

Click Next to proceed through the wizard.

If you are testing an MxVMC, the dialog below is displayed.

You can also use Visual Studio to enable code coverage. See RapiCover.

Regression Test4

•Force Rebuild - Check this box if you want all the Visual Studio SUTs to be rebuilt automatically before the regression Command File is run.

•Code/Model Coverage - Select an option to enable or disable Code Coverage.

•Clear Coverage stats - This option is not used by current coverage tools.

•Click the Next>> button at the bottom of the dialog to proceed through the wizard.
Click to expand.

Click to expand.

There are some options to control the action when the Regression Command File executes.

•Action - In most cases you want to just Execute MxVDev Automatic Regression.  There are two other options:

oCreate Command Files - Choose this option if you want to create a set XML files that represent the command (stimulus) transitions from each Scenario (<ScenarioName>.mxcmd). The files are generated as the Scenario is executed. These files can then be run in another test environment.  For example, the MxSuite MxVI (LabVIEW Connector) can read and execute these files on an NI/LabVIEW test platform. The files are generated in the Regression folder.

oPerform Pass/Fail Analysis on *.mxlog Files - The .MxLog XML file format is the standard MxSuite format for Scenario results.  It includes captured output data from the SUT.  Choose this option if you want to process a set of output MxLog files from a previously executed test.  MxVDev loads the files into the corresponding Scenarios, performs the Pass/Fail analysis and produces a regression report.  This feature can be used to re-analyze results files or, process data from another test environment, such as is generated by the MxSuite MxVI (LabVIEW Connector).
 
 

Show GUI - If you check this option you will see the tests executing in an animated fashion, as the regression runs.  If you turn off this option, you will not see the tests execute, but the tests will run faster.  (Note: MxVDev also runs faster if you select Show GUI, but our GUI is covered by some other application such as Excel, since Windows does not have to repaint the screen.)

Stop on Fail - Select this check box to stop the test if a Scenario fails.

Show report on completion - Select this box to open the report results when the test completes. The report is opened by the browser or application associated with XML files in Windows Default Programs.

Click Next>> to proceed through the wizard.

You can select which Scenarios are run during a regression test cycle. There are a couple of ways to produce the list:

1.Select a folder in the Scenario Folders (left) box. You can select each folder included in the Default Scenario/TestCase Folders List in the Project Settings.

2.Choose Scenarios based on which file or which folder they are stored in.

3.To select Scenarios from another folder, repeat steps 1 and 2.

RegressionByFolder

 

Choose Scenarios based on the Tags that have been associated with them.

RegressionByTags

Note: Scenarios must be tagged before using the Regression Test Wizard.  See Adding Tags and Scenario Tagging.

To define a tag query in the regression editor:

1.On the Scenario Selection panel of the Regression Test Wizard, select the By Tags option.

2.To add a new sub-query, click the Add Row button.

3.Click the Add button.

4.On the Add Group form, either select a previously defined group from the drop down list, or enter the name of a group.

Note:The group name can contain wild card characters (* for any number of characters, or ? for a single character)

5.Right-click on the cell in the new row and the new column. This will display a list of the following tagging options.

•Tag - Selecting a tag from this menu means that the tag must be present for the scenario to satisfy the sub-query.

•No Tag - Selecting a tag from this menu means that the tag must not be present for the scenario to satisfy the sub-query.

•Any Tags Present - The sub-query is satisfied if the scenario is associated with any tags from this group.

•No Tags Present - Ignore this group.

•Don't Care - This group has no effect on the sub-query.

•Custom... - Allows the user to enter a custom tag name. Note:The tag name can contain wild card characters (* for any number of characters, or ? for a single character).

6.Repeat this process for all tags, groups, and sub-queries.

7.The Also include untagged Scenarios checkbox provides a final rule for inclusion. Check the box to include all Scenarios that have no tags defined.

For more information, see Filtering Based on Tags.

Test Query Button

To test the query, click on the Test Query button. The query scans all Scenarios located in and under the ScenariosAndTestCases folder.  For a Scenario to be valid for a sub-query, it must satisfy all tagging options in that subquery's row. For a Scenario to be valid for the entire query, it must satisfy at least one sub-query.  If the "Also include untagged scenarios" checkbox is checked, any untagged Scenarios are also selected by the query.

 

The Regression Command File is a text file.  The summary panel shows the contents of this command file for you to review.

If the Auto Sort button is selected, the Scenario files are sorted alphanumerically before they are executed. If you want to define a particular order, select the Manual Sort button, and use the arrow buttons on the right to arrange the scenarios.

RegressionScenarioSort

Note: If you delete a folder from the Scenario Set, its position in the Manual Sort order is lost. If you add it back, it is placed at the end of the sort order. You can then rearrange the sort order.

Click Finish to finish creating or editing the Scenario Set.

When you have finished creating Scenario Sets, you have one more choice to make before you close the form:

•Close and Run - Click this button to save and execute the Regression Command File.

•Close - Click the Close button to save the command file without running the regression tests.
hmtoggle_plus1Results

When you run a Regression Test, the raw test data is stored in an XML regression test output log file with the extension .mxrlg. The folder is specified in the Regression Test Wizard. This file is used to generate the Regression Report. Since it is in XML, you can also use it as data for reports of your own design.

You can also convert the completed report to PDF. See Convert Report to PDF.
hmtoggle_plus1Running a Single-Scenario Report

You can use the Project Explorer to run a Regression Report on a single Scenario.

1.In the Project Explorer, click on a Scenario (.mxs) file.

2.Drag and drop the Scenario onto a Regression Script (.mxreg) file.

RegressionSingleScenario

The Regression Script (.mxreg file) must be created before running the test. Only the Scenario that is dropped runs regardless of the options set in the script. You can switch off the report summaries since they will be trivial.
hmtoggle_plus1Regression Result Set Comparison (Mx‑VDiff)

MxVDiff allows you to explore in depth the differences between two Regression Result Sets.  In the Project Explorer:

•Right-click on one Regression Result Set and select Compare - Set as Baseline Results.

•Right-click on another Regression Result Set (usually with a later date) and select Compare - Set as Subsequent Results.

The MxVDiff utility is launched, allowing you to drill down and examine in detail any differences between the two result sets.  See MxVDiff for more details.
hmtoggle_plus1Generating a Report from an Incomplete Test

If a Regression Test is interrupted due to a power failure or other problem, it may be possible to generate a report for the Scenarios that completed before the interruption.

Right-click on the test folder (located in the Runs folder in the Project Explorer) and select Generate Regression Reports.

Regression-Generate

If the Regression Test terminated abnormally, it is indicated on the report in the Regression Actual Run Time field as shown here:

Regression-incomplete-marked

 

 

Related Topics:

Tagging

Adding Tags

Regression Report

MxVDiff

MxVDev Options–Regression Report Folder Limit

Report Customization

Scenarios Overview

Working with Scenarios

Troubleshooting–Path Too Long Error