Blog Entries

Allow the calculation of coverage values regardless of optimisation.

Allows to yield, for example, the resulting line coverage of the test suite, while the optimisation during test generation was done for branch coverage. Add the required coverage values to your —output-variables list to activate this feature.

Note: when doing this, the Coverage output variable will contain the average value of the incorporated coverage values!

Provide a Cobertura-like coverage report

Pynguin already provides an HTML report that can be activated by setting —create-coverage-report True. This report is nice for human users but not very usable if one wants to automatically reason about the achieved coverage using tools. We thus also emit an XML report in the style of the Cobertura tool that can be used for further automated tools.

Fix typo on test_parameterizedstatements.py

Fix typo on testcase.py

Improve mutation-based assertion generation

The assertion generation now does not more compare the recorded assertion traces but actually executes the assertions to make the whole process more reliable. Besides, we do not check for is not None as a fallback for object checks any more, but use an isinstance check on the object’s type to have a more precise assertion.

Make statistics tests debuggable

Some accidental circular import made it impossible to run a debugger on tests in pynguin.utils. We resolved this by moving tests to another package.

Partial rework of internal type system

We added an abstraction layer over the existing type hints from a module to make handling and reasoning with types easier. This abstraction is based on the one used by mypy, however, we only cover a small part of what PEP-484 actually defines.

Make assertion generation more strict

Fix killing mutant reporting

Use black to format the generated test cases.

Pynguin now requires black as a run-time dependency to use its code formatting.

Provide a naive inheritance graph to improve input generation.

Improve killing of long-running test-case executions

Add computation of mutation scores for MUTATION_ANALYSIS assertion generation.

The output variables NumberOfCreatedMutants, NumberOfKilledMutants, NumberOfTimedOutMutants, and MutationScore allow to export those values.

Do not enable typing.TYPE_CHECKING for SUT analysis as this may cause circular imports.

Improve the black list of modules that shall not be incorporated into the test cluster.

Annotate failing tests with @pytest.mark.xfail(strict=True).

Improve log output of mutation-based assertion generation.

Add instrumentation to mutated modules to easier kill them.

This change is relevant only to the MUTATION_ANALYSIS assertion-generation strategy.

Write errors in execution threads to the log instead of STDERR to avoid cluttering log output.

Add limits for amount and size of constants in the constant pool.

The configuration options max_dynamic_length and max_dynamic_pool_size allow to set sizes for the maximum length of strings/bytes that shall be stored in the dynamic constant pool and the maximum numbers of constants of a type, respectively. This prevents the constant pool from growing unreasonably large.

Improve handling of type annotations.

Fix computation of cyclomatic complexity.

Computing cyclomatic complexity does not work for functions that are not present in the AST, e.g., default constructors. We now omit those from the computation of the cyclomatic-complexity output variables.

Fix selection of type-inference strategy.

Fix a bug in the type inference regarding cases where not type information is present.

Add a PyLint checker for calls to print().

Extend the blacklist of modules that shall not be analysed.

Raise RuntimeError from tracer when called from another thread.

Provide better exception messages for critical failures.

Apply a further limit to the execution time of a single generated test case to at most 10 seconds.

Exclude empty enum classes from test cluster to fix test generation.

Parsing included modules raised an issue when the enum module is used: the test cluster then had a reference to the enum.Enum class, which obviously does not contain any fields. In the following, generating tests failed, as soon as this class was selected to fulfil parameter values because there was no field to select from, e.g., MyEnum.MY_FIELD. We now exclude empty enums from the test cluster.

Fix a bug in the module analysis regarding nested functions.

Nested functions/closures caused Pynguin’s module analysis to crash with a failing assertion.

Improve the branch-distance computation for bool values.

Allow for more statistics variables regarding number of lines and cyclomatic complexity.