Testing¶

Bases: TestcaseGenerator

A generator that produces tests based on combination of ranges.

This generator creates a test case for each combination of the provided value ranges. Each value range must be associated with exactly one input feature of the model that shall be tested.

Initialize the combination generator.

Parameters:

data: DataFrame

number_test_cases: int = functools.reduce(lambda n, range_: n * len(range_), self.domains, 1)

domains = discrete_domains

num_steps() -> int | None

reset() -> None

step() -> None

Bases: IncrementalEnvironment

A generator that produces test cases for a model.

reset() -> None

save_csv(path: str | Path, *, test_case_count: int | None = None, separator: str = ',') -> None

save_excel(path: str | Path, *, test_case_count: int | None = None, worksheet_name: str = 'Test Cases') -> None

Bases: TestcaseGenerator

A generator that produces random tests based on given domains.

Initialize the stochastic generator.

Parameters:

data: DataFrame

count: int = 0

domains = domains

number_test_cases = test_case_count

num_steps() -> int | None

reset() -> None

step() -> None

Bases: TestcaseGenerator

Generates test inputs considering decision boundaries.

Methods:¶

save_hoeffding_tree() Saves the generated Hoeffding tree to a file.

print_eqclasses() Prints the equivalence classes and their test input counts.

print_testplans() Prints the test plans (intervals used to sample test inputs).

print_hoeffding_tree() Prints the Hoeffding tree structure as a PNG.

Initialize the stochastic generator.

Parameters:

n_testinputs = n_testinputs

seed = get_seed()

test_pipeline = TestPipeline(model, dataset=dataset, specs_file=specs_file, classification=classification, n_testinputs=self.n_testinputs, test_coverage_criterium=test_coverage_criterium, inverse_alloc=inverse_alloc, epsilon=epsilon, seed=self.seed, performance_threshold=performance_threshold, sample_limit=sample_limit, n_predictions=n_predictions, max_depth=max_depth, hoeffding_tree_extra_params=hoeffding_tree_extra_params)

data = DataFrame(self.test_pipeline.execute())

num_steps() -> int | None

reset() -> None

step() -> None

print_eqclasses() -> None

print_testplans() -> None

print_hoeffding_tree() -> None

save_hoeffding_tree(path: str | Path) -> None

Load a Flowcean model and expose its underlying ML model.

Attributes:¶

model: flowcean.core.model.Model The loaded Flowcean model.

Methods:¶

get_ml_model() Returns the underlying machine learning model from the Flowcean model.

get_model_prediction() Returns predictions from the Flowcean model as a LazyFrame.

get_model_prediction_as_lst() Returns predictions from the Flowcean model as a Python list.

Initializes the ModelHandler.

Parameters:

model = model

get_ml_model() -> SupportsPredict | Module

get_model_prediction(input_features: DataFrame) -> LazyFrame

get_model_prediction_as_lst(input_features: DataFrame) -> list

Workflow for test input generation in Flowcean.

Attributes:¶

model_handler: ModelHandler Handles the Flowcean model and its predictions.

Methods:¶

execute() Executes the full test input generation workflow.

save_hoeffding_tree() Saves the generated Hoeffding tree to a file.

save_test_overview() Saves intermediate results and generated outputs.

Initializes the TestPipeline.

Parameters:

model_handler = ModelHandler(model)

model = self.model_handler.get_ml_model()

n_testinputs = n_testinputs

test_coverage_criterium = test_coverage_criterium

dataset = dataset

specs_handler = SystemSpecsHandler(data=dataset, specs_file=specs_file)

feature_names = self.specs_handler.extract_feature_names()

hoeffding_tree = None

classification = classification

inverse_alloc = inverse_alloc

seed = seed

epsilon = epsilon

performance_threshold = performance_threshold

sample_limit = sample_limit

n_predictions = n_predictions

max_depth = max_depth

hoeffding_tree_extra_params = hoeffding_tree_extra_params if hoeffding_tree_extra_params is not None else {}

execute() -> DataFrame

print_eqclasses() -> None

print_testplans() -> None

print_hoeffding_tree() -> None

save_hoeffding_tree(path: str | Path) -> None

Bases: Domain

A domain of continuous values.

This domain describes a continuous distribution of values between a minimum and maximum value for a feature.

Initialize the uniform feature domain.

Parameters:

rng: Random = random.Random()

distribution = distribution

mean = mean

stddev = stddev

min_value = min_value

max_value = max_value

get_value() -> float

set_seed(seed: int) -> None

to_discrete(sampling_distance: float) -> Discrete

Bases: Domain, Iterable[tuple[str, float]]

A domain of discrete values.

This domain describes a discrete set of values for a feature.

Initialize the discrete domain.

Parameters:

rng: Random = random.Random()

values = values

get_value() -> float

set_seed(seed: int) -> None

Bases: ABC

An abstract base class for describing the value domain for a feature.

Initialize the domain.

Parameters:

feature_name: str = feature_name

get_value() -> float

set_seed(seed: int) -> None

Bases: Discrete

A domain with a single value.

This domain contains a single fixed value for a feature.

Initialize the fixed domain.

Parameters:

get_value() -> float

Bases: Predicate

Predicate for Polars DataFrame.

This predicate allows for two different ways to provide the predicate expression:

1. As a Polars expression. This expression is used directly and must evaluate to a single boolean value. For example, the following expression checks if the values in the "feature_a" column are in the list [1, 2, 3] and if the values in the "feature_b" column are greater than 0:

       import polars as pl
   
       PolarsPredicate(
           pl.col("feature_a").is_in([1, 2, 3]).and_(pl.col("feature_b") > 0),
       )
   

2. As a string. The string is parsed as a Polars expression. Any string identifier are replace with the respective feature during evaluation. The string expression must evaluate to a single boolean value as well. For example, the following expression checks if "feature_a" is always greater than "feature_b":

       import polars as pl
   
       PolarsPredicate(
           "feature_a > feature_b",
       )
   

   Boolean expressions like and, or, and not are not supported by this syntax. See AndPredicate, OrPredicate and NotPredicate for combined predicates or use the polars expression syntax above.

Initialize the predicate from a polars expression or a string.

expr = _str_to_pl(expr) if isinstance(expr, str) else expr

Bases: Predicate

Combine multiple predicates with a logical AND operation.

predicates = predicates

Bases: Predicate

Negate a predicate.

predicate = predicate

Bases: Predicate

Combine multiple predicates with a logical OR operation.

predicates = predicates

Bases: ABC

Base class for predicates.

A predicate is a function that takes the prediction of an model and returns a boolean value indicating whether the prediction satisfies a certain condition. Predicates can be combined using logical operators (AND, OR, NOT) to create more complex predicates.