bugfinder.processing.tokenizers.replace_variables

class bugfinder.processing.tokenizers.replace_variables.ReplaceVariables(dataset, deprecation_warning=None)

Bases: AbstractTokenizer

Processing to replace user-created functions from a dataset.

execute()

Run the processing.

static process_file(filepath)

Process a single file looking for user-created variables and replace them with a token VAR to reduce uniqueness in the corpus.

Parameters

filepath (str) – Path of the file to be processed

Returns

number of variables replaced

Return type

int