bugfinder.processing.tokenizers

class bugfinder.processing.tokenizers.AbstractTokenizer(dataset, deprecation_warning=None)

Bases: AbstractProcessing

abstract execute(*args, **kwargs)

Execute the processing. Needs to be implemented by the subclass.

Parameters

• args –

• kwargs –

• bugfinder.processing.tokenizers.replace_functions
  • ReplaceFunctions
    • ReplaceFunctions.execute()
    • ReplaceFunctions.process_file()
• bugfinder.processing.tokenizers.replace_variables
  • ReplaceVariables
    • ReplaceVariables.execute()
    • ReplaceVariables.process_file()
• bugfinder.processing.tokenizers.tokenize_code
  • TokenizeCode
    • TokenizeCode.execute()
    • TokenizeCode.process_file()
    • TokenizeCode.to_regex()