Modify a checkpoint file during restarting to run longer

In this example, the first simulation generates a checkpoint file. The second simulation runs from the checkpoint file after modification of the number of iterations to complete a simulation.

import sys
import pandas as pd
import subprocess

# Run a feasst simulation with the given parameters
def run_fst():
    with open("launch00.txt", "w") as myfile: myfile.write("""
MonteCarlo
Checkpoint checkpoint_file lj.fst
RandomMT19937 seed 1572362164
Configuration cubic_side_length 55.0321208149104 particle_type0 /feasst/particle/lj.fstprt
Potential Model LennardJones VisitModel VisitModelCell min_length max_cutoff
Potential VisitModel LongRangeCorrections
ThermoParams beta 1.111111111111111 chemical_potential0 -1
Metropolis
TrialTranslate weight 1 tunable_param 2
Tune
CheckEnergy trials_per_update 1e4 tolerance 1e-8
TrialAdd weight 2 particle_type 0
Run until_num_particles 500
RemoveTrial name TrialAdd
Run num_trials 1e5
RemoveModify name Tune
Log trials_per_write 1e4 output_file lj.csv clear_file true
Energy trials_per_write 1e4 output_file lj_en.csv
Movie trials_per_write 1e4 output_file lj.xyz
Metropolis num_trials_per_iteration 1e4 num_iterations_to_complete 1e2
Run until_criteria_complete true
WriteCheckpoint
""")
    syscode = subprocess.call("../../../build/bin/fst < launch00.txt", shell=True, executable='/bin/bash')
    if syscode > 0: sys.exit(1)

run_fst()
df = pd.read_csv('lj.csv')
assert len(df) == 101

# Write a second run script that restarts the simulation and adds a lot more trials. Too many. Try to clear those on a second restart.
with open('continue.txt', 'w') as myfile: myfile.write("""Restart lj.fst
Metropolis num_trials_per_iteration 1e8 num_iterations_to_complete 2e8
Checkpoint checkpoint_file lj2.fst num_hours_terminate 0.01
Run until_criteria_complete true
WriteCheckpoint
""")
syscode = subprocess.call("../../../build/bin/fst < continue.txt", shell=True, executable='/bin/bash')

# Write a third run script that restarts the simulation and the clears the previous arguments. Ask for a reasonable number of trials.
with open('continue2.txt', 'w') as myfile: myfile.write("""Restart lj2.fst clear_previous_arguments
Metropolis num_trials_per_iteration 1e4 num_iterations_to_complete 1e1
Run until_criteria_complete true
WriteCheckpoint
""")
syscode = subprocess.call("../../../build/bin/fst < continue2.txt", shell=True, executable='/bin/bash')

df = pd.read_csv('lj.csv')
assert len(df) > 202

# FEASST version: v0.25.3
# Usage: ./fst < file.txt
MonteCarlo
Checkpoint checkpoint_file lj.fst
RandomMT19937 seed 1572362164
# initializing random number generator with seed: 1572362164
Configuration cubic_side_length 55.0321208149104 particle_type0 /home/hwh/feasst/particle/lj.fstprt
Potential Model LennardJones VisitModel VisitModelCell min_length max_cutoff
Potential VisitModel LongRangeCorrections
ThermoParams beta 1.111111111111111 chemical_potential0 -1
Metropolis
TrialTranslate tunable_param 2 weight 1
Tune
CheckEnergy tolerance 1e-8 trials_per_update 1e4
TrialAdd particle_type 0 weight 2
Run until_num_particles 500
RemoveTrial name TrialAdd
Run num_trials 1e5
RemoveModify name Tune
Log clear_file true output_file lj.csv trials_per_write 1e4
Energy output_file lj_en.csv trials_per_write 1e4
Movie output_file lj.xyz trials_per_write 1e4
Metropolis num_iterations_to_complete 1e2 num_trials_per_iteration 1e4
Run until_criteria_complete true
WriteCheckpoint
# FEASST version: v0.25.3
# Usage: ./fst < file.txt
# Restarting from file: lj.fst
Metropolis num_iterations_to_complete 2e8 num_trials_per_iteration 1e8
Checkpoint checkpoint_file lj2.fst num_hours_terminate 0.01
Run until_criteria_complete true

terminate called after throwing an instance of 'feasst::CustomException'
  what():  Throw on proc 0 : #Fatal error0 plugin/utils/include/checkpoint.h:88: Terminating because Checkpoint has reached the user input num_hours_terminate: 0.01. Detect this termination in Bash shell using "$? != 0"
/bin/bash: line 1: 3690205 Aborted                 (core dumped) ../../../build/bin/fst < continue.txt

# FEASST version: v0.25.3
# Usage: ./fst < file.txt
# Restarting from file: lj2.fst
Metropolis num_iterations_to_complete 1e1 num_trials_per_iteration 1e4
Run until_criteria_complete true
WriteCheckpoint

Did this tutorial work as expected? Did you find any inconsistencies or have any comments? Please contact us. Any feedback is appreciated!