Evaluation#
This page describes the scoring system for the competition. Teams are evaluated on their performance across multiple runs and trials, with scores calculated based on task completion, efficiency bonuses, and penalty deductions. The evaluation process covers individual run scores, which are aggregated into an execution score (80%) and combined with human judging (20%) to determine final team rankings.
Note
Detailed submission requirements and procedures will be announced at a later date.
Run Score#
Base Score: How well was the assigned task completed?
Bonus Score: Various factors; how efficient and accurate is the system?
Penalties: Various sources; how often did the team perform an illegal action?
\(R\) |
Total run score |
\(B\) |
Base score |
\(\beta_i\) |
Bonus scores |
\(\rho_i\) |
Deduction for each penalty type |
\(o_i\) |
Number of occurrences for the associated penalty |
Scoring Weights#
The scoring system uses predefined weights to balance the importance of different performance aspects. These weights determine how much each component contributes to the overall score.
Important
The exact scoring weight values are subject to change and are defined in the scoring_weights.yaml configuration file. Teams should refer to this file for the current values used by the scoring system.
The following weights are used in the scoring calculations:
\(\omega_1\) - Kit completion weight
\(\omega_2\) - Module completion weight
\(\omega_3\) - Trial time bonus weight
\(\omega_4\) - Inspection speed bonus weight
\(\omega_5\) - High priority order speed bonus weight
\(\omega_6\) - Sensor cost bonus weight
\(\omega_7\) - Inspection classification bonus weight
Base Score#
The base score is calculated from the submission of kitting orders and module orders.
\(B\) |
Base score for the run |
\(k_c\) |
Number of kits successfully submitted |
\(k_d\) |
Number of kits desired |
\(m_c\) |
Number of modules successfully submitted |
\(m_d\) |
Number of modules desired |
Important
High priority kits are included in \(k_d\)
Kit Acceptance Criteria#
Four good cells (not defective and within voltage specification)
Total kit voltage within tolerance
Kit delivered to shipping area on an AGV
Module Acceptance Criteria#
Four good cells with individual voltages in specification
All cells properly oriented with correct polarity
Total module voltage within tolerance
Bottom and top shells properly installed and secured
All six welds completed successfully:
Four top welds
Two bottom welds
Module delivered to submission area on assembly conveyor
Bonus Points#
Teams can earn bonus points from several sources:
Note
All bonus values are clamped to zero - bonuses cannot be negative.
Important
Teams are only eligible for bonuses if all desired kits and modules were submitted
Important
Target values and thresholds used in bonus calculations are subject to change and are defined in the scoring_weights.yaml configuration file. Teams should refer to this file for the current values used by the scoring system.
Trial Time#
The trial time bonus is awarded for completing the competition in less time than allocated.
\(\beta_1\) |
Trial time bonus |
\(t_e\) |
Run execution duration |
\(t_m\) |
Run time limit |
Inspection Speed#
The inspection speed bonus is awarded for submitting inspection reports faster than the target time.
\(\beta_2\) |
Inspection speed bonus |
\(\gamma\) |
Average inspection duration |
\(\gamma_d\) |
Desired inspection duration |
High Priority Order Speed#
The high priority order speed bonus is awarded for completing high priority orders faster than the time limit.
\(\beta_3\) |
High priority order speed bonus |
\(\tau\) |
Average high priority kit execution duration |
\(\tau_d\) |
Desired high priority kit execution duration |
Sensor Cost#
The sensor cost bonus is awarded for using sensors below the allocated budget. Exceeding the budget results in a penalty instead.
\(\beta_4\) |
Sensor cost bonus |
\(\sigma\) |
Team sensor cost |
\(\sigma_b\) |
Sensor budget |
Inspection Classification#
The inspection classification bonus is awarded for correctly identifying defect types and locations. All elements of the defect report must be correct.
\(\beta_5\) |
Inspection classification bonus |
\(\nu\) |
Number of correctly classified defect reports |
\(\delta\) |
Total number of defective cells |
Penalties#
The competition applies penalties for certain events during runs. Penalties are calculated per occurrence - each penalty has an associated value that is deducted from the total score when the event occurs. Penalties come from the following sources:
Non-defective cell placed in inspection bin
Cell falls into conveyor bin
Cell comes into contact with an invalid surface
Two AGVs collide
A robot collides, either with an object in the environment, or another robot
The sensor cost used is over the provided budget
The following penalty symbols are used in the scoring calculations:
\(\rho_{0}\) - Non-defective cell placed in inspection bin
\(\rho_{1}\) - Cell falls into conveyor bin
\(\rho_{2}\) - Object on invalid surface
\(\rho_{3}\) - AGV collision
\(\rho_{4}\) - Robot collision
\(\rho_{5}\) - Sensor cost over budget
Important
The exact penalty values are subject to change and are defined in the scoring_weights.yaml configuration file. Teams should refer to this file for the current values used by the scoring system.
Warning
For the robot collision, for every 5 seconds the robot is in collision, another occurrence of the penalty will be recorded. In addition, if a robot is in collision with another robot, this penalty will be counted twice.
Note
For the sensor cost penalty, the penalty is calculated for each dollar over the allocated budget. Using less than the allocated budget will result in a bonus being applied to the score.
Execution Score#
The execution score aggregates individual run scores across all trials to determine the final performance ranking. Teams complete five runs per trial, with the two best scores from each trial being averaged together. These trial averages are then summed to create the total execution score.
\(E\) |
Total execution score |
\(n\) |
Number of trials |
\(R_1\) |
Best run score for trial i |
\(R_2\) |
Second best run score for trial i |
Human Evaluation#
In addition to the execution score, teams are evaluated by human judges who assess the overall approach and innovation demonstrated during the competition. Judges independently review videos of trial runs and evaluate teams across three categories.
Each team receives scores from 1 to 5 in the following categories:
Novelty/Innovation: Creative and original approaches to solving competition challenges
Feasibility of Approach: Practicality and robustness of the implemented solution
Alignment with Spirit of Competition: How well the approach embodies the goals and values of the competition
The human evaluation score is the sum of all individual judge scores across the three categories:
\(H\) |
Human evaluation score |
\(n\) |
Number of judges |
\(\eta_j\) |
Judge j’s score for Novelty/Innovation (1-5 scale) |
\(\phi_j\) |
Judge j’s score for Feasibility of Approach (1-5 scale) |
\(\alpha_j\) |
Judge j’s score for Alignment with Spirit of Competition (1-5 scale) |
Final Competition Score#
The final competition ranking is determined by converting both execution scores and human evaluation scores to standardized rankings, then combining them with 80% weight for execution performance and 20% weight for human evaluation.
Ranking Process:
Execution Ranking: Teams are ranked by execution score (highest to lowest)
Human Evaluation Ranking: Teams are ranked by human evaluation score (highest to lowest)
Combined Ranking: Rankings are weighted and combined to determine final standings
\(F_{rank}\) |
Final weighted ranking score (lower is better) |
\(R_E\) |
Execution score ranking (1 = best execution score) |
\(R_H\) |
Human evaluation ranking (1 = best human score) |