Scoring

There are 3 major components of the Trial Score in ARIAC 2023.

1. Cost Factor: How much does the system (sensors) cost?

2. Efficiency Factor: How fast or efficiently did the system complete the task(s)

3. Completion Score: How well did the task(s) get performed? Are all the correct parts there in the proper place?

Cost Factor

The Cost Factor \(CF\) compares the cost of the sensors chosen by the team to the average of all sensor configurations across all teams.

• \(TC\) is the total cost of the sensors in the team’s configuration.

• \(TC_{avg}\) is the average sensor cost across all teams.

• \(w_c\) is a weighting constant for cost factor.

\[CF = w_c \cdot \frac{TC_{avg}}{TC}\]

Efficiency Factor

The Efficiency Factor \(EF_i\) for order \(i\) compares the time to complete order \(i\) for the team to the average of all teams’s times to complete order \(i\).

• \(T_i\) is the time to complete order \(i\)

• \(T_{avg_{i}}\) is the average time to complete order \(i\) for all teams

• \(w_t\) is a weighting constant for efficiency factor.

\[EF_i = w_t \cdot \frac{TC_{avg_{i}}}{T_i}\]

Completion Score

Completion score varies between Kitting, Assembly, and Combined tasks. Each task is generated from Boolean conditions.

Kitting Task Score

• A kitting task has \(n\) parts that need to be placed on the kitting tray.

• A shipment has \(m\) parts on the kitting tray.

• For each task there are two Boolean conditions:

  1. \(isCorrectTrayID \rightarrow A\) is true if the shipment tray ID matches the kitting tray ID.

  2. \(isCorrectDestination\) is true if the shipment was sent to the correct destination (as1, as2, as3, as4, kitting, or warehouse).

• For each quadrant q of the kitting tray there are four Boolean conditions:

  1. \(isCorrectType_{q} \rightarrow B\) is true if the part type in quadrant \(q\) is correct.

  2. \(isCorrectColor_{q} \rightarrow C\) is true if the part color in quadrant \(q\) is correct.

  3. \(isFlipped_{q} \rightarrow D\) is true if the part in quadrant \(q\) is still flipped.

  4. \(isFaulty_{q} \rightarrow E\) is true if the part in quadrant \(q\) is faulty.

Tray Score

\[\begin{split}\texttt{pt}_{tray} = \begin{cases} 3, &\text{if} ~~ A \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Quadrant Score

\[\begin{split}\texttt{pt}_q = \begin{cases} 0, &\text{if} ~~ \lnot B \lor E \\ 3, &\text{if} ~~ B \land C \land \lnot D \land \lnot E\\ 2, &\text{if} ~~ B \land \lnot C \land \lnot D \land \lnot E\\ 2, &\text{if} ~~ B \land C \land D \land \lnot E\\ 1, &\text{if} ~~ B \land \lnot C \land D \land \lnot E\\ \end{cases}\end{split}\]

Bonus Score

\[\begin{split}\texttt{pt}_b = \begin{cases} n, &\text{if} ~~ \sum_{q}^{n}{\texttt{pt}_q} = n\times 3 \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Penalty

A penalty is only applied if more parts are on the tray than needed.

\[\begin{split}\texttt{pn}_{ep} = \begin{cases} m - n, &\text{if} ~~ m>n \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Destination Score

\[\begin{split}\texttt{destination} = \begin{cases} 1, &\text{if}\, isCorrectDestination\, \text{is true} \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Kitting Task Score

\[S_{k} = (\max{[\texttt{pt}_{tray} + \sum_{q}^{n}{\texttt{pt}_q} + \texttt{pt}_b - \texttt{pn}_{ep} , 0]}) \times (\texttt{destination})\]

Assembly Task Score

• An assembly task has \(n\) parts that need to be assembled into the insert.

• For each task there is one Boolean condition:

  1. \(isCorrectStation\) is true if the assembly was done at the correct station (as1, as2, as3, or as4).

• Each slot s in the insert has the following Boolean conditions:

  1. \(isAssembled_{s} \rightarrow A\) is true if the part in slot \(s\) is assembled. This implicitly means that the part is of the correct type.

  2. \(isCorrectColor_{s} \rightarrow B\) is true if the part in slot \(s\) is of correct color.

  3. \(isCorrectPose_{s} \rightarrow C\) is true if the part in slot \(s\) has the correct pose.

Slot Score

\[\begin{split}\texttt{pt}_s = \begin{cases} 3, &\text{if} ~~ A \land (B \land C)\\ 2, &\text{if} ~~ A \land (B \lor C)\\ 1, &\text{if} ~~ A \land (\lnot B \land \lnot C)\\ 0, &\text{if} ~~ \lnot A \\ \end{cases}\end{split}\]

Bonus Score

\[\begin{split}\texttt{pt}_b = \begin{cases} n \times 4, &\text{if} ~~ \sum_{s}^{n}{\texttt{pt}_{s}} = n\times 3 \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Station Score

\[\begin{split}\texttt{station} = \begin{cases} 1, &\text{if}\, isCorrectStation\, \text{is}\, \text{true} \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Assembly Task Score

\[S_{a} = (\sum_{s}^{n}{\texttt{pt}_s} + \texttt{pt}_b) \times (\texttt{station})\]

Combined Task Score

• A combined task has \(n\) parts that need to be gathered from the environment and assembled to the insert.

• For each task there is one Boolean condition:

  1. \(isCorrectStation\) is true if the assembly was done at the correct station (as1, as2, as3, or as4).

• Each slot s in the insert has the following Boolean conditions:

  1. \(isAssembled_{s} \rightarrow A\) is true if the part in slot \(s\) is assembled. This implicitly means that the part is of the correct type.

  2. \(isCorrectColor_{s} \rightarrow B\) is true if the part in slot \(s\) is of correct color.

  3. \(isCorrectPose_{s} \rightarrow C\) is true if the part in slot \(s\) has the correct pose.

Slot Score

\[\begin{split}\texttt{pt}_s = \begin{cases} 0, &\text{if} ~~ \lnot A \\ 5, &\text{if} ~~ A \land (B \land C)\\ 4, &\text{if} ~~ A \land (B \lor C)\\ 3, &\text{if} ~~ A \land (\lnot B \land \lnot C)\\ \end{cases}\end{split}\]

Bonus Score

\[\begin{split}\texttt{pt}_b = \begin{cases} n \times 4, &\text{if} ~~ \sum_{s}^{n}{\texttt{pt}_{s}} = n\times 5 \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Station Score

\[\begin{split}\texttt{station} = \begin{cases} 1, &\text{if}\: isCorrectStation\: \text{is true} \\ 0, &\text{otherwise} \\ \end{cases}\end{split}\]

Combined Task Score

\[S_{c} = (\sum_{s}^{n}{\texttt{pt}_s} + \texttt{pt}_b) \times (\texttt{station})\]

Completion Score

The final completion score \(CompletionScore\) combines the kitting, assembly, and combined task scores present in that trial.

Completion Score

\[CompletionScore = \sum_{i=0}^{n_k}{S_{k_i}} + \sum_{j=0}^{n_a}{S_{a_j}} + \sum_{k=0}^{n_c}{S_{c_k}}\]

Trial Score

The trial score \(TrialScore\) combines the cost factor, efficiency factors and completion scores into a single score for ranking the teams.

Trial Score

\[TrialScore = CF \times \sum_{i=0}^{n}{(h_i \times EF_i \times CS_i)}\]