Reproducibility is not always a linear process. Many researchers perform optimization steps that can improve the computation efficiency. A common one involves running numerous different processes of steps at the same time and then summing the result together. For example, if I wanted to sum all the natural numbers (e.g., 1, 2, 3, etc.) up to 1,000,000 together, I could instead have 20 processes sum 50,000 each and then add the final subtotals together to speed the process up. However, operations that are performed at the same time, or concurrently, almost always need to be commutative; otherwise, the results may differ across runs.
The Commutative Property
The commutative property is typically something you learn within primary education. An operation is considered commutative if the order of the operands does not change the result. For example:
This can be generalized further to any binary operation `✱
` for some set `S
`:
This states that for any element x and y in the same set, the operation `✱` is commutative if `x ✱ y = y ✱ x`
. In the first example, the set is all natural numbers, so the commutative property would hold true when adding any positive numbers together.
Non-Commutative Operations
Of course, not all operations are commutative. For example, dividing two numbers is not commutative:
Another example, which is likely more relevant for researchers, is matrix multiplication:
As such, if the order of these operands could not be guaranteed, then the results may differ across runs. For these cases, you either need to guarantee that the operands will always be in the same order *or* convert the operation into a commutative one. Once again using division, if we rewrite the the equation using multiplication and rational numbers, the operation is now commutative:
Anti-Commutative Operations
Some non-commutative operations actually have an interesting property where the result is the inverse when swapping the arguments, which known as being anti-commutative. For example, subtracting two numbers is anti-commutative:
This is also true when taking the cross product of two vectors:
With anti-commutative operations, you can also take the absolute value of the values to make the results consistent, along with the previous options:
Now you know some of the pitfalls of reproducibility with commutative operands!
Some Additional Thoughts
Some Matrix Multiplications are Commutative
Some matrices when multiplied together are, in fact, commutative. The identity matrix commutes with any matrix. Additionally, if the product of two symmetric matrices is itself symmetric, then the operation is commutable. Of course, it would be more appropriate to say that the product of two symmetric matrices is transpose-commutative as: