Description
The function concrete_bern() applies a sigmoid operation independently to each entry of the probability matrix.
As a consequence, each row is not normalized, and its values are not constrained to sum to one, as required by Equation (2).
Observed behavior during training
Examples of row outputs:
[1.0, 0.87]
[0.0008, 0.0]
This indicates that rows do not represent valid probability distributions.
Impact on formulation
Due to the lack of normalization, the resulting matrix multiplication M M^T may produce diagonal entries greater than 1.
This contradicts the formulation described in the paper for the separation term, where the matrix is expected to preserve probabilistic consistency.
Question
Should a row-wise normalization step be applied after concrete_bern() to ensure consistency with Equation (2)?
Description
The function
concrete_bern()applies a sigmoid operation independently to each entry of the probability matrix.As a consequence, each row is not normalized, and its values are not constrained to sum to one, as required by Equation (2).
Observed behavior during training
Examples of row outputs:
This indicates that rows do not represent valid probability distributions.
Impact on formulation
Due to the lack of normalization, the resulting matrix multiplication
M M^Tmay produce diagonal entries greater than 1.This contradicts the formulation described in the paper for the separation term, where the matrix is expected to preserve probabilistic consistency.
Question
Should a row-wise normalization step be applied after
concrete_bern()to ensure consistency with Equation (2)?