TRF062
_init_weights must initialize tensors with the transformers initialization module, not torch.nn.init.
| Default | Enabled |
| Scope | All models |
| Source | mlinter/trf062.py |
| Show in terminal | mlinter --rule TRF062 |
What it does
Flags in-place torch.nn.init initializers (nn.init.normal_(...), however imported) called inside _init_weights. Helpers that only compute a value, such as calculate_gain, are not flagged.
Why is this bad?
The transformers init functions skip tensors already loaded from a checkpoint. torch.nn.init relies on a patch applied during from_pretrained, which a function imported by name escapes, so a loaded weight can be silently re-initialized.
Example
from torch import nn
+
+from ... import initialization as init
def _init_weights(self, module):
super()._init_weights(module)
if isinstance(module, AcmeLayerScale):
- nn.init.constant_(module.lambda1, self.config.layer_scale_init_value)
+ init.constant_(module.lambda1, self.config.layer_scale_init_value)
Suppressing this rule
Add a # trf-ignore: TRF062 comment on the flagged line or the line directly above it. See Suppressing rules for whole-file directives and when a suppression is the wrong answer.
