TRF062

_init_weights must initialize tensors with the transformers initialization module, not torch.nn.init.

   
Default Enabled
Scope All models
Source mlinter/trf062.py
Show in terminal mlinter --rule TRF062

What it does

Flags in-place torch.nn.init initializers (nn.init.normal_(...), however imported) called inside _init_weights. Helpers that only compute a value, such as calculate_gain, are not flagged.

Why is this bad?

The transformers init functions skip tensors already loaded from a checkpoint. torch.nn.init relies on a patch applied during from_pretrained, which a function imported by name escapes, so a loaded weight can be silently re-initialized.

Example

 from torch import nn
+
+from ... import initialization as init


 def _init_weights(self, module):
     super()._init_weights(module)
     if isinstance(module, AcmeLayerScale):
-        nn.init.constant_(module.lambda1, self.config.layer_scale_init_value)
+        init.constant_(module.lambda1, self.config.layer_scale_init_value)

Suppressing this rule

Add a # trf-ignore: TRF062 comment on the flagged line or the line directly above it. See Suppressing rules for whole-file directives and when a suppression is the wrong answer.