the number of encoder and decoder blocks is wrong in the code

[AvAkanksh]

Kronos/model/kronos.py

Lines 59 to 68 in d5ffd46

	# Encoder Transformer Blocks
	self.encoder = nn.ModuleList([
	TransformerBlock(self.d_model, self.n_heads, self.ff_dim, self.ffn_dropout_p, self.attn_dropout_p, self.resid_dropout_p)
	for _ in range(self.enc_layers - 1)
	])
	# Decoder Transformer Blocks
	self.decoder = nn.ModuleList([
	TransformerBlock(self.d_model, self.n_heads, self.ff_dim, self.ffn_dropout_p, self.attn_dropout_p, self.resid_dropout_p)
	for _ in range(self.dec_layers - 1)
	])

the number of encoder and decoder blocks which are created are one less than mentioned because u have used range(self.enc_layers -1) and range(self.dec_layers -1)

so i think this needs a change in the code.

please tell me if i am wrong.

[AvAkanksh]

@shiyu-coder

[gtktreegrid]

Only N-1 transformer blocks are needed to reach N total encoding layers

[AvAkanksh]

@gtktreegrid y so ? From where do we get that one extra encoding block ? Please let me know I am not sure of it

[randyy179]

I agree @AvAkanksh , the inconsistency here is strange. @shiyu-coder

[gtktreegrid]

Maybe a bad drawing but something like:

post_quant_embed → TransformerBlock × 3 → head (linear)
_______________________└─── 3 blocks ───┘ └─ "4th layer"─┘

[mvanhorn]

Implemented in #248 - changed range(self.enc_layers - 1) to range(self.enc_layers) (same for decoder) so the configured layer count matches the actual number of TransformerBlocks created.

[aino-debasish]

Maybe a bad drawing but something like:

post_quant_embed → TransformerBlock × 3 → head (linear) _______________________└─── 3 blocks ───┘ └─ "4th layer"─┘

so the head layer is accounting for the missing one ! sounds reasonable