You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
13 lines
228 B
13 lines
228 B
{
|
|
"attn_dropout_p": 0.1,
|
|
"d_model": 512,
|
|
"ff_dim": 1024,
|
|
"ffn_dropout_p": 0.25,
|
|
"learn_te": true,
|
|
"n_heads": 8,
|
|
"n_layers": 8,
|
|
"resid_dropout_p": 0.25,
|
|
"s1_bits": 10,
|
|
"s2_bits": 10,
|
|
"token_dropout_p": 0.1
|
|
}
|