6 Commits

Author	SHA1	Message	Date
Wing Lian	628b754824	set output_router_logits for mixtral config: (#995)	2023-12-22 12:57:02 -05:00
Wing Lian	5ea3aa31f0	Fix Deepspeed loading (#950) ... * add check for zero3 * freeze parameters * fixes for deepspeed loading * fix model parameter check * unfrozen parameters in example mixtral and logging when unfreezing	2023-12-13 16:03:23 -05:00
Wing Lian	5f79b8242f	new evals_per_epoch and saves_per_epoch to make things cleaner (#944) ... * new evals_per_epoch and saves_per_epoch to make things cleaner * update per PR feedback	2023-12-12 15:35:23 -05:00
Wing Lian	7fabc4d95e	Mixtral official (#942) ... * multipack support for official mixtral implementation * fix patch to load multipack for mixtral * chore: lint	2023-12-11 23:44:33 -05:00
Wing Lian	35f9b0f149	update to latest transformers for mixstral support (#929) ... * update to latest transformers for mixstral support * pin transformers * fix typo	2023-12-10 10:32:27 -05:00
Wing Lian	68b227a7d8	Mixtral multipack (#928) ... * mixtral multipack * use mixtral model * sample yml * calculate cu_seqlens properly * use updated flash ettention setting * attn var checks * force use of flash attention 2 for packing * lint * disable future fix for now * update support table	2023-12-09 21:26:30 -05:00