5 Commits

Author	SHA1	Message	Date
NanoCode012	5c7e89105d	Fix: modelloader handling of model_kwargs load_in*bit (#1999) ... * fix: load_in_*bit not properly read * fix: load_*bit check * fix: typo * refactor: load * bit handling * feat: add test dpo lora multi-gpu * fix: turn off sample packing for dpo * fix: missing warmup_steps * fix: test to load in 8bit for lora * skip 8bit lora on h100, add 4bit lora on h100 to multi gpu tests * chore: reduce max_steps --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>	2024-10-30 14:41:34 -04:00
Wing Lian	00568c1539	support for true batches with multipack (#1230) ... * support for true batches with multipack * patch the map dataset fetcher to handle batches with packed indexes * patch 4d mask creation for sdp attention * better handling for BetterTransformer * patch general case for 4d mask * setup forward patch. WIP * fix patch file * support for multipack w/o flash attention for llama * cleanup * add warning about bf16 vs fp16 for multipack with sdpa * bugfixes * add 4d multipack tests, refactor patches * update tests and add warnings * fix e2e file check * skip sdpa test if not at least torch 2.1.1, update docs	2024-02-01 10:18:42 -05:00
Wing Lian	b3a61e8ce2	add e2e tests for checking functionality of resume from checkpoint (#865) ... * use tensorboard to see if resume from checkpoint works * make sure e2e test is either fp16 or bf16 * set max_steps and save limit so we have the checkpoint when testing resuming * fix test parameters	2023-11-15 23:05:55 -05:00
Wing Lian	6dc68a653f	use temp_dir kwarg instead	2023-11-06 18:33:01 -05:00
Wing Lian	c74f045ba7	chore: lint	2023-11-06 18:33:01 -05:00