8779997ba5
* native support for modal cloud from CLI * do lm_eval in cloud too * Fix the sub call to lm-eval * lm_eval option to not post eval, and append not extend * cache bust when using branch, grab sha of latest image tag, update lm-eval dep * allow minimal yaml for lm eval * include modal in requirements * update link in README to include utm * pr feedback * use chat template * revision support * apply chat template as arg * add wandb name support, allow explicit a100-40gb * cloud is optional * handle accidental setting of tasks with a single task str * document the modal cloud yaml for clarity [skip ci] * cli docs * support spawn vs remote for lm-eval * Add support for additional docker commands in modal image build * cloud config shouldn't be a dir * Update README.md Co-authored-by: Charles Frye <cfrye59@gmail.com> * fix annotation args --------- Co-authored-by: Charles Frye <cfrye59@gmail.com>
18 lines
372 B
Python
18 lines
372 B
Python
"""
|
|
Module for handling lm eval harness input arguments.
|
|
"""
|
|
from typing import List, Optional
|
|
|
|
from pydantic import BaseModel
|
|
|
|
|
|
class LMEvalArgs(BaseModel):
|
|
"""
|
|
Input args for lm eval harness
|
|
"""
|
|
|
|
lm_eval_tasks: List[str] = []
|
|
lm_eval_batch_size: Optional[int] = 8
|
|
lm_eval_post_train: Optional[bool] = True
|
|
lm_eval_model: Optional[str] = None
|