tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Feat: Add dataset loading from S3, GCS (#765)

Browse Source 
* Feat: Add dataset loading from S3, GCS

* chore: update docs

* chore: add more info on cloud loading
This commit is contained in:
NanoCode012
2023-11-16 14:33:58 +09:00
committed by GitHub
parent 1bc11868eb
commit 3cc67d2cdd
3 changed files with 110 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
requirements.txt
View File

@@ -11,7 +11,7 @@ deepspeed
addict
fire
PyYAML>=6.0
datasets
datasets>=2.14.0
flash-attn>=2.3.0
sentencepiece
wandb
@@ -33,3 +33,8 @@ art
fschat==0.2.29
gradio
tensorboard
# remote filesystems
s3fs
gcsfs
# adlfs