tocmo0nlord/axolotl
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Merge pull request #451 from OpenAccess-AI-Collective/eval-is-causal

Browse Source 
is_causal fix for evals?
This commit is contained in:
Aman Gupta Karmani
2023-08-21 10:43:46 -07:00
committed by GitHub
parent 58cf7e7fed a213d9972a
commit 5e9c6afa10
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/axolotl/monkeypatch/llama_attn_hijack_flash.py
View File

@@ -158,7 +158,7 @@ def flashattn_forward(
else: else:
# turn off FA causal mask after first inference autoregressive iteration # turn off FA causal mask after first inference autoregressive iteration
# only on first autoregressive step q,k,v have same seqlen # only on first autoregressive step q,k,v have same seqlen
is_causal = past_key_value is not None is_causal = key_states.shape == query_states.shape
if cu_seqlens is not None and max_seqlen is not None: if cu_seqlens is not None and max_seqlen is not None:
# special handling using sample packing # special handling using sample packing