Hello, I had similar issue as you did when loading in as 4 bit. Just curious if you found a solution yet?
Mike Song
mikesong724
AI & ML interests
None yet
Recent Activity
commented on
an
article
27 days ago
No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL
updated
a model
over 3 years ago
mikesong724/deberta-wiki-2010
updated
a model
over 3 years ago
mikesong724/deberta-wiki-2006
Organizations
None yet
commented on
No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL
27 days ago
updated
2
models
over 3 years ago