Chapter 12: bitsandbytes error

[avafor]

Running this part of the code in chapter 12, I get the following error. Although I have the latest version of bitsandbytes.

`from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"

bnb_config = BitsAndBytesConfig(
load_in_4bit=True, # Use 4-bit precision model loading
bnb_4bit_quant_type="nf4", # Quantization type
bnb_4bit_compute_dtype="float16", # Compute dtype
bnb_4bit_use_double_quant=True, # Apply nested quantization
)

model = AutoModelForCausalLM.from_pretrained(
model_name,
device_map="auto",

# Leave this out for regular SFT
quantization_config=bnb_config,

)
model.config.use_cache = False
model.config.pretraining_tp = 1

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = ""
tokenizer.padding_side = "left"`

71 raise ImportError( 72 f"Usingbitsandbytes4-bit quantization requires Accelerate:pip install 'accelerate>={ACCELERATE_MIN_VERSION}'" 73 ) 74 if not is_bitsandbytes_available(): ---> 75 raise ImportError( 76 "Using bitsandbytes4-bit quantization requires the latest version of bitsandbytes:pip install -U bitsandbytes`"
77 )
79 from ..integrations import validate_bnb_backend_availability
80 from ..utils import is_bitsandbytes_multi_backend_available

ImportError: Using bitsandbytes 4-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes``

Do you have any suggestions on how to debug the issue?
Thank you very much in advance.

[MaartenGr]

Apologies for the delay. I just ran the Colab notebook and did not encounter any issues using the pip install ... instructions shown at the top. What kind of environment are you working in?

[paulinhok14]

In which OS are you running it @avafor ?
If it is on Windows, I highly recommend you shift to 8-bit quantization if possible. There are so many bugs with 4-bit quant on it.