How can i correctly use a CPU to perform inference of a quantized model

[neavo]

I made some attempts, such as:

self.model = AutoModelForTokenClassification.from_pretrained(
    "resource/kg_ner_gpu",
    device_map = "cpu",
    torch_dtype = torch.float16,
    quantization_config = BitsAndBytesConfig(
        load_in_4bit = True,
        load_in_8bit = False,
        bnb_4bit_quant_type = "nf4",
        bnb_4bit_compute_dtype = torch.float16,
    ),
    local_files_only = True,
    low_cpu_mem_usage = True,
)

self.classifier = pipeline(
    "token-classification",
    model = self.model,
    tokenizer = self.tokenizer,
    aggregation_strategy = "simple",
)

But the speed is very slow, so is there a correct code snippet as an example?