Making the output of LLAMA Server exact the same like LLAMA in the instruct mode

[alexcardo]

Please help, help, help! I spent months in an attempt to solve this issue. I asked this question many times and nobody answered. I can't believe that no one is interested in solving this issue.

I can't tune up llama server. No, it's quite simple, but the response is always awful with the same prompt. LLAMA Sever doesn't work like LLAMA INSTRUCTION MODE. Meanwhile, when I run llama.cpp in the instruction mode I receive exactly what I want. Below, I'll give you examples:

1. Llama.cpp INSTRUCTION mode:

alex@M1 llama.cpp % ./main -m ~/ai/mistral-7b-instruct-v0.2.Q4_K_M.gguf -ins --color --multiline-input -ngl 99

Write me a short description of the Lancome brand
Lancôme is a renowned French luxury beauty brand, established in Paris in 1935 by Armand Petitjean. Known for its iconic fragrances, makeup, and skincare products, Lancôme has been synonymous with elegance, sophistication, and innovation for nearly a century. The brand's offerings cater to both men and women, focusing on enhancing natural beauty through high-performance formulas and luxurious textures. From its iconic fragrances like Tresor and La Vie Est Belle, to its bestselling makeup collections such as Teint Idole foundation and Artliner Precision Point eyeliner, Lancôme continues to set industry standards and inspire confidence with each new product launch. The brand's commitment to sustainability and ethical business practices further solidifies its position as a leader in the beauty industry.

Another example:

Write me a short description of the Lancome brand
Lancôme is a French luxury beauty brand known for its high-quality skincare, makeup, and fragrances. Established in Paris in 1935 by Armand Petitjean, it was one of the first perfume houses to create exclusively feminine fragrances. The brand's name is derived from the Latin word "lanscum," meaning "poplar tree bud," symbolizing rebirth and renewal. Lancôme offers a wide range of products including foundations, concealers, powders, mascaras, eyeliners, lipsticks, blushes, skincare items, and perfumes. The brand is renowned for its sophisticated and elegant packaging, as well as its commitment to innovation and research in beauty. Lancôme has been the preferred choice of numerous celebrities and fashion icons throughout history, and is currently part of the L'Oréal Luxe division.

Everything is correct. Next, I'm trying to do the same with LLAMA Server:

~/ai/llama.cpp$ ./server -m models/mistral-7b-instruct-v0.2.Q4_K_M.gguf -c 2048 --host XXX.XXX.XXX.XX --port 8000

Ok, let's do it with the most simple code and make an API request:

import requests
import json

url = "http://XXX.XXX.XXX.XX:8000/completion"
headers = {"Content-Type": "application/json"}
data = {
        "prompt": "Write me a short description of the Lancome brand\n",
        'temperature': 0.8,
        'top_k': 40,
        'top_p': 0.95,
        'repeat_penalty': 1.1,
        'tfs_z': 1,
        'min_p': 0.05,
        'typical_p': 1
    }

response = requests.post(url, headers=headers, json=data)

print(response.json()['content'])

Below are the examples of the response:

Lancome is a French luxury cosmetics brand, owned by the L'Oréal Group. It was founded in Paris in 1935 by Armand Petitjean and named after his girlfriend, Marie-Antoinette Lancret-de-Fortuny. Lancome quickly gained popularity for its elegant packaging and high-quality products, including fragrances, makeup, and skincare. The brand is known for its iconic products such as the Hypnose mascara, Absolue foundation, and La Vie Est Belle perfume. With a focus on innovation and research, Lancome continues to create advanced formulas and chic designs that cater to women's diverse beauty needs. The brand is also committed to sustainability, using eco-friendly packaging and ingredients whenever possible.

WTF? named after his girlfriend, Marie-Antoinette Lancret-de-Fortuny Where did the model get these garbish data?

Lancome is a luxury French cosmetics, perfumes, and skincare brand founded in 1935 by Armand Petitjean. The name "Lancome" is derived from the Latin word "lanka," meaning "slim," and "komos," meaning "beauty party." Initially focusing on makeup, Lancome soon expanded its offerings to include fragrances and skincare products. With a commitment to innovation and high-quality ingredients, Lancome has become synonymous with elegance and sophistication. Their products are sold in over 130 countries worldwide and are beloved by celebrities, fashion influencers, and beauty enthusiasts alike. From iconic fragrances like La Vie Est Belle to their best-selling skincare line, Lancome continues to push boundaries and set trends in the world of beauty.

WTF? from the Latin word "lanka," meaning "slim," and "komos,"

Lancome is a renowned French luxury beauty brand that was established in Paris in 1935. It was founded by Armand Petitjean, who named the company after his favorite flower, the orchid (l'Orchidee in French). Lancome quickly gained popularity for its innovative skincare products and fragrances, using only the finest ingredients to create high-quality formulations. The brand prides itself on its commitment to scientific research and development, continuously pushing the boundaries of beauty technology. Lancome's extensive product range includes makeup, skincare, perfumes, and accessories, catering to both men and women. Its iconic products include the advanced skincare line, the famous Teint Idole foundation, and best-selling fragrances such as La Vie Est Belle and Idole. With a rich heritage, Lancome continues to be a leader in the global beauty industry, inspiring and empowering people to express their unique beauty.

WTF? who named the company after his favorite flower, the orchid (l'Orchidee in French)

Finally, example with CURL (how it was done in the official instruction).

alex@M1 remote-llama % curl --request POST \
    --url http://XXX.XXX.XXX.XX:8000/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Write me a short description of the Lancome brand\n"}'
{"content":"\nLancome is a French luxury beauty brand that was founded in Paris in 1935 by Armand Petitjean. The name Lancome is derived from the Latin word for lake, \"lacuna,\" and the French word for beauty, \"beau.\" The brand has a rich history of innovation in skincare, makeup, and fragrances. Its iconic products include the Advanced Génifique serum, Hydra-Absolue cream, and Artliner liquid eyeliner. Lancome's commitment to excellence and beauty is reflected in its slogan \"where art meets science.\" The brand is known for collaborating with top international makeup artists and influencers to create trendsetting looks and collections. Lancome products are sold worldwide in department stores, specialty beauty shops, and online at lancome.com.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"models/mistral-7b-instruct-v0.2.Q4_K_M.gguf","n_ctx":2048,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"models/mistral-7b-instruct-v0.2.Q4_K_M.gguf","prompt":"Write me a short description of the Lancome brand\n","slot_id":0,"stop":true,"stopped_eos":true,"stopped_limit":false,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":26494.377,"predicted_n":180,"predicted_per_second":6.793894417672097,"predicted_per_token_ms":147.19098333333335,"prompt_ms":812.901,"prompt_n":12,"prompt_per_second":14.76194518151657,"prompt_per_token_ms":67.74175},"tokens_cached":192,"tokens_evaluated":12,"tokens_predicted":180,"truncated":false}%

... again, hallucinations: The name Lancome is derived from the Latin word for lake, "lacuna," and the French word for beauty, "beau."

While the output on the same remote machine using the instruct mode:

Write me a short description of the Lancome brand
Lancôme is a renowned French cosmetics brand that has been captivating beauty enthusiasts since its inception in 1935. Known for its sophisticated elegance and commitment to innovation, Lancôme offers an extensive range of high-quality products designed to enhance the natural beauty of women and men around the world. From iconic fragrances like "Trésor" and "La Vie Est Belle," to their renowned skincare lines such as "Advanced Génifique" and "Absolue", Lancôme's extensive collection caters to various needs and preferences. The brand prides itself on its commitment to research and development, continuously pushing boundaries with groundbreaking formulas and advanced technologies. With a rich heritage and an unwavering dedication to beauty, Lancôme continues to inspire confidence and allure for generations.

Why every basic instruct response writes the brand name correctly Lancôme while the server's response writes Lancome?

In other words, every time I make an API request to the server, I experience the hallucinations of the model, while in the instruction mode it behaves correctly.

Why does it happen? Why I can't get exactly the same (quality) response from the LLAMA server, while LLAMA.CPP gives me exactly what I want in the instruction mode. I've been trying this with numerous queries and completely the same parameters but had no quality response with LLAMA Server.

P.s.: I also mentioned that the strange behavior (hallucinations) happen only in the Completion mode. Even if I use the web interface for LLAMA Server, everything works just fine in the Chat mode. I suppose, LLAMA.CPP drops some crucial parameters in the completion mode.

[FSSRepo]

The server application has some limitations and does not work as expected in instruction mode.

You should keep in mind that you need to pass the exact same prompt template for the model to give the best responses. Could you provide an example?

[qnixsynapse]

Try surrounding the prompt with [INST] [/INST] tags.