Instructions to use meta-llama/LlamaGuard-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use meta-llama/LlamaGuard-7b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="meta-llama/LlamaGuard-7b") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("meta-llama/LlamaGuard-7b") model = AutoModelForCausalLM.from_pretrained("meta-llama/LlamaGuard-7b") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use meta-llama/LlamaGuard-7b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "meta-llama/LlamaGuard-7b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "meta-llama/LlamaGuard-7b", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/meta-llama/LlamaGuard-7b
- SGLang
How to use meta-llama/LlamaGuard-7b with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "meta-llama/LlamaGuard-7b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "meta-llama/LlamaGuard-7b", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "meta-llama/LlamaGuard-7b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "meta-llama/LlamaGuard-7b", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use meta-llama/LlamaGuard-7b with Docker Model Runner:
docker model run hf.co/meta-llama/LlamaGuard-7b
Release few-shot prompting example
"""The guidelines that Llama Guard is trained on may not be the same as the desired guidelines for the target
domain. For such cases, we can leverage the zero-shot or few-shot abilities of LLMs for adapting Llama Guard
to a different taxonomy and set of guidelines that meet requirements for the target use case.
Zero-shot prompting involves using category names, or category names as well as category descriptions of the
target domain in the prompt at inference time.
Few-shot prompting is similar to zero-shot but additionally includes 2 to 4 examples for each category in the
prompt. The learning happens in-context, i.e., we do not train on these examples. We include a mix of unsafe
and safe examples, where the safe examples are hard negatives."""
Could you please provide the template or an example of the few-shot prompting used in the paper?
Thanks!