# Deploy falcon 7b problems
**URL:**
**Author:**  [@Smart2233](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSmart2233)%5C
**Post date:** [June 4, 2023, 9:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploy-falcon-7b-problems%2F42024%2F1 "2023-06-04T09:56:43Z")
Hey, I’m trying to deploy falcon 7b model via SageMaker.
At first I was trying to use the “Deploy Amazon Sage Maker” code from [https://huggingface.co/tiiuae/falcon-7b](https://huggingface.co/tiiuae/falcon-7b):
```auto
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel
try:
role = sagemaker.get_execution_role()
except ValueError:
iam = boto3.client('iam')
role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']
# Hub Model configuration. https://huggingface.co/models
hub = {
'HF_MODEL_ID':'tiiuae/falcon-7b',
'HF_TASK':'text-generation'
}
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
transformers_version='4.26.0',
pytorch_version='1.13.1',
py_version='py39',
env=hub,
role=role,
)
# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1, # number of instances
instance_type='ml.m5.xlarge' # ec2 instance type
)
predictor.predict({
"inputs": "Can you please let us know more details about your ",
})
```
Then i got this error:
```auto
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "InternalServerException",
"message": "Loading /.sagemaker/mms/models/tiiuae__falcon-7b requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code\u003dTrue` to remove this error."
}
```
Then i was trying to create inference endpoint in other way - using custom inference.py i’ve created:
```auto
from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch
model_name = 'tiiuae/falcon-7b'
def model_fn(model_dir):
tokenizer = AutoTokenizer.from_pretrained(model_name)
pipeline = transformers.pipeline(
"text-generation",
model=model_name,
tokenizer=tokenizer,
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="auto",
)
return pipeline, tokenizer
def predict_fn(data, pipeline_and_tokenizer):
pipeline, tokenizer = pipeline_and_tokenizer
sequences = pipeline(
data,
max_length=200,
do_sample=True,
top_k=10,
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id,
)
return sequences
```
After deploying it using this code:
```auto
from sagemaker.huggingface.model import HuggingFaceModel
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
model_data=model_url, # path to your model and script
role=role, # iam role with permissions to create an Endpoint
transformers_version="4.26.0", # transformers version used
pytorch_version="1.13.1", # pytorch version used
py_version='py39', # python version used
model_server_workers=1
)
model_name = 'falcon7bmodel'
# deploy the endpoint endpoint
predictor = huggingface_model.deploy(
initial_instance_count=1,
instance_type="ml.g5.2xlarge",
endpoint_name=model_name + "endpoint",
)
```
I was trying to predict and got this error:
```auto
File "/opt/conda/lib/python3.9/site-packages/transformers/pipelines/text_generation.py", line 205, in preprocess
prefix + prompt_text, padding=False, add_special_tokens=False, return_tensors=self.framework
TypeError: can only concatenate str (not "dict") to str
```
Does someone deployed falcon-7b successfully and can help me here to understand what i’m doing wrong here?
Thanks!
---
**Author:**  [@cw-franabellan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcw-franabellan)%5C
**Post date:** [June 5, 2023, 11:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploy-falcon-7b-problems%2F42024%2F2 "2023-06-05T11:28:49Z")
I have also tried to deploy falcon-7b model in Sagemaker and keep having the same error (`trust_remote_code = True`). Any help would be appreciated!
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [June 5, 2023, 11:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploy-falcon-7b-problems%2F42024%2F3 "2023-06-05T11:32:02Z")
cc @philschmid
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [June 6, 2023, 7:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploy-falcon-7b-problems%2F42024%2F4 "2023-06-06T07:24:58Z")
@Smart2233 it seems that you `predict_fn` is incorrect `data` will be a dictionary with “inputs”, see [Deploy FLAN-UL2 20B on Amazon SageMaker](https://www.philschmid.de/deploy-flan-ul2-sagemaker) for how to fix it.
@cw-franabellan the code @Smart2233 should work with the change in the `infernece.py`
---
**Author:**  [@Smart2233](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSmart2233)%5C
**Post date:** [June 6, 2023, 7:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploy-falcon-7b-problems%2F42024%2F5 "2023-06-06T07:49:09Z")
@philschmid Thanks for the answer! extracting inputs from data did fix the issue.
Although I still not sure why my first approach did not worked.(trust\_remote\_code error)