# Train a CausalLM for machine translation **URL:** **Author:** ![AntonioMastroianni](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/antoniomastroianni/32/38400_2.png) [@AntonioMastroianni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAntonioMastroianni)%5C **Post date:** [January 1, 2025, 5:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-a-causallm-for-machine-translation%2F133865%2F1 "2025-01-01T17:45:58Z") For an experiment of mine, I am trying to train from scratch a causalLM, in particular `Qwen/Qwen2.5-0.5B-Instruct`, for a machine translation task. Since this an experiment and I am aware that achieving good performances would require both a great amount of time and resources, I decided to use as training dataset around 25% of my data (around 30000 observations). My idea was to train the model to solve the fllowing task specifically: \ \ source sentence \ \ target sentence \. Unfortunately, after training the model, giving it new sentences as input results in getting back the same source sentence plus random words if max\_length hasn’t been reached. What I would like to know if this this just caused by the model not being trained on enough data or if there is an error in setting my code right. After performing data preprocessing, I transform data using: ````auto model = "Qwen/Qwen2.5-0.5B-Instruct"``` ```` ```auto def preprocess(example): source = example['Source_clean'] target = example['Target_clean'] # Combine source and target into a single sequence sequence = ( f" {source} " f" {target} " ) return {"sequence": sequence} # Apply preprocessing dataset = dataset.map(preprocess) ``` Add special tokens to tokenizer ```auto tokenizer = AutoTokenizer.from_pretrained(model) tokenizer.pad_token = tokenizer.eos_token tokenizer.add_special_tokens({ 'additional_special_tokens': [ '', '', '', '' ] }) ``` Then I apply tokenization ```auto def tokenize_function(example): # Tokenize the combined input sequence = example['sequence'] # Tokenize the sequence using the updated tokenizer tokenized = tokenizer(sequence, padding='max_length', truncation=True, max_length=96) # You can adjust max_length as needed return tokenized # Apply tokenization tokenized_dataset = dataset.map(tokenize_function, batched=True) tokenized_dataset = tokenized_dataset.map( lambda example: example, remove_columns=['Source_clean', 'Target_clean', 'sequence'] ) ``` So that `input_ids` and `attention_mask` are the only two feaures in the data left. **Model Setup** ```auto #model setup config = AutoConfig.from_pretrained(model) #config.num_hidden_layers = 12 # Ensure the model configuration is set up for causal language modeling config.is_decoder = True # Set the model as a decoder for causal language modeling config.add_cross_attention = False # Optional, only for encoder-decoder models model = AutoModelForCausalLM.from_config(config) ``` **Callbacks** ```auto class ClearMemoryCallback(TrainerCallback): def on_epoch_end(self, args, state, control, **kwargs): print(f"Clearing GPU memory after epoch {state.epoch}...") torch.cuda.empty_cache() gc.collect() # Inside callbacks class StateCallback(TrainerCallback): def on_epoch_end(self, args, state, control, metrics=None, **kwargs): logging.info(f"Epoch {state.epoch} ended.") if metrics: logging.info(f"Metrics: {metrics}") ``` ```auto # Set data collator data_collator = DataCollatorForLanguageModeling( tokenizer=tokenizer, mlm=False, # Causal model doesn't use masked LM pad_to_multiple_of=8 ) # Set training arguments training_args = TrainingArguments( output_dir="/content/drive/MyDrive/qwen-translation2/checkpoint", eval_strategy="steps", eval_steps=500, save_steps=500, logging_steps=100, per_device_train_batch_size=2, per_device_eval_batch_size=2, gradient_accumulation_steps=64, gradient_checkpointing = True, num_train_epochs=3, learning_rate=5e-5, warmup_steps=500, weight_decay=0.01, save_total_limit=3, fp16=True, # Mixed precision for faster training push_to_hub=False, max_grad_norm=0.1, load_best_model_at_end=True, metric_for_best_model='eval_loss', ) model.gradient_checkpointing_enable() ``` ```auto # Training setup from transformers import Trainer optimizer = Adam8bit(model.parameters(), lr=5e-5) trainer = Trainer( model=model, args=training_args, data_collator=data_collator, train_dataset=tokenized_dataset['train'], eval_dataset=tokenized_dataset['test'], tokenizer = tokenizer, optimizers=(optimizer, None), callbacks=[ClearMemoryCallback()] ) ``` ```auto # Move model and data to the same device device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = model.to(device) model.config.use_cache = False logging.info("Starting training...") trainer.train() logging.info("Finished training.") ``` After training, if I don’t set `save_safetensors = 'False` the traning stops just before the end of the last epoch and I get the following warning: > There were missing keys in the checkpoint model loaded: [‘lm\_head.weight’] **Model Testing** ```auto folder = "/content/drive/MyDrive/qwen-translation2/checkpoint" # Load the model and tokenizer from the folder trained_model = AutoModelForCausalLM.from_pretrained(folder) trained_tokenizer = AutoTokenizer.from_pretrained(folder) # Set the model to evaluation mode model.eval() ``` ```auto prompt = "Once upon a time in a faraway land" # Ensure that attention mask and pad_token_id are set correctly inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True) # Add attention mask to the generation function outputs = model.generate( input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"], # Pass attention mask max_length=96, # Maximum length of generated text early_stopping=True, num_return_sequences=1, # Generate a single output no_repeat_ngram_size=2, # Prevent repeating n-grams temperature=0.7, # Sampling temperature (lower is more deterministic) top_p=0.9, # Nucleus sampling (controls diversity) top_k=50, # Limits the sampling to the top 50 tokens do_sample=True, # Enable sampling pad_token_id=model.config.pad_token_id # Explicitly set pad_token_id ) generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True) generated_text ``` Could you help me in solving this issue? Thanks for the attention ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [January 1, 2025, 11:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-a-causallm-for-machine-translation%2F133865%2F2 "2025-01-01T23:22:47Z")
Maybe unresolved issue? Or maybe it hasn’t been made into an issue on the transformers github. > **[There were missing keys in the checkpoint model loaded: \['proj\_out.weight'\]....](https://github.com/openai/whisper/discussions/2302)** > > I am trying to fine tune the model using the technique shown in this tutorial: https://huggingface.co/blog/fine-tune-whisper My dataset is much larger and my PC cannot handle it all at once so I am... > [@Unable to load checkpoint after finetuning](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-checkpoint-after-finetuning%2F50693%2F)%3A > > Hi, So I’m running into trouble when trying to load the model weights after finetuning a pretrained base model, steps I’m taking: model= BertForSequenceClassification.from\_pretrained("model/repo") model.classifier = torch.nn.Sequential( torch.nn.Dropout(0.4), torch.nn.Linear(model.config.hidden\_size, 256), torch.nn.ReLU(), torch.nn.Dropout(0.4), torch.nn.Linear(256, 2), ) getting a warning here which I guess is fine for the moment: Some weights of the model checkpoint at… > [@Resuming training: There were missing keys in the checkpoint model loaded: \['lm\_head.weight'\]](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fresuming-training-there-were-missing-keys-in-the-checkpoint-model-loaded-lm-head-weight%2F103831%2F)%3A > > Hey all, Trying to learn a bit about training AI models, so I figured I would try to kind of recreate the “Nothing, Forever” AI but with a whatsapp friend group of mine as training set. However, the text set that i’m using is large (100k lines) so training is taking a while and I would like to interrupt it at times. It all seems to work fine, except when i resume a checkpoint I always see this message: There were missing keys in the checkpoint model loaded: ['lm\_head.weight']. Should I be wo…