# Simple NLP Example not working **URL:** **Author:** ![ocm](https://avatars.discourse-cdn.com/v4/letter/o/aca169/32.png) [@ocm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Focm)%5C **Post date:** [September 12, 2021, 2:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F1 "2021-09-12T14:41:58Z") Hi, Iā€™m looking for an example to learn how to use TPUs on Colab running PyTorch. Iā€™m glad to find the [Simple NLP Example](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/accelerate/simple_nlp_example.ipynb) which is unfortunately not working. Running w/o modifications leads to following error message running the last cell: > from accelerate import notebook\_launcher > > notebook\_launcher(training\_function) ```auto --------------------------------------------------------------------------- ImportError Traceback (most recent call last) in () 1 from accelerate import notebook_launcher 2 ----> 3 notebook_launcher(training_function) 1 frames /usr/local/lib/python3.7/dist-packages/torch_xla/ __init__.py in () 99 from ._patched_functions import _apply_patches 100 from .version import __version__ --> 101 import _XLAC 102 103 ImportError: /usr/local/lib/python3.7/dist-packages/_XLAC.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK3c1010TensorImpl20is_contiguous_customENS_12MemoryFormatE --------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt. To view examples of installing some common dependencies, click the "Open Examples" button below. --------------------------------------------------------------------------- ``` I found a workaround description [here](https://medium.com/the-owl/installing-pytorch-xla-in-google-colab-without-35d37d1d03c6) which says: `... downgrading PyTorch to torch-1.8.2+cpu`, but that leads to another error message ```auto ProcessExitedException: process 0 terminated with signal SIGSEGV ``` What is necessary to run that example? Do you know any other example that meets my requirements (Colab, TPUs, PyTorch) and runs? Thanks for any comment ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C **Post date:** [September 13, 2021, 7:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F2 "2021-09-13T07:33:47Z")
Iā€™m guessing this is due to a version mismatch between PyTorch XLA and PyTorch (PyTorch XLA is installed with a version for PyTorch 1.9 and Colab now uses PyTorch 1,10). Iā€™ve asked for an updated link to install the proper version of PyTorch XLA, but in the meantime, you cna solve the issue by downgrading PyTorch to 1.9.1 in the Colab you are running. ---
**Author:** ![ocm](https://avatars.discourse-cdn.com/v4/letter/o/aca169/32.png) [@ocm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Focm)%5C **Post date:** [September 14, 2021, 8:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F3 "2021-09-14T20:28:45Z")
On request of the pytorch version I get ā€œ1.9.0+cu102ā€ What exactly am I supposed to do? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C **Post date:** [September 14, 2021, 9:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F4 "2021-09-14T21:34:53Z")
Ah youā€™re right, I was confused. Itā€™s working now with no change. Perhaps there was some maintenance isse? ---
**Author:** ![ocm](https://avatars.discourse-cdn.com/v4/letter/o/aca169/32.png) [@ocm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Focm)%5C **Post date:** [September 15, 2021, 7:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F5 "2021-09-15T19:18:38Z")
Great, thanks ---
**Author:** ![MoritzLaurer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moritzlaurer/32/18932_2.png) [@MoritzLaurer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMoritzLaurer)%5C **Post date:** [November 19, 2021, 8:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F6 "2021-11-19T08:40:02Z")
Weirdly enough, the notebook example is working for me with the normal RAM TPU from colab (if I change the pytorch version to `!pip3 install torch==1.9` from the new colab default torch==1.10). But I get the same SIGSEGV error, when changing to the high-RAM TPU from colab. the exact same code works with the low-ram TPU, does does not work with the high-ram TPU. probably an issue with google colabā€™s setup in the background ā€¦ Colab with traceback here: [simple\_nlp\_example.ipynb - Google Drive](https://drive.google.com/file/d/1Gp0-7XhJb_h7O4WlmOE-g5K5VB0bGOs6/view?usp=sharing) ---
**Author:** ![drscotthawley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/drscotthawley/32/62161_2.png) [@drscotthawley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrscotthawley)%5C **Post date:** [June 2, 2022, 8:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F7 "2022-06-02T20:53:22Z")
I too am seeing the SIGSEV error when I run the Simple NLP example on an A100 pod: ```auto --------------------------------------------------------------------------- ProcessExitedException Traceback (most recent call last) Input In [25], in () 1 from accelerate import notebook_launcher ----> 3 notebook_launcher(training_function, num_processes=4) File ~/envs/wav2vec/lib/python3.9/site-packages/accelerate/launchers.py:129, in notebook_launcher(function, args, num_processes, use_fp16, mixed_precision, use_port) 126 launcher = PrepareForLaunch(function, distributed_type="MULTI_GPU") 128 print(f"Launching training on {num_processes} GPUs.") --> 129 start_processes(launcher, args=args, nprocs=num_processes, start_method="fork") 131 else: 132 # No need for a distributed launch otherwise as it's either CPU or one GPU. 133 if torch.cuda.is_available(): File ~/envs/wav2vec/lib/python3.9/site-packages/torch/multiprocessing/spawn.py:198, in start_processes(fn, args, nprocs, join, daemon, start_method) 195 return context 197 # Loop on join until it returns True or raises an exception. --> 198 while not context.join(): 199 pass File ~/envs/wav2vec/lib/python3.9/site-packages/torch/multiprocessing/spawn.py:140, in ProcessContext.join(self, timeout) 138 if exitcode < 0: 139 name = signal.Signals(-exitcode).name --> 140 raise ProcessExitedException( 141 "process %d terminated with signal %s" % 142 (error_index, name), 143 error_index=error_index, 144 error_pid=failed_process.pid, 145 exit_code=exitcode, 146 signal_name=name 147 ) 148 else: 149 raise ProcessExitedException( 150 "process %d terminated with exit code %d" % 151 (error_index, exitcode), (...) 154 exit_code=exitcode 155 ) ProcessExitedException: process 0 terminated with signal SIGSEGV ``` That error was also preceded by a bunch of warnings BTW: ```auto Launching training on 4 GPUs. huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) ``` ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C **Post date:** [June 2, 2022, 9:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F8 "2022-06-02T21:21:57Z")
> [@drscotthawley](#): > > That error was also preceded by a bunch of warnings BTW: The warnings are normal (though something we should try and fix in the script if possible), however Iā€™ll look at the nb launcher error. Given we just made it work for the fastai integration (with now changes), I _can_ say that _should_ work. To make sure I can reproduce this, itā€™s an a100 system? Could you also run `accelerate env` and print that here as well please? šŸ™‚ ---
**Author:** ![drscotthawley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/drscotthawley/32/62161_2.png) [@drscotthawley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrscotthawley)%5C **Post date:** [June 2, 2022, 9:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F9 "2022-06-02T21:43:31Z")
Hey Zach!! šŸ™‚ Yes, Itā€™s an A100. ```bash $ accelerate env Copy-and-paste the text below in your GitHub issue - `Accelerate` version: 0.9.0 - Platform: Linux-5.4.0-1060-aws-x86_64-with-glibc2.27 - Python version: 3.9.9 - Numpy version: 1.22.4 - PyTorch version (GPU?): 1.11.0+cu115 (True) - `Accelerate` default config: Not found ``` Do I need to setup a default config? ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C **Post date:** [June 2, 2022, 11:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F10 "2022-06-02T23:07:24Z")
> [@drscotthawley](#): > > Do I need to setup a default config I belive so, run `accelerate config` and answer the questions before trying again šŸ˜„ ---
**Author:** ![drscotthawley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/drscotthawley/32/62161_2.png) [@drscotthawley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrscotthawley)%5C **Post date:** [June 2, 2022, 11:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F11 "2022-06-02T23:20:30Z")
> [@muellerzr](#): > > accelerate config Ok, I ran that and answered the questions, and restarted and re-ran but still got the SIGSEV error. Hereā€™s the new `accelerate env` output: ```auto - `Accelerate` version: 0.9.0 - Platform: Linux-5.4.0-1060-aws-x86_64-with-glibc2.27 - Python version: 3.9.9 - Numpy version: 1.22.4 - PyTorch version (GPU?): 1.11.0+cu115 (True) - `Accelerate` default config: - compute_environment: LOCAL_MACHINE - distributed_type: MULTI_GPU - mixed_precision: fp16 - use_cpu: False - num_processes: 4 - machine_rank: 0 - num_machines: 1 - main_process_ip: None - main_process_port: None - main_training_function: main - deepspeed_config: {} - fsdp_config: {} ``` **Update:** tried reconfiguring with FSDP on (and then chose the defaults presented thereafer), restarting, and re-running. Same SIGSEV error. ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C **Post date:** [June 9, 2022, 6:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F12 "2022-06-09T18:41:21Z")
> [@drscotthawley](#): > > Ok, I ran that and answered the questions, and restarted and re-ran but still got the SIGSEV error. Hereā€™s the new `accelerate env` output: Thanks for your patience with this! For the time being Iā€™d recommend using `!accelerate launch myscript` instead of the notebook launcher. Weā€™re working on a fix for that but I can confirm that using it in script form works fine **Or** donā€™t use the High VRAM instance ---
**Author:** ![smangrul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smangrul/32/3791_2.png) [@smangrul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmangrul)%5C **Post date:** [June 10, 2022, 11:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F13 "2022-06-10T11:04:57Z")
Hello, Please refer [`notebook_launcher` throws `SIGSEGV` error when using pretrained transformer models for NLP tasks Ā· Issue #440 Ā· huggingface/accelerate (github.com)](https://github.com/huggingface/accelerate/issues/440) for more context as per my deep dive. ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C **Post date:** [June 10, 2022, 2:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F14 "2022-06-10T14:09:17Z")
> [@drscotthawley](#): > > **Update:** tried reconfiguring with FSDP on (and then chose the defaults presented thereafer), restarting, and re-running. Same SIGSEV error. Good news is move the `from_pretrained` from outside of the train loop and use the already downloaded one available and it will run fine. The bug stems from that function specifically. For keeping an eye on progress: - [Transformers issue to track](https://github.com/huggingface/transformers/issues/17659) - [Accelerate TPU issue](https://github.com/huggingface/accelerate/issues/434) - [Accelerate high-ram GPU issue](https://github.com/huggingface/accelerate/issues/440) ---
**Author:** ![drscotthawley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/drscotthawley/32/62161_2.png) [@drscotthawley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdrscotthawley)%5C **Post date:** [June 22, 2022, 7:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F15 "2022-06-22T07:12:28Z")
Thanks Zach! Iā€™d put this issue on the backburner for a while but am glad to see this progress. Seems that some colleagues of mine _are_ able to use `accelerate` on High VRAM instances, but they arenā€™t using the notebook environment. (This is important because I _do_ need to be able to run on High VRAM, so just ā€œdonā€™t do itā€ wasnā€™t going to help me. LOL). **EDIT:** BTW, Iā€™d been seeing this error on an AWS A100 too, not just Colab. ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C **Post date:** [June 23, 2022, 3:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F16 "2022-06-23T03:02:38Z")
> [@drscotthawley](#): > > just Colab. Should be fixed now, i just need to update the nb but the Accelerate specific bugs should be solved outside of that specific cause. Declare the model outside the training function, and pass it in as an argument instead. A PR with this fix will go live tommorow as well. The model should only be declared once on a TPU and its passed back and forth, rather than duplicated in a notebook/forked process ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C **Post date:** [June 23, 2022, 1:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsimple-nlp-example-not-working%2F9913%2F17 "2022-06-23T13:45:11Z")
The notebook has been updated to show how to properly do this and it should be fixed. Please let me know if you still see this issue!