# Splitting dataset via length
**URL:**
**Author:**  [@sachin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsachin)%5C
**Post date:** [September 1, 2022, 9:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-via-length%2F22395%2F1 "2022-09-01T09:38:32Z")
So there is a few parts to this question.
1. How do I find the length of entire dataset?
2. How do I shuffle the dataset and assign the first N to valid and rest to train.
Here is a code snippet with what I am trying to do:
```python
data = datasets.load_dataset("liweili/c4_200m", cache_dir="/kaggle/working/", streaming=True, split="train")\
.shuffle(seed=42, buffer_size=10_000)
c4_train = data.skip(10000)
# c4_valid = how do I assign the first 10000 that I skipped?
def group_batch(batch):
return {k: [v] for k, v in batch.items()}
train_dl = c4_train.map(group_batch, batched=True, batch_size=32)
```
`len` isn’t working on any of these objects ☹
---
**Author:**  [@rwheel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frwheel)%5C
**Post date:** [September 1, 2022, 9:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-via-length%2F22395%2F2 "2022-09-01T09:59:04Z")
Hi @sachin ,
Using your code as example:
> [@sachin](#):
>
> How do I find the length of entire dataset?
`data.shape`
(Doc: [Main classes](https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/main_classes#datasets.Dataset.shape))
> [@sachin](#):
>
> How do I shuffle the dataset and assign the first N to valid and rest to train.
`data.train_test_split(test_size=0.1)`
For this last question, take a look to this documentation: [Process](https://huggingface.co/docs/datasets/v2.4.0/en/process#split)
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [September 1, 2022, 1:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-via-length%2F22395%2F3 "2022-09-01T13:36:21Z")
Hi ! In streaming mode you don’t get a Dataset object but an IterableDataset. We can’t know in advance the length of an iterable dataset (e.g. to know the number of examples in a CSV file you have to download it completely).
You can set the first examples to be your validation split this way:
```python
c4_train = data.skip(10000)
c4_valid = data.take(10000)
```
---
**Author:**  [@rwheel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frwheel)%5C
**Post date:** [September 1, 2022, 4:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-via-length%2F22395%2F4 "2022-09-01T16:06:14Z")
You’re right! I didn’t realize that he was loading the dataset in streaming mode. Sorry!