# Probsparse\_attention in Informer
**URL:**
**Author:**  [@Gozdi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGozdi)%5C
**Post date:** [March 22, 2023, 7:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprobsparse-attention-in-informer%2F34428%2F1 "2023-03-22T19:41:48Z")
Hi
I was reading this blog post [Multivariate Probabilistic Time Series Forecasting with Informer](https://huggingface.co/blog/informer) and it has a section where probsparse\_attention is implemented and I think it has a bug.
```auto
# get input sizes with logs
L_K = key_states.size(1)
L_Q = query_states.size(1)
log_L_K = np.ceil(np.log1p(L_K)).astype("int").item()
log_L_Q = np.ceil(np.log1p(L_Q)).astype("int").item()
# calculate a subset of samples to slice from K and create Q_K_sample
U_part = min(sampling_factor * L_Q * log_L_K, L_K)
# create Q_K_sample (the q_i * k_j^T term in the sparsity measurement)
index_sample = torch.randint(0, L_K, (U_part,))
K_sample = key_states[:, index_sample, :]
Q_K_sample = torch.bmm(query_states, K_sample.transpose(1, 2))
```
shouldn’t U\_part be
```auto
U_part = min(sampling_factor * log_L_K, L_K)
```
without the L\_Q? L\_Q needs to be significantly smaller than L\_K if we want U\_part to be smaller than L\_K and we want that, because when it’s equal to L\_K, Q\_K\_sample is just Q @ K and avoiding that is the whole point.
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [March 22, 2023, 8:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprobsparse-attention-in-informer%2F34428%2F2 "2023-03-22T20:08:27Z")
Hi,
I’ve pinged @kashif and Eli to take a look!
---
**Author:**  [@kashif](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkashif)%5C
**Post date:** [March 22, 2023, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprobsparse-attention-in-informer%2F34428%2F3 "2023-03-22T20:27:11Z")
Indeed it’s a bug in the pseudo-code of the blog… If you want to you can open a PR to fix the blog-post or I will do it by tomorrow.
Thanks again!
---
**Author:**  [@elisim](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Felisim)%5C
**Post date:** [March 31, 2023, 1:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprobsparse-attention-in-informer%2F34428%2F4 "2023-03-31T13:19:30Z")
Hi Gozdi,
Thank you for reading our blogpost so thoroughly, and for pointing out this issue!
Actually, at the beginging our implementation indeed was like you said, i.e.:
`U_part = min(sampling_factor * log_L_K, L_K)`
But, after looking more thoroughly in the original implemetion, `U_part` is set to `c*m*ln(n)` for stability issues, i.e.:
`U_part = min(sampling_factor * L_Q * log_L_K, L_K)`
Citing the authors:
> we set U\_part to `c*m*ln(n)` to make the value of U\_part not too small, so as to ensure the stability and better effect of the model.
Reference: [About the U\_part · Issue #161 · zhouhaoyi/Informer2020 · GitHub](https://github.com/zhouhaoyi/Informer2020/issues/161)
Finally, you can see that originally our implementation was like you described, and than fixed 🙂:
Thank you for noticing this! I added this discusion to the blogpost for the next readers.
Eli