In Diffusion Models: Why does not use 'Positional Encoding' in self-attention layers?

[lionking6792]

Thx for nice practicing about DM.
Actually, I'm really curious about why does not use 'Positional Encoding' (which was used in ViT or VanillaTransformer.. etc..) in self-attention layers?
Is that any reason and can we ensure self-attention in DDPM U-Net can maintain its position(pixel-wise) information?

[explainingai-code]

Hi,

This DDPM code was created replicating the blocks from the official version for stable diffusion with the goal to create a minimalistic diffusion model purely for understanding purposes(and then progress to Stable Diffusion from here).

As far as I saw in the official repo, the authors don't add any positional encoding information prior to attention in their transformer block.
That was the only reason for me also not adding in here and without adding it, I was getting decent results so then didn't end up adding it.

I don't think in the stable diffusion repo, the authors anywhere mention why that is the case though. The same is asked here as well. So to answer why its not added, I honestly don't know. Maybe after adding positional information the result might indeed end up being significantly better. Or maybe the presence of padding in the resnet convolutions already give the network enough indication of positional information in its feature map representations, enough that it manages to do fine without any explicitly provided positional information.

If you want to experiment with adding the positional information to the attention block , you can just replicate what is done in diffusers library here, where they simply add sinusoidal embeddings prior to every attention block .
Do let me know what results you end up with, in case you decide to try it out or if you end up finding more information on this.