Processor only accepts 3 channel images

[swtb3]

Hello,

I have been trying to finetune this model using my own data which is single channel images and some text describing features of the images.

The processer only seems to handle 3 channel images. To get around this I have stacked the images onto themselves to fill 3 channels. However, this seems like it will have a significant impact on the amount of data I will train the model on.

Is there currently support for single channel in a pretrain BridgeTower checkpoint? If not are there plans to include it? Is stacking images the best approach when working with single channel images for this model?

[LooperXX]

Hello,
since BridgeTower use CLIP as our vision encoder, we are also subject to certain limitations of CLIP, such as only being able to process 3-channel images as you mentioned. You can try to solve this problem by searching for "CLIP processing on single-channel images".

[wangyi5156]

你好，由于 BridgeTower 使用 CLIP 作为我们的视觉编码器，我们也会受到 CLIP 的一些限制，正如你提到的，只能处理 3 通道图像。你可以尝试通过搜索“CLIP 在单通道图像上的处理”来解决这个问题。

Hello,If CLIP is not used as the encoder, will there be no problem of the 3-channel limit?