wav

/

TemporalNet2

like 11

I'm new to this and aware that TemporalNet works best with other controlnet conditionings like HED. What tensor/image I should be using for the HED edge detection?

Here's how I intended to apply HED in the code:

1. Add multiple conditioning controlnet in pipe:
   controlnet=[ ControlNetModel.from_pretrained("wav/TemporalNet2", torch_dtype=torch.float16), ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-hed", torch_dtype=torch.float16) ]

2. Initiliase HED detector

3. In stylize_video(), convert flow_img tensor to PIL.Image using torchvision.transforms.ToPILImage

4. Get HED image from above PIL.Image

5. In pipe, use arguments:
   control_image=[control_img, hed_image],
controlnet_conditioning_scale=[1.0, 1.0]