Tutorial: https://youtu.be/XO5eNJ1X2rI

What does this workflow?

A background animation is created with AnimateDiff version 3 and Juggernaut. The foreground character animation (Vid2Vid) with AnimateLCM and DreamShaper.

Seamless blending of both animations is done with TwoSamplerforMask nodes.

This method allows you to integrate two different models/samplers in one single video. This example uses two different checkpoints, loras and animateDiff models, but the method can also be used for image compositions where you want to use, for example, a realistic model for the foreground, and an artistic drawing model for the background.

Workflow is tested with SD1.5. SDXL or other SD models could be used, but controlnet models, loras, etc., should be changed to the corresponding version.

How the workflow works

1- Background: adjust a background of your choice to the size of the frames of the original video.

2- Background animation: create an animation using the background picture. Choice in the example is to use AnimateDiff version 3. Latent will be later used for the TwoSamplerMask. LooseControlNet and Tile are recommended. Loras are optional.

3- Create masks: Masks for the foreground character from the starting video are created

4- Foreground animation: create an animation of the foreground using the frames from the video. Choice in the example is to use AnimateLCM. IP Adapter is used. Recommended ControlNets are ControlGif, Depth and OpenPose. Foreground mask is used in controlNets

5- TwoSamplers: provides two different samplers for rendering in step 6. Sampler 1 (background) uses AnimateDiff version 3, while sampler 2 AnimateLCM. The settings, thus, need to be different.

6- Rendering: First option is to directly upscale (latent to 1.5x). Higher resolution, but time consuming. Second options to directly use the TwoSamplersforMask node, which uses. With the foreground mask we can have seamless integration of the background and foreground.

7- Face Detailer and Frame interpolation: Face detailer and frame interpolation are added to correct face distortion and improve smoothness of the video.

Additional information and tips

•Check out the assets to use in the example

•Check notes in the workflow for additional instructions