Character Portrait Generation Workflow

This workflow is designed to create highly detailed and aesthetically pleasing character portraits, using a three-step process to progressively refine the generated image. It relies on open-source diffusion models, which transform descriptive text prompts into realistic character images. The workflow is structured as follows:

Workflow Overview

The generation process is divided into three key steps:

• Base Generation: This step establishes the initial character image. Using the main diffusion model, it generates a “base” image, which provides a visual foundation for the character, capturing primary features and general details.

• Body Refining: After the base image is generated, this step enhances the body structure and details. A specialized model is applied to detect and refine body components, creating a cohesive and realistic character form.

• Face Refining: The final step focuses on facial detail enhancement. This process applies targeted refinement to facial elements, creating polished, lifelike features that complete the character's appearance.

Core Models and Components

Several models work together within this workflow to produce the final image. Key components include:

• Model: The core model that iteratively transforms an initial noise image into the final character portrait based on your text prompts.

• CLIP: A model used to encode your prompt, guiding the diffusion process toward the description provided.

• VAE (Variational Autoencoder): Responsible for converting the generated image from latent space (an abstract space where sampling occurs) to pixel space, making the image viewable.

• LoRAs (Low-Rank Adapters): Optional “adapter” models that can be applied to steer the diffusion process toward a specific style or concept. They allow for flexible customization by adjusting style or thematic emphasis without altering the main model.

Currently, this workflow uses the following extensions:

• ComfyUI-ImpactPack : https://github.com/ltdrdata/ComfyUI-Impact-Pack

Usage Notes

This workflow is designed for flexibility and experimentation. Here are some tips for effective use:

• Iterating on Base Generation: To find a good starting image, bypass the last two sections and use lower step counts (e.g., 15-20) in the first. Experiment with prompts until the desired look is achieved. Once satisfied, fix the seed of the first section and proceed with the refining steps by ractivating the refining steps.

• Refining Tips: If body or face refining steps do not detect features or produce unintended artifacts, adjust the “threshold” or “steps” parameters in those sections for more control.

You'll find below some additional tips & notes about this workflow.

Recommended models

Any model under the SDXL or SD1 architecture will do for this workflow. However, I've obtained the best results with the following models:

• Model: Prefect Pony XL (https://civitai.green/models/439889/prefect-pony-xl). Note that this model is based on the PonyXL model and has been designed to produce NSFW and explicit content. Therefore, be sure to place "rating_explicit" in the negative prompt and "rating_safe" in the positive prompt in order to avoid generating explicit content.

• LoRA: MoreArt (https://civitai.green/models/124347) and Epic Fantasy Style (https://civitai.green/models/470073).

• VAE: Pony Enhanced VAE (https://civitai.green/models/660613).

For those who lack the computing power to run SDXL models, I've found the following models to give good results:

• Model: ArthemyComics (https://civitai.green/models/54073/arthemy-comics).

• LoRA: EpicCanvas-Genesis (https://civitai.green/models/197362/epiccanvas-genesis).

• VAE: Color101 (https://civitai.green/models/70248/color101-vae).

Refinement Steps

While hands are often challenging to represent accurately in text-to-image diffusion models, this workflow does not include a dedicated “hand refining” step. Based on practical experience, such a step often introduces unwanted artifacts or “hallucinations” instead of improving the hand rendering. Instead, the body refining step is typically effective in refining hands, as they are part of the broader body structure.

Comparing Outputs

To facilitate comparison and track refinement progress, each image outputted by the three workflow steps (Base Generation, Body Refining, and Face Refining) is displayed. This allows users to observe how each step contributes to the final character portrait’s level of detail and realism.