kohya-ss enhanced text-to-image model trainer
Collection of PRs for kohya-ss text-to-image model trainer
Validation loss
Track a separate validation dataset for loss to compare with. Helps detect overfitting of the training dataset.
Preference optimization
- PR: Add preference optimization (Diffusion-DPO, MaPO)
- Paper: Diffusion Model Alignment Using Direct Preference Optimization
- Paper: Margin-aware Preference Optimization for Aligning Diffusion Models without Reference
Rank stabilized
Stablize the scaling component of alpha and rank for larger LoRA ranks.
LoRA+
Improve training performance by setting a higher LR for the B tensor in LoRA
Soft min SNR
Smooth out Min-SNR (signal-to-noise ratios) approach for minimum SNR.
- PR: Soft min SNR gamma
- Paper: Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers
Drop keys
Remove certain keys from being trained in the LoRA.
Momentum
Log and record momentum changes. In concert with CyclicLR and OneCycleLR PyTorch learning rates which modify the momentum.