[projectsoulstealer]

2.2. Test LION optimizer as well.
Comparsion A-2: Standartized LoCON
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
LION got quite eager and started to overoptimize on 3rd epoch to the left.
Surprisingly, 3rd epoch on the right got pretty much same result.

Comparsion B-2: Standartized LoHA
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
LOHA+LION was, in fast, SO successful, it started to completely copy initial pictures from the 3rd epoch! By all means, I am forced to pich 2nd epoch!
And pretty much same goes for SKS as well! 2nd epoch is all we can allow ourselves

Comparsion C-2: Expanded LoHA
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
LoRA got so powerful, that, in fact, it started to change every single image into its dataset! I cannot select anything but 1st epoch from both sides

Comparsion D-2: Standartized LoCON / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
From the left, 4th epoch seems best one
Same could be said about right side - 4th epoch

Comparsion E-2: Standartized LoHA / Slower training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
From the left side, Epoch 3
From the right, Epoch 4

Comparsion F-2: Expanded LoHA / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
2nd from the left
2nd from the right

Comparsion G-2: Standartized LoCON / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-6 / 5e-7
* Optimizer = Lion8bit
Comparsion Image
4th epoch for both sides

Comparsion H-2: Standartized LoHA / Slowest training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
4th from both sides

Comparsion I-2: Expanded LoHA / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
Exactly 1st epoch on both

And, of course, compare SKS-less with high dropout rate AND SKS-full with small dropout rate

Comparsion J-2: Standartized LoCON
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
Both sides stopped at 3rd epoch

Comparsion K-2: Standartized LoHA
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
Exactly 1st epoch on both

Comparsion L-2: Expanded LoHA
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
Exactly 1st epoch on both

Comparsion M-2: Standartized LoCON / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
Both at 5th epoch

Comparsion N-2: Standartized LoHA / Slower training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
4th epoch from both sides

Comparsion O-2: Expanded LoHA / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
1st epoch from both sides

Comparsion P-2: Standartized LoCON / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-6 / 5e-7
* Optimizer = Lion8bit
Comparsion Image
5th epoch from both sides

Comparsion Q-2: Standartized LoHA / Slowest training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
4th epoch from both sides

Comparsion R-2: Expanded LoHA / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
1st epoch from both sides