Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5

Artist: Bahnbahn
#4

2.2. Test LION optimizer as well.
Comparsion A-2: Standartized LoCON
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
LION got quite eager and started to overoptimize on 3rd epoch to the left.
Surprisingly, 3rd epoch on the right got pretty much same result.

Comparsion B-2: Standartized LoHA
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
LOHA+LION was, in fast, SO successful, it started to completely copy initial pictures from the 3rd epoch! By all means, I am forced to pich 2nd epoch!
And pretty much same goes for SKS as well! 2nd epoch is all we can allow ourselves

Comparsion C-2: Expanded LoHA
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
LoRA got so powerful, that, in fact, it started to change every single image into its dataset! I cannot select anything but 1st epoch from both sides

Comparsion D-2: Standartized LoCON / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
From the left, 4th epoch seems best one
Same could be said about right side - 4th epoch

Comparsion E-2: Standartized LoHA / Slower training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
From the left side, Epoch 3
From the right, Epoch 4

Comparsion F-2: Expanded LoHA / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
2nd from the left
2nd from the right

Comparsion G-2: Standartized LoCON / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-6 / 5e-7
* Optimizer = Lion8bit
Comparsion Image
4th epoch for both sides

Comparsion H-2: Standartized LoHA / Slowest training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
4th from both sides

Comparsion I-2: Expanded LoHA / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
Exactly 1st epoch on both

And, of course, compare SKS-less with high dropout rate AND SKS-full with small dropout rate

Comparsion J-2: Standartized LoCON
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
Both sides stopped at 3rd epoch

Comparsion K-2: Standartized LoHA
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
Exactly 1st epoch on both

Comparsion L-2: Expanded LoHA
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-4 / 5e-5
* Optimizer = Lion8bit
Comparsion Image
Exactly 1st epoch on both

Comparsion M-2: Standartized LoCON / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
Both at 5th epoch

Comparsion N-2: Standartized LoHA / Slower training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
4th epoch from both sides

Comparsion O-2: Expanded LoHA / Slower training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
1st epoch from both sides

Comparsion P-2: Standartized LoCON / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-6 / 5e-7
* Optimizer = Lion8bit
Comparsion Image
5th epoch from both sides

Comparsion Q-2: Standartized LoHA / Slowest training
* Network Dim/Alpha = 8/4
* Conv Dim/Alpha = 4/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
4th epoch from both sides

Comparsion R-2: Expanded LoHA / Slowest training
* Network Dim/Alpha = 16/8
* Conv Dim/Alpha = 8/1
* Learning Rate = 1e-5 / 5e-6
* Optimizer = Lion8bit
Comparsion Image
1st epoch from both sides
Reply


Messages In This Thread
Artist: Bahnbahn - by projectsoulstealer - 09-23-2023, 12:41 PM
RE: Artist: Bahnbahn - by projectsoulstealer - 09-30-2023, 11:20 AM
RE: Artist: Bahnbahn - by projectsoulstealer - 10-18-2023, 09:49 AM
RE: Artist: Bahnbahn - by projectsoulstealer - 10-18-2023, 05:16 PM
RE: Artist: Bahnbahn - by projectsoulstealer - 10-19-2023, 08:56 PM

Forum Jump:


Users browsing this thread: 1 Guest(s)

MyBB SQL Error

MyBB has experienced an internal SQL error and cannot continue.

If you're a visitor of this website, please wait a few minutes and try again. If this problem persists, please contact the site owner.

If you are the site owner, please check the MyBB Documentation for help resolving common issues, or get technical help on the MyBB Community Forums.