Fundamentals for LLMs and Image Diffusion

 

I am enjoying Midjourney very much.  I presume I would be as impressed with any LLM (large language model) driven image diffusion system but this is the only one I have played with.  I feel completely awed and blindsided by the range of capabilities I have seen here.  Where did this come from?  How does it work?  To save you the effort of finding out, I am endeavoring to collect a list of topics it will be necessary to have at least a passing familiarity with in order to understand this modern miracle.  It probably wont be necessary for you to be a genius at any of these topics, perhaps, but it may be necessary for you to know something about all of these if you want to know what is going on beneath the hood and maybe predict where this kind of technology is going.

 

To give you an idea about why I find this so compelling and disturbing, here is the result of "/imagine vintage photograph of king kong climbing the Chrysler building".

 

 

 

A partial list of technologies involved include 

 

Latent Variable Models

Variational Autoencoder

Generative Adversarial Networks (GANs)

Semantic Image Synthesis

 

and more to come.

GAN Generated Rainforest Flower Images

draft

Generated by a variation of a GAN from rainforest flowers as suitably modified.

Assume that these images are used in a large rendered scene where the textures are also used to generate suitable geometry.

[3/19/2019 Notes: Back in the day, this would have been enough to get me into many magazines and get lots of publicity!  Not anymore!  Now I must walk the path of ascetic renunciation of all worldly fame!  Sad!]

Channelling Concept Art from A Horror Movie

draft

More entertaining accidents from my tests of GANs and machine learning.  Perhaps concept art from a horror movie I did not know I was working on?

Alien Merit Badges

draft

Given a diet of 200 real and fake merit badges from the internet, suitably rotated and skewed, my little convnet GAN generates some acceptable alien merit badges.  

Alien Protest March Slogans Take 2

Alien Protest March posters, take 2. Generated character by character by LSTM machine learning network trained on 6,000 slogans from the Woman's March in Boston, three years ago, selected by hand. Imagine that Stoth is a bad alien, etc. Use your imagination! Number on left is sequence number from a much larger list.

46 Ruin!!!
63 ist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resist! Resis
67 I AM I March For All
96 I WAnk FOn-Y MATION
104 respect up you didn't balchut
109 Trump is Stoth. Hear Our Voice!
112 No! Unfather
155 We Will Not Go barak! Make America Kind Again!!!
156 Hate is everything
168 I am majorrach
336 Women Enothotbobia
376 I March For America Kind Again
404 Love is not normal
445 I am March For My Lame
463 We Are Bouth Not Be Selfory Humans!
471 You're Vigal Rights! No Seen Star Mumun. Trump is Poor Defend Against The Sidety

Text Generation from Women's March

draft

Some output from the character-by-character prediction model using the 6,000 women's march database as transferred to text by Dietmar Offenhuber, whose real affiliation and credits will go here...

Here are some results at an intermediate error rate.  Not too bad, but remember I am cherry picking.  A few highlights are highlighted!


beginning epoch 27
history = {'loss': [0.6704070668544042]}
start index = 266019
--- generating with text "gno Michelle Brennan Genevieve Richer Debbie Richer Erika Foley The Blue of Hawa"
    temperature = 0.5
       "rd Care strong up women's rights are human rights/My mind to trumps us! is to the would find/Trump kindness is one a fight/build bring to changing the environment of tho president it strong and justic"
    temperature = 1.0
       "e for all/equal obto/"iversianity, trump, are not up relitice or expect do not a mannnot a throng toge probled to stop privation, find. Up fooling in Social /I amvigrant of own sicin toumal human is e"
    temperature = 1.5
       "tlanc: + reprothect, LGT!, murcore pro-yor tilence/love trumps half a whoice! ;empridilet, oradary all i great!/Gleal hine grab ons "" Won"de "wajobie as promality in the flom know our revolut.!/Dreat"

beginning epoch 28
history = {'loss': [0.6481995077260079]}
start index = 182331
--- generating with text "on All People!/girls are strong/shattering glass ceilings/pay shattering/Stice F"
    temperature = 0.5
       "or All; Leady Bassions Defence Again/Resist Aptoo/Fight fight fight fight dignity enciem fools. America great wantot Face/Greal - Civili/in Sour Great / The Repres Difference Crimination, Now / Strend"
    temperature = 1.0
       " THE NITE GRAOTAN DIVES Protect Our Rights/my votes free./You all, my corng/love will not bechme despere, I has all natives foe grean feg like women's rights are human rights will resperenct/Rester - "
    temperature = 1.5
       "like Not Hate Strong People Defendit?/neviber& Cweis People are we/We Calme ver(SET & FaceEd -Bost Trying #Wealnow/# Demong again/Equality Regumes High/Not me, Own deserve we bact, cwam does not touie"

beginning epoch 29
history = {'loss': [0.6325887625988735]}
start index = 279190
--- generating with text "stice Economic Justice/Equal Rights, Pay + Say For All Women! Less Aggression Cl"
    temperature = 0.5
       "imate Change is Real Immigrants Matter!/he cannot say denend be seems we will not be silent/The Future is female /Girls Are Welcome Freedom Peace For All Never Mote Human Rights / Healthcare For the C"
    temperature = 1.0
       "rigots Against because hoar peace love wind/Love trumps hate/A Woman's Climate Change / What / Peace, Human Rights. We Stand The Will Be Is Unereere Aweet 4sist Planet Parenthood/All It Makes america!"
    temperature = 1.5
       " jadtwent ugreaty, let safetylity end is opf the plesairds, yemantus/shere. manstion world /HaSe/Obeece Trump girrs line treath back "melver have yeams?'sm Sunde Wassle the Ciad zotgefen !!/n/abagzian"

Generating Text Character by Character

draft

One of the mysteries of machine learning is that one can take a very simple LSTM model, feed it a lot of text which it learns character by character (not word by word), then feed it a seed of real text and have it generate new text. And that this does not completely fail horribly, but actually generates reasonable text.

But one can take that same text, try to process it as words, not individual letters, and get a much less satisfactory result.

Maybe I am doing something wrong? Oh, no doubt.
 

Automatically Generating an Alien Alphabet

draft

As part of our "learning machine learning" project, we experimented with GANs to generate images of a certain "type", learning by example as it were.  

In the following example, I used two versions of an uncomplicated font typeface, Luxisans, all capital letters.  One version was monochrome and the other had a noise texture applied as follows:

These images were then rotated, sheared, and modulated in intensity to generate more data.

Here are some of the images it generated.

Machine Learning and Donald J. Trump

draft

Machine learning is not perfect, no one claimed that it was or would be. I fed my little "convnet" GAN pictures of Donald J Trump to see if it could learn to make pictures of Trump, but all that came out was garbage.

GANs That Work and GANs That Fail

draft

So after the previous "circle test". I try two more tests using the exact same GAN networks, batch size, etc.  One is a "square test", similar to the "circle test" and one is a "grid test" of 100 variations on a grid.  See examples of both below.

And here are some results from the "square test"

I think these are looking pretty good.

But now when we get to the grid test, well, not so good.

This makes me wonder if GANs can really handle anything other than very clear graphic shapes.  Or maybe a completely different type of network is required.

Incremental Notes on GANs

draft

So there are a variety of things I have learned about GANs in the last few weeks.  

1. I am limited by my GPU memory.  The more memory you have, the more you can fit into the GPU, the larger the batch size.  A batch size of 6 may have very different results from a batch size of 2, or a batch size of 50.

2. It is only the GPU (in my case a 1080 with 8 GB memory) that makes this at all practical or possible.  I have 8/16 pretty good processors here and they can run for 10 minutes and not do a single batch, whereas the GPU can do a batch in 60 seconds or less.

3. 256 resolution is much more practical than 512 resolution.

4. It is not at all clear whether GANs can do any of the simplistic projects I wanted to do with galaxies, nebulae or march posters.

5. With the simplistic test patterns, at least 150,000 samples are required.

Test Patterns for Machine Learning

draft

I am slowly reducing my test patterns for machine learning to the simplest form that will tell me something.  

When this is a little further along, I may package them all up and put it on kaggle as a lesson for the others.

I should mention that these images have been through the Tensorflow/Keras data augmentation process, and are hence smashed in a variety of ways.  Here is an example of an original image as created via Nuke.  Still not perfect, but better.  For what I am doing, it really doesnt matter.

Machine Learning and Nebulae

draft
Damn this machine learning can be fun! The images are definitely getting better and I think it would be petty to complain that the "generated" images dont look much like the input "real/training" images. That would be the concern of small minded people and everyone here I am sure operates on a higher level. Oh yes, also, the "metric" of good vs bad (error, incorrectness, what have you) also seems to have no relationship whatsoever to the images being "judged". 

Again, we must rise above such nonsense!

Nebulae, who needs nebulae?