Robots

Good Morning Friends and Happy Juneteenth! Yesterday was my Birthday in addition to being Father’s Day, and I am super thankful for all the assorted well wishes. This morning is going to be a bit of a departure from my regularly scheduled ARPG nonsense. Over the weekend I spent some time messing around with Stable Diffusion running locally, and I thought I would talk a bit about it. Let’s get some stuff out of the way first. I do not condone Art Generation models as a method of replacing the work of actual artists. You will see a bunch of images adorning this blog, and they are all for the most part the paid creation of my good friend Ammo. In fact, as we speak she is working on yet another one of my hair-brained ideas and I have no clue what the final bill will even be, but will pay it happily as I always do. For me, the “AI Art” landscape is more a toy than a tool and in the past, I have enjoyed feeding it nonsense and seeing what it comes up with out of that chaos. For example… I have no clue who the fuck this dude is but any time I feed it a prompt with the name “Gideon” in it, I end up with this visage.

I’ve known for a while that you could run various generation tools locally off your graphics cards, but I always assumed it would be a tedious process. I started down this present rabbit hole when I found out that there was literally a one-click installer that set everything that you needed up for you. There is a distribution of Stable Diffusion called “Easy Diffusion” that offers a quick install for Windows, Mac, and Linux, and within about 15 minutes I was up and running and generating nonsense. I am sure there is probably something inferior about the path that I took to get to this destination, but I honestly don’t really care. I wanted to see if it could be done, and have gotten sufficient enjoyment from this digital version of wooly willy.

The end result is a web server running on your local machine, that you can then access from any other machine on your network. By default, this runs on port 9000 and gives you a fairly detailed interface to control the process. You are going to need some prompts and honestly… the best guide that I have found about this process is this one. More specifically it was helpful in understanding the concept of negative prompts… aka the things that you want to steer the engine away from creating. This is going to be really important if you are working with anything that could be considered the human form because like a pre-teen boy… it seems to be drawn to replicating boobs in the strangest ways. There are negative prompts that you can supply to the process in order to sort of steer it away from that particular uncanny valley.

The other thing you are probably going to want is some more models to play with. You could get really deep in the weeds in trying to explain exactly what a model is… but effectively think of it as encoded data that tells Stable Diffusion how it should produce images. The best place for these seems to be CivitAI.com and on the front page, you will find a number of the more popular options. I’ve played with several of these and after fiddling around a bit, I think I probably like DreamShaper the best because it tends to lean towards more imaginative imagery rather than attempting to replicate reality. Wherever you installed Easy Diffusion, you should find a directory along the lines of “EasyDiffusion\models\stable-diffusion”. When you download a model from CivitAI it will be a “.safetensors” file and you just need to drag it into this directory in order to use it. Something important to note… each model is roughly 2 gigabytes in size so they can rapidly fill your drive if you download too many.

Let’s walk through a multi-hour deep dive that I did yesterday in trying to get something interesting out of the generator. I wanted a Kaiju attacking a city, but I assumed that maybe the model wouldn’t understand the term Kaiju. Essentially when writing a prompt I find it best to sort of use simple language. So for example this is the prompt that I supplied to get the above image.

A giant monster attacking a modern city with the military fighting back against it

This was maybe a little too on the nose and had a distinct Godzilla feel to it. Since I tend to generate five images at a time, you can often see it going in a bunch of directions. Some of these were more akin to something that you would find in a Doom video game, but I mostly liked the general direction it was going. Essentially what I usually do is feed the image that was just generated back into the prompt in order to iterate on that idea. I find that over the course of what ends up being a few hundred generations, it slowly narrows down the focus to get closer to what you were actually wanting.

I wanted to go in a different direction, so I tweaked my prompt a little bit and fed it the previous image.

A giant monster that looks like Cthulhu attacking a modern city with the military fighting back against it

Basically, I supplied that I wanted it to look more like Cthulhu to steer it away from Godzilla… which worked like a charm. However, remember that bit where I said that the models seem to really want to draw boobs? I included this specific image just to show that point.

I took a bit of a pivot because I didn’t want this to end up being a pretty boring drab scene. So for the next prompt, I started adding some style elements to it. I also wanted the city to look a bit more ruinous.

a vaporwave scene featuring A giant monster that looks like Cthulhu attacking a modern city knocking down buildings some of the rubble on fire

I noticed that some of the elements of the monster how were mechanical looking, which made me start shifting in a slightly different direction. Could I get this to make it give me a scene of a Kaiju fighting a Robot over a ruined landscape? Unfortunately at this point going forward… I don’t have exact prompts. The images that generate are named based on your prompt, and my phrase got too long for the file character limit.

I kept the prompt above but instead added that it was fighting a Giant Robot that was wielding a laser sword. The generator got confused as it often does… and just started making the monster look more robotic. While extremely cool looking… this was not really what I was going for, but it was a key step in the process nonetheless.

I tried a few more rounds of generation, feeding my favorite from the previous round into the next round… but no matter what I tried it seemed to be hung up on a single “monster” figure. I am including this one mostly just because it looked pretty freaking badass. I have no clue what is going on with that building but I am on board with the bio-organic mech with a giant laser sword thing that is going on here.

It was around this point that I decided that I needed to tweak things up a bit further. So I specifically called out that there were two figures, the monster, and the giant robot and this finally began to produce paydirt. It started out a bit ephemeral at first, with this pseudo-robot-building thing in the background. However, that gave me a thread to expand on, again feeding one generation into the next round of five generations.

After a few more rounds of generation, the idea was starting to finally take root. I say idea because in fiddling with this nonsense, it does seem like the generator gets something stuck in its head and you have to sort of forcibly dislodge it at times. I was getting somewhere though, but knew it would take many more instances of taking an image that was the closest to what I wanted, and then feeding it into the next loop… and doing this over and over until the results started to turn in the direction I wanted it to.

Then after legitimately two or three hundred rounds of this nonsense and the course of an entire afternoon wasted… we have this glorious piece of nonsense. I think what I find so interesting about Stable Diffusion so far, is that it can serve as this rapid ideation platform. If you want to quickly iterate on some ideas you had in your head, you can come up with something that is still… very visually wrong at times but contains the flavor of what you wanted. I could see this being honestly an amazing tool for an “actual artist” to test out some ideas and have the machine keep iterating on something until they get a layout and subject matter that they wanted… that could then serve as the scaffolding to build something interesting. Even for wordsmiths, this could be super interesting because I can already wrap a story in my head around what is going on in this picture.

It is also sometimes just super interesting to feed it a prompt and see where it goes. This is a hundred or so generations off of the prompt “Belghast”. No clue why but it seemed to really latch onto a military and zombies theme when I used my chosen moniker as the prompt.

This delightfully ominous scene was generated with this prompt:

a skinwalker stalking a group of hunters in the woods

So while I would not at all consider that to be a successful prompt as far as subject matter goes… it still looks freaking cool nonetheless. Again this image has a story to it that is just begging to be told. I feel like trying to make one of these image generators create exactly the image that you were wanting… is a path to madness. However, if you sort of go with the flow and iterate on the patterns that you are seeing emerging… it can produce some really interesting things. While I don’t exactly consider this art on the same level as the things being produced for me by Ammo… there is definitely an art form that is emerging from guiding the machine. It somewhat reminds me of carving a woodcut block, and allowing yourself to lean into the imperfections of the material… rather than trying to fight against them. Anyways I thought this was interesting. No clue if anyone else cares about it, and I have no clue if I will ever walk through one of my generative steps again in the future. The cool thing about this blog is it is a “me” blog more than it is a blog devoted to any one particular topic. I’m enjoying creating nonsense with Stable Diffusion and thought I would share that with you all. The post Fun with Stable Diffusion appeared first on Tales of the Aggronaut.