My best, non technical attempt to explain scenes is that they are more abstracted - rather than directly calculating each pixel for every frame independently, the Scene code allows you to draw shapes and objects. These shapes and objects can have relationships with others and react to each other. The code is stateful too - an object’s previous state can influence its future state, like simulating physics.
Many thanks @bcrichards, really appreciate that. I’m sort of getting it.
So, by comparison, shapes in shaders are illusions; defined pixel behaviors which when combined we perceive as shapes but which aren’t really ‘there’ as defined specific shapes in the code - would that be getting close to the difference?
What about the second part of my question? Can Scenes be coded to accept more than one Texture input at a time or is that a technical impossibility?
Another (possibly stupid) question for the Visual experts here: is there a way to quickly convert a Shader into a Processor Shader? Sort of feels like there should be but maybe I’m missing something fundamental?
I think that the categories Shader and Shader processor may be somewhat interchangeable. I think it’s a matter of adding in required Texture input to a shader and it will then have the capacity to function as a processor.
Thanks again @bcrichards, very helpful. Would you be able to confirm this with instructions to carry it out @giku? Would be really useful to know if it’s that easily done. Cheers.
@Robin have you tried experimenting with any LLMs yet?
It's a fairly straightforward process in copying the code from a shader, paste it into your favorite LLM, along with some instructions like "convert this shader code to a Shader Processor".
Of course you will want to attach the appropriate "contract file" which are available both here in the forums or in the visual docs.
So step one, attach the txt file (shader contract or scene contract)
step two, provide the instructions; "Convert the following shader code to a shader processor"
step three, paste the shader code after your instructions.
Btw, this also works for converting shaders into masks. I've made dozens of masks this way.
Thanks, that doesn’t seem to work though. Pasting the code from a shader into a processor shader module doesn’t result in the processor shader module having a texture input? Highly probably I’m misunderstanding though!
Thanks @Intrepolicious, appreciate you taking the time. Yes, I’ve experimented with AI coding shaders and have asked it to convert from shader to processor shader so I’m aware that’s possible.
I was initially just wondering whether it was really easy process to convert one to the other so that I could avoid having to do that though - like adding a simple line of code at the beginning to change one to the other?
Robin, the way you have it written out, it’s opposite. Paste the processor shader code into a shader. It worked for me - just as an example, it copies the exact functionality so there’s no point here. But the experiment shows, yes, Shader module can have Texture input.
Thanks @bcrichards. Yes, that works but I was wanting to know if there was a quick and easy way to make a shader into a processor shader. Your experiment shows that in theory it should be possible but it doesn’t actually reveal how to do it. Or am I still missing something? Apologies for my lack of understanding.
Thanks @pedro. Your example code works as you said it should.
With regard to the line ‘texture2d<float> InTexture [[input]]’ to make a shader a processor shader, I can see that it’s there in the code of processor shaders but I can’t work out the logic of where it needs to be pasted into the code, at what position, at what line?
Is there any way for someone with no coding knowledge at all to be able to work out where it needs to be posted in the code?
Absolut no coding knowledge? No. But you can get by, by understanding the bare minimum. Like function render() parameters
a shader is a little program that processes the pixels on the screen. If you give it a texture input then you can use that texture (image, video, other shaders) and perform math operations on it. In my example I multiply the input texture by a blueish color
a shader (processor or not) does just that, output a manipulated color, that you have to somehow compute in a way that works for every pixel. If you want to get more familiar with the whole thing I suggest “the book of shaders”. It’s not MSL but the principles are the same
@Robin i think there may be some misunderstanding in my side too :) yes it’s possible, but I’m the wrong person to talk to about how to write your own code. I’ve had plenty of success using LLMs for Drambo code. If I were feeling more studious I’d invest time in actually learning the code rather than just generating it.
Thanks @pedro. Yes, I think I just about get what the shader is doing regarding pixels but the actual code is mostly just meaningless to me. I can sometimes identify and manage to adjust parameters in existing code but that’s it. I just have zero background in coding.
I’ll look into ‘the book of shaders’ but think I’m going to have to take the shader kit rather than the coding route.
Out of interest, everyone here who was on the beta seems to know how to code shaders to a greater or lesser extent; was there anyone on the beta who started off without any understanding at all? It’d be interesting to hear how they managed to learn and get a grasp on things if they happen to read this?
Thanks @gravitas, really useful to know that. Encouraging that a lot of the people I assumed knew about coding are actually using AI to do it for them; makes me feel a bit better about my complete lack of understanding!
Regarding getting used to the modules and routing, I’m starting to do that as I experiment within the Shader Kit concept.
One thing which would be useful which I don’t feel is laid out in the Docs really, is some clear information on general rules for what order modules are placed in. For example, does a transform module come before or after a colour module and before or after a mask? Maybe there aren’t any rules in which case fair enough, but some things work when they’re in one place and not another and I don’t quite understand the logic of why that is (though of course there will be legitimate reasons) so that would suggest that there are rules which I’m not understanding yet. Maybe the Recipes section in the Docs will illuminate this once it arrives?
Funny to think that when visuals first started beta, AI wasn’t even a thing, I ported several GLSL shaders painstakingly by hand. Now it’s mostly a matter of explaining it to an LLM and you have it 90% done at least
It is explained in the docs, basically you have to think in this order: xy(z) manipulation—> shader processing—> color. A shader does all in one, but with shader kit modules that’s the flow
have you tried the tutorial presets? Playing with those may clear many of your doubts. If you get lost just ask here for directions
As soon as I have some free time, I want to start posting demonstrations of things that can be done, and maybe try to inspire some folks.
@Robin don't worry about not knowing how to code, I'm right there with you. I have learned a few things along the way, and can identify certain lines and what some values represent here and there, but I came into this not knowing a thing about coding, and still don't know all that much. The LLMs will try to explain things as you go. You can even ask them to teach you at your own pace.
Some of my big hobbies back in the day (along with music of course) were designing maps and levels for games like Doom, Quake, Half Life, etc and all the editing applications were very much like CAD programs, where having skills with geometry and architectural design definitely help. Always felt like I had an edge in photoshop and other paint/art applications, and always excelled at those things.
One time (to try and make a really long story short) I even designed a 3D model/replica playable map of the restaurant I was working at at the time. I had every inch of detail all the way down to the silverware and salt & pepper shakers on the tables, a fire on the stove in the kitchen, pots and pans, you name it. It was a death match map for Half Life 2 Deathmatch. I would host it, and myself and the crew from work would jump in and play it at night after work, picking up tea urns with the gravity gun and throwing them at each other was so much fun!!! Anyway, I started recording videos and was posting them on YT. This was when the corporate people found out.. as I was stupidly using the name of the business as the map name. Well long & short, I got threatened with a huge lawsuit and had to take everything down and delete it, as they didn't want to associate themselves with such "violence". My design (Which was all done by hand, and by memory) was SO good, that it really looked like the actual restaurant, and depicted people running around shooting each other. They weren't having it..
Edit: Hmm I don't know why I felt the need to offer a glimpse into my past experiences, but there it is. Maybe too much coffee this morning lol
Anyway, back to coding, yeah those with coding skills have plenty of access in most of all visual modules.
Those that don't still have access through modules, and a little creative know how.
This exact thing right here, is what I believe is/was at the heart of what @giku's vision is for Drambo Visual; making it modular, where coders and non-coders alike can create. He's taken most of the typical things one might code, and "modularized" them into these amazing building blocks that anyone can use. He did all this of course, while at the same time leaving the back door open on most modules where they are indeed "codeable" where users can roll their own, if so inclined.
All of the recent visual apps facilitate shader coding in one way or another; VS - Visual Synthesizer does, so does Lucinate (by the AUM dev) but none have the flexibility of being fully modular like Drambo Visual.
I think what I'm trying to say is that "coders" can code visuals and shaders anywhere.
Cool, thanks @pedro…yeah, I’ve gone through some but not all of the tutorials; I’m pretty sure I will be posting more questions here as I work my way through things. Cheers.
Yes, the more demonstrations yourself and others can post, the better I feel.
Yeah, Drambo Block is a real problem and, I’m afraid, applies with Visual just as much as the audio area. The thing I would suggest though is that it should be acknowledged that there are parts of Drambo which are really complicated to get your head around if you’re coming to it without any previous modular experience. In the case of Visual, if it’s taken dedicated beta testers 3 years to get to grips with it, that’s potentially a big ask for some regular users?
Personally, I’m willing to put in the time to learn (and code via AI where necessary) but I’m trying to work out how I can use it to meet my own creative goals, my own aesthetic, rather than being lead by it to create the regular ‘shader’ type of visuals which are of limited interest to me. This might be why I’m having problems; because I’m trying to achieve things that Visual isn’t really designed to do? Time will tell, but there are things I can see I will be able to achieve with it already so I’m certainly inspired to keep trying things even if I will no doubt have to accept that some things are simply better done elsewhere in other apps. Cheers.
Yep. Use it all the time. That, AirDrop, and Handoff are among the few remaining reasons I stick with MacOS rather than Linux for desktop work on this trusty but old 2015 MacBook Pro.
Comments
My best, non technical attempt to explain scenes is that they are more abstracted - rather than directly calculating each pixel for every frame independently, the Scene code allows you to draw shapes and objects. These shapes and objects can have relationships with others and react to each other. The code is stateful too - an object’s previous state can influence its future state, like simulating physics.
Many thanks @bcrichards, really appreciate that. I’m sort of getting it.
So, by comparison, shapes in shaders are illusions; defined pixel behaviors which when combined we perceive as shapes but which aren’t really ‘there’ as defined specific shapes in the code - would that be getting close to the difference?
What about the second part of my question? Can Scenes be coded to accept more than one Texture input at a time or is that a technical impossibility?
Another (possibly stupid) question for the Visual experts here: is there a way to quickly convert a Shader into a Processor Shader? Sort of feels like there should be but maybe I’m missing something fundamental?
I think that the categories Shader and Shader processor may be somewhat interchangeable. I think it’s a matter of adding in required Texture input to a shader and it will then have the capacity to function as a processor.
Thanks again @bcrichards, very helpful. Would you be able to confirm this with instructions to carry it out @giku? Would be really useful to know if it’s that easily done. Cheers.
You can try it yourself Robin. Just take code from Shader processor, copy and paste it to a Shader module.
@Robin have you tried experimenting with any LLMs yet?
It's a fairly straightforward process in copying the code from a shader, paste it into your favorite LLM, along with some instructions like "convert this shader code to a Shader Processor".
Of course you will want to attach the appropriate "contract file" which are available both here in the forums or in the visual docs.
So step one, attach the txt file (shader contract or scene contract)
step two, provide the instructions; "Convert the following shader code to a shader processor"
step three, paste the shader code after your instructions.
Btw, this also works for converting shaders into masks. I've made dozens of masks this way.
Thanks, that doesn’t seem to work though. Pasting the code from a shader into a processor shader module doesn’t result in the processor shader module having a texture input? Highly probably I’m misunderstanding though!
Thanks @Intrepolicious, appreciate you taking the time. Yes, I’ve experimented with AI coding shaders and have asked it to convert from shader to processor shader so I’m aware that’s possible.
I was initially just wondering whether it was really easy process to convert one to the other so that I could avoid having to do that though - like adding a simple line of code at the beginning to change one to the other?
Robin, the way you have it written out, it’s opposite. Paste the processor shader code into a shader. It worked for me - just as an example, it copies the exact functionality so there’s no point here. But the experiment shows, yes, Shader module can have Texture input.
If you add this line to a shader it becomes a shader processor. Can have several texture inputs too
Example: put an image before, use this code in the shader for a blue version of your image
float4 render(device Data &data, uint2 tid,
texture2d<float> InTexture [[input]]
) {
return InTexture.read(tid)*float4(0.5,0.5,1.0,1.0);
}
Thanks @bcrichards. Yes, that works but I was wanting to know if there was a quick and easy way to make a shader into a processor shader. Your experiment shows that in theory it should be possible but it doesn’t actually reveal how to do it. Or am I still missing something? Apologies for my lack of understanding.
Thanks @pedro. Your example code works as you said it should.
With regard to the line ‘texture2d<float> InTexture [[input]]’ to make a shader a processor shader, I can see that it’s there in the code of processor shaders but I can’t work out the logic of where it needs to be pasted into the code, at what position, at what line?
Is there any way for someone with no coding knowledge at all to be able to work out where it needs to be posted in the code?
Absolut no coding knowledge? No. But you can get by, by understanding the bare minimum. Like function render() parameters
a shader is a little program that processes the pixels on the screen. If you give it a texture input then you can use that texture (image, video, other shaders) and perform math operations on it. In my example I multiply the input texture by a blueish color
a shader (processor or not) does just that, output a manipulated color, that you have to somehow compute in a way that works for every pixel. If you want to get more familiar with the whole thing I suggest “the book of shaders”. It’s not MSL but the principles are the same
@Robin i think there may be some misunderstanding in my side too :) yes it’s possible, but I’m the wrong person to talk to about how to write your own code. I’ve had plenty of success using LLMs for Drambo code. If I were feeling more studious I’d invest time in actually learning the code rather than just generating it.
Thanks @pedro. Yes, I think I just about get what the shader is doing regarding pixels but the actual code is mostly just meaningless to me. I can sometimes identify and manage to adjust parameters in existing code but that’s it. I just have zero background in coding.
I’ll look into ‘the book of shaders’ but think I’m going to have to take the shader kit rather than the coding route.
Out of interest, everyone here who was on the beta seems to know how to code shaders to a greater or lesser extent; was there anyone on the beta who started off without any understanding at all? It’d be interesting to hear how they managed to learn and get a grasp on things if they happen to read this?
Cool, no problem at all, I really appreciate the time you’ve taken explaining things. Cheers.
I’ll look into ‘the book of shaders’ but think I’m going to have to take the shader kit rather than the coding route.
good choice, giku has gone through great lengths to make it available to us, without coding
was there anyone on the beta who started off without any understanding at all?
can only speak for myself, but three years ago I had no idea what a shader was, thought a GPU was like a CPU only faster
now I still don’t know what a gpu is, the complex math eludes me, but I can fake it, copy other people’s shaders and have fun with it
For anyone that uses excel or any spreadsheet, this video really was enlightening for me back then, in changing the mindset from cpu to gpu
https://www.youtube.com/watch?v=JnCkF62gkOY
Cool, good to know, thanks for sharing. Cheers.
On my part, I didn't know how to A.I code for DSP until a few months ago when I discovered that we had a Code module for the DSP.
About a month later or so ( I documented my exploration for future reference)
I started coding for the Scene module and about two weeks ago I started coding for the Shader kit.
I use three A.I engines to cross reference the code especially when the LLM's make mistakes or I cannot describe what I need to hear or see.
If you don't know how to code as yet then my advise would be to get to grips with how the modules and routing interact with each other
using stock dRambo modules, once you're familiar with how that side of things work then explore the A.I coding side of things.
As a starting point, there are two sections with Shaders, for the first section think of it more as 2D generated images as in X/Y
and for the second section of Shaders that's X/Y/Z.
Thanks @gravitas, really useful to know that. Encouraging that a lot of the people I assumed knew about coding are actually using AI to do it for them; makes me feel a bit better about my complete lack of understanding!
Regarding getting used to the modules and routing, I’m starting to do that as I experiment within the Shader Kit concept.
One thing which would be useful which I don’t feel is laid out in the Docs really, is some clear information on general rules for what order modules are placed in. For example, does a transform module come before or after a colour module and before or after a mask? Maybe there aren’t any rules in which case fair enough, but some things work when they’re in one place and not another and I don’t quite understand the logic of why that is (though of course there will be legitimate reasons) so that would suggest that there are rules which I’m not understanding yet. Maybe the Recipes section in the Docs will illuminate this once it arrives?
Funny to think that when visuals first started beta, AI wasn’t even a thing, I ported several GLSL shaders painstakingly by hand. Now it’s mostly a matter of explaining it to an LLM and you have it 90% done at least
Time sure flies
It is explained in the docs, basically you have to think in this order: xy(z) manipulation—> shader processing—> color. A shader does all in one, but with shader kit modules that’s the flow
have you tried the tutorial presets? Playing with those may clear many of your doubts. If you get lost just ask here for directions
Crazy ain't it? Remember when we first started, we were like "aw cool; we made a circle!"
Now look at @giku 's baby all grown up. ♥️
As soon as I have some free time, I want to start posting demonstrations of things that can be done, and maybe try to inspire some folks.
@Robin don't worry about not knowing how to code, I'm right there with you. I have learned a few things along the way, and can identify certain lines and what some values represent here and there, but I came into this not knowing a thing about coding, and still don't know all that much. The LLMs will try to explain things as you go. You can even ask them to teach you at your own pace.
Some of my big hobbies back in the day (along with music of course) were designing maps and levels for games like Doom, Quake, Half Life, etc and all the editing applications were very much like CAD programs, where having skills with geometry and architectural design definitely help. Always felt like I had an edge in photoshop and other paint/art applications, and always excelled at those things.
One time (to try and make a really long story short) I even designed a 3D model/replica playable map of the restaurant I was working at at the time. I had every inch of detail all the way down to the silverware and salt & pepper shakers on the tables, a fire on the stove in the kitchen, pots and pans, you name it. It was a death match map for Half Life 2 Deathmatch. I would host it, and myself and the crew from work would jump in and play it at night after work, picking up tea urns with the gravity gun and throwing them at each other was so much fun!!! Anyway, I started recording videos and was posting them on YT. This was when the corporate people found out.. as I was stupidly using the name of the business as the map name. Well long & short, I got threatened with a huge lawsuit and had to take everything down and delete it, as they didn't want to associate themselves with such "violence". My design (Which was all done by hand, and by memory) was SO good, that it really looked like the actual restaurant, and depicted people running around shooting each other. They weren't having it..
Edit: Hmm I don't know why I felt the need to offer a glimpse into my past experiences, but there it is. Maybe too much coffee this morning lol
Anyway, back to coding, yeah those with coding skills have plenty of access in most of all visual modules.
Those that don't still have access through modules, and a little creative know how.
This exact thing right here, is what I believe is/was at the heart of what @giku's vision is for Drambo Visual; making it modular, where coders and non-coders alike can create. He's taken most of the typical things one might code, and "modularized" them into these amazing building blocks that anyone can use. He did all this of course, while at the same time leaving the back door open on most modules where they are indeed "codeable" where users can roll their own, if so inclined.
All of the recent visual apps facilitate shader coding in one way or another; VS - Visual Synthesizer does, so does Lucinate (by the AUM dev) but none have the flexibility of being fully modular like Drambo Visual.
I think what I'm trying to say is that "coders" can code visuals and shaders anywhere.
The one thing we have to get some people to overcome though, which seems to be a strange phenomenon, and for some reason, common.
It's an ailment we'll refer to as "Drambo Block" 😅
look at @giku 's baby all grown up. ♥️
hehe better cherish these moments cause I know from experience they are fleeting
Cool, thanks @pedro…yeah, I’ve gone through some but not all of the tutorials; I’m pretty sure I will be posting more questions here as I work my way through things. Cheers.
@Intrepolicious, really entertaining long post, thanks.
Yes, the more demonstrations yourself and others can post, the better I feel.
Yeah, Drambo Block is a real problem and, I’m afraid, applies with Visual just as much as the audio area. The thing I would suggest though is that it should be acknowledged that there are parts of Drambo which are really complicated to get your head around if you’re coming to it without any previous modular experience. In the case of Visual, if it’s taken dedicated beta testers 3 years to get to grips with it, that’s potentially a big ask for some regular users?
Personally, I’m willing to put in the time to learn (and code via AI where necessary) but I’m trying to work out how I can use it to meet my own creative goals, my own aesthetic, rather than being lead by it to create the regular ‘shader’ type of visuals which are of limited interest to me. This might be why I’m having problems; because I’m trying to achieve things that Visual isn’t really designed to do? Time will tell, but there are things I can see I will be able to achieve with it already so I’m certainly inspired to keep trying things even if I will no doubt have to accept that some things are simply better done elsewhere in other apps. Cheers.
Yep. Use it all the time. That, AirDrop, and Handoff are among the few remaining reasons I stick with MacOS rather than Linux for desktop work on this trusty but old 2015 MacBook Pro.