Masked Blend Mode Shows Ambient Occlusion from Behind Object

Node-based Shader Editor

Masked Blend Mode Shows Ambient Occlusion from Behind Object

Postby moosefreak » Wed Nov 22, 2017 12:25 am

Hi I've been trying to create a shader with an opacity mask similar to the standard shader "Cutout" rendering mode, but it appears to be incompatible with screen-space ambient occlusion. Here is a comparison of the Standard Shader and Amplify:

Standard Shader:
Image

Amplify Shader:
Image

Nodes:
Image


The other option I've tried is using a custom blend mode with Opaque and Transparent or AlphaTest but that causes AO to go around the entire object's quad which is almost worse.
moosefreak
 
Posts: 6
Joined: Wed Nov 22, 2017 12:13 am

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby Amplify_Borba » Wed Nov 22, 2017 10:30 am

Hello, thank you for getting in touch!

In order to best help you, we'll need some further information regarding this issue.

What occlusion solution are you currently using in your project, and what are its parameters?

Are you using forward or deferred rendering path?

Looking forward to your reply!
Customer Relations at Amplify Creations
Learn more about our offering: Amplify Creations Products
Amplify Shader Editor won the Asset Store Best Tool Award - Thank you for your support!
User avatar
Amplify_Borba
 
Posts: 1239
Joined: Mon Jul 24, 2017 9:50 am

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby moosefreak » Thu Nov 23, 2017 1:45 am

Hi, I’m using the Unity Post Processing Stack, the default Ambient Occlusion. i’ve also tried the Cinematic Image Effects Ambient Occlusion with the same issue. I’ve used every parameter with no change in both solutions. I’m using forward rendering, but get the same issue in deffered
moosefreak
 
Posts: 6
Joined: Wed Nov 22, 2017 12:13 am

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby moosefreak » Thu Nov 23, 2017 4:34 am

Here are my Ambient Occlusion settings in the Post Processing Behavior for Unity's Post Processing Stack:
Image

This is what the Normals debug view looks like in the Post Processing Behavior:
Image

This is what the Depth debug view looks like in the Post Processing Behaviour:
Image

What the Ambient Occlusion debug view looks like in the Post Processing Behavior:
Image


So it seems apparent that the cutout (masked blending mode: Transparent-Cutout + AlphaTest) in Amplify does not write to the Normals buffer, causing the problem. Whereas the Unity Standard Shader does.

Is this a bug? It's been like this in the past few versions, (I am fully up to date) not sure if it ever worked before though.
moosefreak
 
Posts: 6
Joined: Wed Nov 22, 2017 12:13 am

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby Amplify_RnD_Rick » Thu Nov 23, 2017 3:26 pm

Hey moosefreak,

We've been doing some tests on our end and it seems this is a Unity limitation. But don't worry as we have figured out a way to surpass it.

So what is happening is exactly like you said, the Depth Normals are not being correctly written, as it's ignoring the clipped values.

This issue is only present when rendering in the Forward Rendering path as when on deferred, the G-Buffer is used and it is correctly constructed.

So lets head to how to fix it in Forward. The way you make it write the depth normals correctly is by setting the properties you use on your shader to create the mask, to have the exact name as the ones used by the Unity internal Depth Normal shader and also their usage to also be exactly the same as Unity as well.

It's a bit confusing but here's a screenshot exemplifying what I've said:

AOTest.jpg
AOTest.jpg (178 KiB) Viewed 6591 times


Notice that we've created a Texture Sampler node with the name MainTex and a Color node with the name Color. The Alpha channel of both nodes are multiplied with each other and its result is set into the Opacity Mask port.

Again, for this method to work, both the properties names must be those ones and usage of each nodes alpha channel to create the opacity mask must be that exact way.

Please let us know if you have further questions as we will gladly help you out.
Amplify_RnD_Rick
 
Posts: 40
Joined: Wed Mar 01, 2017 6:33 pm

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby moosefreak » Thu Nov 23, 2017 11:09 pm

Wonderful! Just want to emphasize (for anyone looking for something similar and because it took me a while to figure out) the Color node HAS to be a Property and not a Constant.

Thank you so much for your help.

I do have a question though, if this is a limit of Unity will this never be fixed in Amplify? And does this make it impossible to change the opacity mask with, say an outline and still keep the changed mask from showing ambient occlusion from behind it?

I have a few nodes and a custom expression that produces an outline that I was planning to use for the opacity mask. But it only seems to let the original MainTex write to the normal buffer. Is there any way to fix this, like wrap my nodes up in a "virtual MainTex"? Something like that? Here's an example showing what I'm talking about.

You can still see the AO through the outline, but not the main texture sample.
Image

Image

The nodes may seem confusing, one is custom. But just the idea of manipulating the opacity mask, is that feasible? Thank you SO much for your help so far though.
moosefreak
 
Posts: 6
Joined: Wed Nov 22, 2017 12:13 am

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby Amplify_RnD_Rick » Fri Nov 24, 2017 11:11 am

No problem at all, we're here to help!

Yes, the Color node must be set as a property.

Unfortunately it's not possible to fix this on from ASE due to the nature of the issue. To give a bit more context, Unity when rendering Depth Normals for the Transparent Cutout Render Type executes the following sub-shader.

Code: Select all
SubShader {
    Tags { "RenderType"="TransparentCutout" }
    Pass {
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
struct v2f {
    float4 pos : SV_POSITION;
    float2 uv : TEXCOORD0;
    float4 nz : TEXCOORD1;
    UNITY_VERTEX_OUTPUT_STEREO
};
uniform float4 _MainTex_ST;
v2f vert( appdata_base v ) {
    v2f o;
    UNITY_SETUP_INSTANCE_ID(v);
    UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
    o.pos = UnityObjectToClipPos(v.vertex);
    o.uv = TRANSFORM_TEX(v.texcoord, _MainTex);
    o.nz.xyz = COMPUTE_VIEW_NORMAL;
    o.nz.w = COMPUTE_DEPTH_01;
    return o;
}
uniform sampler2D _MainTex;
uniform fixed _Cutoff;
uniform fixed4 _Color;
fixed4 frag(v2f i) : SV_Target {
    fixed4 texcol = tex2D( _MainTex, i.uv );
    clip( texcol.a*_Color.a - _Cutoff );
    return EncodeDepthNormal (i.nz.w, i.nz.xyz);
}
ENDCG
    }
}


This is rendered independently on what you have on your own shader.
The main line to take into account, which is the one affecting you is:
clip( texcol.a*_Color.a - _Cutoff );

So any type of variation on how the clip is done on your shader that doesn't match that simply wont be taken into account when creating the depth normals.

There's no easy way to turn to overcome this problem, you can either overwrite Unity internal Depth Normals shader, or somehow have the texture set on the MainTex to already have the outline. P.e. having a separate shader which creates the outline and render it into a render texture which is then used on this one.
Amplify_RnD_Rick
 
Posts: 40
Joined: Wed Mar 01, 2017 6:33 pm

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby moosefreak » Fri Nov 24, 2017 12:11 pm

Thanks for the quick reply. They've all been wonderful, thoughtful replies. Very informative and extremely helpful, thank you. I love your product haha.

I'll have to find a workaround, as my texture already is a rendertexture of many different sprites being animated! haha and I don't think I want to use 2 rendertextures.

I'm considering finding a way to manipulate the mask clip value to let me increase the area around the alpha. Might have to add a shadow or something behind it to do that though. Just thinking out loud.

I'm not very experienced modifying shader code (hence your product) so not sure where I'd begin tackling Unity's Internal Depth Normal Shader manually. But you don't need to go in depth into that haha, you've provided me with more than enough help.

Is this something I, you, or we should report to Unity on a bug/issue tracker or something?
moosefreak
 
Posts: 6
Joined: Wed Nov 22, 2017 12:13 am

Re: Masked Blend Mode Shows Ambient Occlusion from Behind Ob

Postby Amplify_RnD_Rick » Mon Nov 27, 2017 5:10 pm

That is a really good question on which I really can only give you my personal opinion! :D
You should report this issue to let Unity know that their implementation is not flexible enough and has flaws, but honestly this is something that is this way for so long that I'm not seeing it fixed on a near future.
Amplify_RnD_Rick
 
Posts: 40
Joined: Wed Mar 01, 2017 6:33 pm


Return to Amplify Shader Editor

Who is online

Users browsing this forum: No registered users and 0 guests