Layer masks in Photoshop – Get perfect smooth transparency on your masks

Colin Smith

Subscribe on YouTube:

How to get smoother masking inside of Photoshop.

Alright, so I’m going to show you something about masking that few people realize. So we’re just gonna use this picture of a woman and a cat on top. So this is not so much about the artwork is this is just understanding the technique I want to show you.

Portrait of a woman with long straight blonde hair on a gray background in Photoshop

Ginger cat photo placed on a layer above the woman's portrait

So the first thing we want to do is create a hidden mask or an inverted mask. So hold down the alt or the option key and then go in the layers panel and click on create mask. What that does is it creates a black mask which hides the contents of the layer.

Hidden black layer mask added to the ginger cat layer

Now to bring back the contents of the layer, we’re going to paint white in here. So let me just hit the D key to reset the foreground background colors.

Let’s grab a brush and we’re going to start to paint. Now. Some people use flow. Some people use opacity. Check out my tutorial that talks about the difference.

Brush tool icon in the Photoshop toolbar

I’m going to drop the flow down to 10% and we’re gonna start to paint in a little bit here. See, so what happens when you want to paint in a specific value, like maybe you want to do 20% here.

Cat's green eyes and fur painted onto the woman's face through the mask

This is the mistake that most people do. Most people drop the opacity down to 20% and that is wrong. And I’m going to show you exactly why.

Brush options bar with Opacity at 20 percent and Flow at 100 percent

Opacity is basically how transparent it is in the flow is how fast you get there. See my tutorial on Flow vs Opacity

How a layer mask works is simple. When you paint with white, it reveals the contents of that particular layer. If you paint with black, it hides the contents of that particular layer. Check out my other tutorial on masking basics. Let’s continue.

Diagram of a blue top layer masked with a brush stroke revealing the yellow layer below

What most people would do if they wanted to paint 20% is to drop opacity down to 20% and this is wrong. And I’ll show you why.

If you paint at 20% opacity you lay down some paint. Now, it doesn’t matter if you’re using a mouse or a graphics tablet.

Soft white brush dab on a gray mask preview

But here’s the thing. Yes, it’s adding 20%. But if I lift and then I paint again, it adds another 20%. If I lift, it adds another 20% and so on.

This creates very splotchy paint (or a splotchy mask).

Splotchy overlapping strokes from painting the mask at low opacity

And then at this point to try and get everything smooth and even, is very, very difficult. So opacity is not the way to do that. I’m going to turn the opacity all the way up. Rather than using Opacity,  go under the color picker.

Foreground and background color swatches circled in the toolbar

Choose (B) Under HSB. This is going to give us brightness. Now choose 20. So rather than choosing an opacity, we’re choosing a shade of Gray with 20% white.

Color Picker set to 20 percent brightness, a dark gray of 333333

Click ok, now  watch what happens if I go up to the other side here and I start to paint and I lift it and I paint on top of it.

Even gray strokes on the mask without splotchy overlaps

Look at this. I’m getting exactly 20%.

But this does even more because when we use Opacity or flow, what it’s going to do is it’s just essentially going to keep stacking more and more paint.

Remember the first area I painted on the left, supposed to be 20%. Now if I paint, this would just take it straight back to 20. So if it’s a lighter, it will darken it. If it’s darker, it will lighten it and look how smooth and perfect we can get that.

Smooth gray mask strokes painted with a solid gray color

Rather than use transparency, use gray. See how the tones map to transparency values.

Chart matching layer opacity to mask brightness on a Venetian carnival mask photo

Now if we combine this with flow and let’s take that flow down to a very low amount. So this enables us to blend these edges and get them beautiful and smooth and precise.

How would this apply on the image? Simple. Let’s choose the mask.

Color Picker set to 40 percent brightness gray, 666666

Choose 40% by entering 40 into B in the Color picker.

We’re going to get 40% of that fur. Look at that. Just a little bit of texture there. Let’s do the same top of the cheeks.

Woman's face blended with smooth cat fur and green cat eyes

Why don’t you try this method for yourself next time you’re trying to create a little bit of transparency in your mask and you’ll find that this technique will get you much smoother, more even masks.

So I’m curious. Guys, did you know this already? Was this something new? Let me know in the comments underneath.

And by the way, if you’re new, welcome to Photoshop Cafe. Do me a favor and hit that subscribe button. And that way you won’t miss any of my concise tutorials. So anyway, guys, I hope you like this.

And until next time I’ll see you at the cafe.

Colin


Follow along for more tips

Recommended for this tutorial

Learning Layers in Photoshop.

Understand layers, masks and blend modes, the foundation of everything in Photoshop.

$54.99 · one-time purchase · instant download

Photoshop Vault: free Layer Blend Modes ebook, actions, brushes, presets and cheat sheets

Free for you

The Photoshop Vault

Get my free Layer Blending Modes ebook along with dozens of exclusive Photoshop goodies: actions, brushes, presets and cheat sheets.

You're on the list, so your free goodies are waiting for you in the Vault.

Get Free Photoshop Goodies Open the Vault

22 comments

  1. Jorge Parra Photography

    Very nice tip!! best possible transitions for most composites. Thanks for this great piece of digital wisdom!!

  2. I use masks often and am aware of opacity vs flow, but had not heard this. I do sometimes paint at a low O or F and adding has been unsuccessful so my solution, usually is to paint at 100% and reduce O. This has been very helpful. Thank you

  3. This is brilliant! and no, I did not know about using Brilliance with the color-picker

  4. Total game-changer for me. I have almost worn out out my keys trying to get even application of less than 100% black masks. One of the things I really like about your tutorials is that that each one just focuses on learning a single skill, learning it quickly, and showing a practical example.
    When I watch a tutorial that goes through a series of techniques to “fix” a photo, I seldom retain the information. It’s too much to take in all at once and just becomes a blur. By breaking things down into small pieces, I am much more likely to practice and repeat.

  5. Dennis Zito

    Hi Colin,
    Nope!!! Did not know this. I’ve always gone with low flow and about 80% opacity. Now, I going to try this out. It looks like a much much better technique.
    Dennis

  6. colourbug

    I am so pleased I joined the Cafe….the videos are helping a great deal. Soooooo useful in my work. I download the notes so I can make extra notes while I watch the video.
    Thankyou Colin.

  7. mike yates

    thanks Colin for that masking tip, i did not know that . so i shall be trying it on my next edit once again thanks

  8. Margaret Black

    Colon, actually i have never seen this method for masking before. It makes a lot of sense. I do have trouble with unevenly painted masks because of the buildup of the paint. I am looking forward to trying this method of painting masks.
    Thank you for this tip.

  9. SUSAN Denise BERGNA

    Nope. I’ve never tried this and it makes so much sense. Thanks! Great job at the PS Summit, by the way. Just finished the ACR presentation. Dynamite!

  10. A new and innovative way to better blend in composites. Thanks Colin.
    This is one of the issues about PS: there are so many options to do the same thing that the software intimidates new users. It’s so good to have detailed, step by step S2F instructions.

  11. Pat Vojtech Keene

    Thanks, Colin. I’ve had this problem before with overlapping opacity/flow increasing as edges double up, creating blotched look. I have wasted lots of time trying to fix it. Thank you!!

  12. Eva Nestell

    Thanks Colin, really useful! I’m just wondering, if I do a mistake and want to “paint back” the mask, how do i do that? I have always done the opposite, used “white” masks and painted away with black the parts I don’t want. I then paint with white if I need to “undo” something. Now I’m a bit confused about how to undo working with a black mask.

Leave a comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Copy link