There are many ways to achieve this... I will just show you my favorite one. depending on their structure, rocks look very different. For this tutorial I decided to make sharp reefs in the water ... but you can use the same technique for anything stony!
First, we will start with a white canvas. Make a new layer and name it "outline" . On this layer we'll draw the outline, the shadows and the texture of this rock! I use soft brush to make a rough sketch of the rocks ... Make it simple, just a general idea of how the rocks will look like!
Now I add some details. At this point it is smart to determine a light source! I put the light source in the top-left corner, so the shadows should be visible on the bottom-right side of the rocks. Use a smaller and harder brush to make a few details on the rock. Use a photo reference for painting if you have a hard time putting out details.
Continue with doodling, until you paint all of the rocks. Details are not important at this stage, we'll add them later!
Now we take a soft brush again, with opacity set to about 15% and start with shading. Remember, the light is coming from the upper left corner so the shadows should be on the opposite side of the rocks!
When you are finished with shading, use a smudge tool to smooth the surfaces. I use a hard round brush with about 40% strength!
Keep smoothing until you feel satisfied with the result. You can even add details to the rock simply by using a smudge tool. But be careful not to blur the edges of the rock.
Usually rocks aren't smooth ... So, we'll add few details and textures to it! I use a rough spattered brush, with the opacity set to 10-20%. Feel free to improvise at this point.
We are now done with drawing the rock! All we have to do now is to color it. Make a new layer beneath the "outline" layer and name it "color". Set the "outline" layer's blending mode to "multiply". That way the dark areas of this layer effect the bottom ones, but you can still see the true color of the layers beneath. Choose a shade for your rock (for this one I use blue-gray) and color the rock on "color" layer with the brush.
Now all we have to do is adding some gloss to the brighter areas. I use a dodge tool for this task.
We are finished now! Well at least with the rock part of this picture. Remember ... rocks usually have a rough texture, but their shape could be anything ... from rounded to squared or even flat... nature itself has the greatest imagination. With this technique you can reproduce all of them!