Smooth your edges

1. Starting

a. Create a new file in Photoshop.

b. Make it 400x400 pixels - 72dpi - RGB color - Transparent.

c. Fill your layer with a background color of your choice, in this case "black"

d. Name this layer "background", or something similar.

e. Lock your background layer.

2. Make the shape

a. Create a new layer.

b. Name it "shape".

c. Select the "polygonal lasso tool" , or you can use the "pen tool".

d. Create your shape.

3. The channel

a. When you have created your shape hold the selection and select the channel palette. (Window>Show channels)

b. Create a new channel by clicking the button.

c. Fill your shape selection with the color white by pressing "alt - backspace".

d. Deselect the selection. (Ctrl - D)

e. Apply gussian blur "Filters>Blur>Gussian Blur" and set its values to 4 pixels. (The higher the value is, the smoother the corners will be.)

4. Adjusting the levels

Now press "Ctrl-L" or go to "Image>Adjust>Levels" and type in the following value, 130 - 1.00 - 170. You can play around with the values, as you want, the given values are suitable for the shape Im making.
(The values, 110 - 1.00 - 145 are often used too)

5. Load channel as selection

Load channel as selection by pressing at the bottom of the channel palette. (Shortcut: Hold down the Ctrl key and click on the channel to select the shape)

6. Fill the selection

a. Now go back to the layer palette and make sure you have the "shape" layer selected and fill the selection with your chosen color. (Alt-backspace)

b. Press "Ctrl - D" to deselect the selection.

7. Final shape

Now you can do whatever you want with your shape. Apply a texture or an effect to make your shape the way you want it.