@migf1: I believe that @hector-1 is referring to the fact that when he rotates the square brush shape less than 90 degress the edges of the the brush and therefore the paint has pixelated/jagged edges.
@hector-1, I have a question. If you are trying to create a white background why don't you just flood fill that area with the the flood fill tool using the color white?
I have used "flood fill tool" to to create a white background, but this doesn't get a full whitening. See attached screenshots.
In order to whiten outside the edges of the object I use the paint square brush tool, but when I rotate this, its sides turn dentate and the edges of the image do not look well.
How could I avoid the sides of square brush tool get dentate when it is rotated?
To smooth out the brush edges try dropping its Hardness setting (even down to 0). If the image is not huge and you care for fine details, you may also try dropping the Step setting down to 1 (if that slows down the brushing, dial it up to 10 again).
An alternative approach could be to cover the areas to be whitened with a shape, using the Pen tool with stroke-color set to white and anti-alias box checked, along with the fill-color set to white.
Yet another approach could be to select the desired areas with the Freehand selection tool in Point-to-Point mode, probably adding a little bit of feathering or smoothing too, and then flood-fill the selected areas with white color (feathering and smoothing can also be done after you select the areas, from the menu: Selections -> Modify -> Smooth, before flood-filling them).
Ps. Hi JoeB, thanks for clarifying hector's question.
