Right click the one you want to change > Properties
Then there should be a "Change Icon" button which, if you select "Browse" you can select any thing you see (Except Folder) for an icon (i.e. If you have a game with an icon, say, Half-Life, you just select "hl.exe" and it will have the Half Life icon).
Most image editors have an "Export to icon" or something similar. Get your picture and use that function to save it as an .ico, then go to Right click>Properties>Change Icon, and then browse for your file, then hit okay.
The GIMP can save things in the .ico format, but making one is extremely complicated. You have to do it layer by layer. For example, layer 1 is the main image, layer 2 is the image while you are dragging it, layer 3 is another action, etc. Very complicated.