Oh man you are taking me back a decade plus now to my masters in materials engineering.
IIRC any work you do on metals introduces faults in the microstructure such as cracks, dislocations, etc. The more work done the more of these faults there are. These interact in ways which make the metal harder, think of it like the defects are interacting together preventing the previously ductile metal from kind of flowing (very simplified explanation due to the fact I have not used that information for over 10 years). The work can be from extruding, forming or anything that exerts a force on the metal I think.
Then annealing is where you heat the metal up to a temperature below its melting or softening temperature. This temperature is high enough for the defects to effectively move and disappear (IIRC) which allows the metal to be more ductile and effectively softer.
I believe that other types of materials would operate similarly. Anything crystalline (especially larger crystals) too can have these defects and the same hardening/softening interaction with working or annealing.
I hope my rather pathetic attempt at remembering a rather basic part of education helps. If you want to know more then I can recommend W. D. Callister's book on materials science is a good general book aimed at undergraduates but easy to follow. Perhaps not the best book on the subject but it covers pretty much most types of materials and materials topics in a straightforward way. Also used to be cheap hence the undergraduate target audience. If it was aimed at professional materials audience it would probably be about 5+ times more expensive!!!!