If your pain is accompanied by swelling, use ice. Ice reduces swelling by slowing the rush of blood to the aggravated joint. (People with circulation problems should talk to their doctor before using ice on a regular basis.)
Heat, on the other hand, causes expansion, which means that it's probably counterproductive for swelling. But heat can loosen tissues and relax stiff joints.
In the morning or before exercise, heat can warm up the muscles around your knee. At the end of the day, if your knee swells, try an ice pack. That said, osteoarthritis pain is tricky and often a "whatever works" approach is needed.
You may alternate between heat and ice to find what works best. You can ice your knee with store-bought cooling gel packs, a resealable plastic bag filled with ice, or a bag of frozen vegetables, but ice the knee for no longer than 20 minutes at a time.
Standard protocol for injuries calls for 20 consecutive minutes of ice every two hours over 72 hours after an injury; however, the Journal of Sports Medicine reports that 10 minutes on/10 minutes off/10 minutes on, every two hours, also works.
There is no protocol for heat, which is typically applied with a heating pad or a hot, wet towel, but limit yourself to 15 consecutive minutes.
Source: Prepared by the Editors of The Johns Hopkins Medical Letter: Health After 50