First thing we need to talk about is the length of a wave. A high energy beam of light has a short wavelength because all the energy is compacted into a small space, while a low energy beam of light is spread out and loose. Colours like blue, green and UV have shorter wavelengths (more energy) while red and IR have low energy. (Note to the reader: my dad works on lasers, so he knows all this, I just put that bit in for everyone else)
But how come IR feels warmer if it has less energy than visible light?
The answer is actually part Physics and part Biology. Different sizes of wave will interact with atoms in different ways. Radio and Microwaves, for instance will impact an atom or molecule and make it spin. Infra-red, packing a bit more punch, will make the bonds stretch and twist (more of a change to twist something than to simply rotate it). Visible and UV light are the same size as the electron shells, so they make the electrons on the outside of the atom/molecule dance around. X-rays will make the core electrons right near the centre of the atom dance, and a gamma ray is so compacted in energy that it will actually make an atom's nucleus vibrate.
So different amounts of beam energy have different effects on what it's hitting, roughly corresponding to size i.e. large waves will interact with a whole molecule, tiny waves will interact with the individual nucleus. Infra-red (and microwaves) happen to be the ones which bend, twist and stretch molecules.
In human skin we've got tiny sensors called "thermoreceptors" which pick up on the vibration energy of the surrounding cells. If I.R. waves are hitting human skin, the molecules in skin will twist, vibrate and stretch, triggering the thermoreceptors which sends a signal to the brain as "hot". Visible light, while packing more energy, just causes the electrons in the skin-molecules to dance around, a more impressive feat, but something the thermoreceptors don't pick up on.
So while IR is much lower in energy than visible light, our skin doesn't have sensors to pick up on electron dancing, only molecule twisting. So our skin-linked nerves detect IR but not visible light. By contrast, the human eyeball has chemicals which do pick up on visible light making the electrons vibrate, but not IR, that's why your eyeballs never feel hot!