Both are excellent calibers and you'd enjoy both IMO.
I like the Hornet because it really packs a punch for such a tiny cartridge. But if you dont handload, they are pretty pricey for such a small bullet.
I have been shooting 36gr Barnes Varmint Grenades out of my Hornet, and they CREAM jack rabbits! But they dont feed terribly well, so I tend to shoot the 32gr V-Max out of mine most often powered by 11gr Lil Gun and a small pistol primer.
My only complaint about the Hornet, is that it doesnt produce the accuracy that I would like past 150 yards. That is where my .204 really shines. I'm not saying the Hornet wont shoot past that distance, I just cant seem to find a load that produces the results I would personally like to see.
If you dont handload, I think the .204 might be the better option due to ammo availability and also more options with a similar price point relative to the Hornet. But you can also find HSM loads for substantially cheaper than other factory loads (HSM also does Hornet loads, but they arent as easy to come by in my neck of the woods).
The .204 is really a great caliber. A varmint sniper's dream if you ask me