could be possible that the leakage (if there is leakage) could be from another component in the PC case,
No. If the case reads live w.r.t. earth, it's not solidly connected to earth. By default, the PSU will leak enough to read on a meter, directly through its case or via anything connected to the PSU which will have an earthed DC common rail.
Voltsticks have their uses but can give confusing indications unless you know exactly how they work. Any voltage capacitively coupled to the tip is compared with the voltage on the touch pad. You're big and fairly conductive, so you capacitively couple the touch pad to your surroundings quite well. These are assumed to be earthy, so it's comparing the voltage near the tip with earth. If you are not close to earth potential, the voltstick has no reference and it may be hard to interpret the indication without a reasonable knowledge of physics and electronics.
For example, suppose you stand near a fridge that is not earthed. Normal leakage in the fridge may raise the casing to say 100V, but the current is so small you probably won't get a tingle, so you're not aware it's not earthed. The floating fridge couples capacitively to you and if the capacitance from fridge to you is half that from you to earth, you'll end up at 33V. Again, very high impedance, tiny currents, you won't know anything about it. But when you point your voltstick at an appliance that is earthed, it sees the 33V difference between you and that, and lights up. It has no way of knowing which end of itself is earthy.
So I think you should ignore what the voltstick says at this stage although it might be trying to tell you something about your surroundings. Many regulars on this forum detest voltsticks because of this ambiguity, and the tendency for non-electricians to use them for safety critical purposes which should never be done. Search for your leakage using calibrated instruments!