Like everyone here, I'm missing Lucien's perceptive insights on this and other forums.
I do hope his vision of the Museum of Electricity and Electronic Technology will come to fruition in some form or other.
ChrizK - I'll contribute to the limited extent I can, making some assumptions as to what is in your switch box!
I can see it uses small printed circuits as "bus bars" across the input contacts and also across some of the rocker switch pins.
One can see the tracks on the nearer pcb but not on the rear connectors, so I'm making some possibly incorrect assumptions in drawing the scrappy circuit below (with a leaking biro!)
I don't think it is as sophisticated as Lucien initially suggested, but his closing comments are pertinent. I can't really add anything about the behaviour you experience, save to say the following:
By cross-linking the left and right speaker outputs you seem to have created the situation where the 'negative' right and left channel speaker outputs are permanently connected together. Depending on the design of your amp, this may create an immediate problem with its performance and overload protection. If your amplifier -ve speaker output is ground anyway, then it's not a problem.
And then, if you operate both switches simultaneously (which you have generally tried to avoid!) you will connect the amplifier left and right speaker outputs together, which is probably not a good thing. For a stereo signal it will be feeding one amplifier output back into the other channel, so 'fighting' with each other.
Assuming you continue to avoid the above, I think the switch you have ought to do what you want, but I fear Lucien's comments about the quality of the switch contacts is perhaps the answer.
My suggestion of what's in the box:
In my younger day you used to be able to buy, cheaply for a shilling or so, an ex-military Cutler-Hammer 4-pole, double throw, (centre off if you want) toggle switch, with which you could wire up the necessary configuration, without any danger of connecting outputs together etc. I was going to suggest that would be a better approach, but I see the prices have gone up !!
If your amplifier has the -ve speaker output connected to ground you can use a double pole double throw switch rather than the above!