The Best Practice Guides No. 4, which covers electrical installation condition reports suggest a classification of C3, improvement recommended... if you only had C3s on a report it would be classed as satisfactory.
https://www.----------------------------/media/1200/best-practice-guide-4-issue-4.pdf
Page 16 I thin if you'd like to look it up for yourself.
Yes, it would be a safety improvement, but it is not required. And if someone has told you it's not compliant with building regulations... they are incorrect. It is not compliant with the current edition of the wiring regulations (BS 7671) which is the standards document which defines the standards to which all electrical work in the UK should be carried out.
If we installed a plastic consumer unit in a house today, it would be non-compliant, but as it was installed some time ago, yes it's non-compliant but it's not necessary to remove it if the only reason is 'it's plastic'.
Hope that helps.