I've actually seen it where this guy put an "M3" badge on a 325is. You could totally tell it wasn't an M3. Thought it was funny.
My understanding of some of those who have debadged cars was that others wouldn't know what they really had. "Sleepers" if you will....
Personally, I would leave the "M" badge on there. I would be proud to be the owner of a car that has a badge that screams a lot of passion and heritage