Glad we are on the same page. Interestingly, I came away with a very different interpretation based on what we've seen so far.
From the flashbacks with the Separatist droids, we can assume that he was a child during the Clone Wars. The fact that he says he's been wearing his helmet since he was a child means that not only has he been raised as Mandalorian since he was found, but that bringing up orphans as Mandalorians predates the "Great Purge" and isn't in response to it. The big Mandalorian in episode 3 calls him a coward for working with the Empire. Note that he doesn't call him out as not being a Mandalorian.
In the first episode, when his pauldron is made, the armorer says that the extra beskar would sponsor many foundlings to which he says he was once a foundling and she responds that she knows. I took that to be more of him agreeing with the extra going to "charity", sort of saying, I was there too, so of course I should pay if forward...kinda like tithing.
As for the crest/signet, it appears to be related to a rite of passage, hence why it has to be revealed, and why the armorer was going to use the mudhorn for him to reflect his victory.
As for everyone asking if he's a Mandalorian, I think it's similar to how it was in Rebels with the Jedi. The Mandalorians were legendary warriors. Since the purge, it's very rare to see them, so when you see a badass in fancy armor, your first question might be are you a Mandalorian.
Just my take on it.