How to manage a small custom model with limited data at deployment? Delexicalization to rescue.

I was trying to finetune a content moderation model on custom categories (like avoiding comments on politics, war, governments types, legal advice, some company specific policies). I wanted model to not comment on say things like "Shall America support UK in aiding over [org] blah.". I could simply prompt an LLM to avoid such content but we want a swiss cheese model for such critical safety requirements (reason we use content moderation classifier at first place). I picked up an existing moderation model with fixed categories and extended it with my custom classes. Now, I had very limited data to fine-tune this model for my company's custom categories (say not comment on external organizations, countries, government strategies/things to do). So I used some existing small datasets on kaggle wherever available and generated synthetic data using LLM and finetuned my moderation model. Now I tested this small moderation model on out of domain questions - spelling errors, different names of original countries, company specific or regional terms, etc (say KSA for Saudi, MENA for gulf). And it was missing such different names and new orgs (yes model's inherent NER was bad). What shall I ha