We have an INotificationProvider which does this.

var settings = orchardServices.WorkContext.CurrentSite.As<CustomSettingsPart>();


if (!string.IsNullOrEmpty(settings.Property){
yield return new NotifyEntry
                {
                    Message = T("Property needs setting"),
                    Type = NotifyType.Error
                };
}

The problem is, when updating the CustomSettingsPart through admin, because of the ordering (I'm assuming) if I update another *SettingsPart it'll never update because it'll fail the INotificationProvider check.