A fine-tuned
hfl/chinese-macbert-base model for classifying Chinese vulnerability descriptions from the
China National Vulnerability Database (CNVD) into three severity levels:
Low,
Medium, and
High.
The dataset is imbalanced: Low (9.0%), Medium (55.2%), High (35.8%).
1from transformers import pipeline
2
3classifier = pipeline(
4 "text-classification",
5 model="CIRCL/vulnerability-severity-classification-chinese-macbert-base"
6)
7
8description = "TOTOLINK A3600R存在缓冲区溢出漏洞，攻击者可利用该漏洞在系统上执行任意代码或者导致拒绝服务。"
9result = classifier(description)
10print(result)
-
Low severity recall: the Low class has the lowest recall. Approximately 60% of Low-severity entries are misclassified, mostly as Medium. This reflects the vocabulary overlap between Low and Medium descriptions in CNVD data. Class-weighted loss and focal loss were tested but all degraded Medium recall disproportionately without a net benefit.
-
Keyword dependency: the model biases toward a vulnerability type's typical severity. For example, buffer overflow descriptions are predicted as High regardless of the actual assigned severity. On entries where the actual severity deviates from the type's typical severity, accuracy drops from ~89% to ~55%.
-
Negation blindness: the model does not understand negation. Descriptions like "does NOT allow remote code execution" can still produce high-confidence High severity predictions.
-
CVE overlap: 81% of CNVD entries have a corresponding CVE. The model primarily adds value for the ~19% of CNVD-only entries (concentrated in Chinese domestic software) where no CVE severity assessment exists.
These limitations were identified through independent analysis in
VulnTrain#19.
