Inquiry About PEFT Compatibility Issue
Hello,
I wanted to bring to your attention an issue I’ve encountered when using the Geneformer code with the newest version of PEFT (v0.14, released in December). This version apparently introduces a stricter validation for the task_type parameter specified in LoraConfig.
The Geneformer code snippet setting this parameter in perturber_utils.py:
peft_config = LoraConfig(
lora_alpha=128,
lora_dropout=0.1,
r=64,
bias="none",
task_type="TokenClassification",
)
The new PEFT version checks whether the provided task_type is among the accepted TaskType values, and it seems that "TokenClassification" is not recognized as valid. This validation step was not enforced in previous versions, and the code ran without issues.
To address this, I attempted two approaches:
- Changing task_type to "TOKEN_CLS" (which I presumed to be the correct value). However, this resulted in issues during the model evaluation step. Specifically, the classifier_predict function fails as predict_logits contains NaN values.
- Setting task_type to None, which allowed the code to run to completion.
Could you please clarify the correct value for task_type? Additionally, do you have any insight into why using "TOKEN_CLS" causes the evaluation step to fail?
Thank you very much for your time and assistance
Thank you for your question! If you provide the accepted TaskType values and what kind of classification you are trying to do, we may be able to advise which one should be chosen. TokenClassification is classifying genes, whereas TOKEN_CLS I assume is classifying the sequence using the CLS token, which would in this case be classifying cells.