dsrestrepo/BERT_Lab_Values_10B_lab_id_no_repetition

Model Details

Model Name: NumericBERT

Model Type: Transformer

Architecture: BERT

Training Method: Masked Language Modeling (MLM)

Training Data: MIMIC IV Lab values data

Training Hyperparameters:

• Optimizer: AdamW
• Learning Rate: 5e-5
• Masking Rate: 20%
• Tokenization: Custom numeric-to-text mapping using the TextEncoder class

Text Encoding Process

Overview: Non-negative integers are converted into uppercase letter-based representations, allowing numerical values to be expressed as sequences of letters.

Normalization and Binning:

• Method: Log normalization and splitting into 10 bins.
• Representation: Each bin is represented by a letter (A-J).

Token Construction:

• Format: <<lab_id_token>><<lab_value_bin>>
• Example: For a lab value of type 'Bic' with a normalized value in bin 'C', the token might be BicC.
• Columns Used: 'Bic', 'Crt', 'Pot', 'Sod', 'Ure', 'Hgb', 'Plt', 'Wbc'.

Training Data Preprocessing

• Column Selection: Numerical values from selected lab values.
• Text Encoding: Numeric values are encoded into text using the process described above.
• Masking: 20% of the data is randomly masked during training.

Model Output

• Description: Outputs predictions for masked values during training.
• Format: Contains the encoded text representing the predicted lab values.

Limitations and Considerations

• Numeric Data Representation: The custom text representation may have limitations in capturing the intricacies of the original numeric data.
• Training Data Source: Performance may be influenced by the characteristics and biases inherent in the MIMIC IV dataset.
• Generalizability: The model's effectiveness outside the context of the training dataset is not guaranteed.

Contact Information

• Email: davidres@mit.edu
• Name: David Restrepo
• Affiliation: MIT Critical Data - MIT