eHyPRETo: Enhanced Hybrid Pre-Trained and Transfer Learning-based Contextual Relation Classification Model

Introduction: relation classification (RC) plays a crucial role in enhancing the understanding of intricate relationships, as it helps with many NLP (Natural Language Processing) applications. To identify contextual subtleties in different domains, one might make use of pre-trained models. Methods: to achieve successful relation classification, a recommended model called eHyPRETo, which is a hybrid pre-trained model, has to be used. The system comprises several components, including ELECTRA, RoBERTa, and Bi-LSTM. The integration of pre-trained models enabled the utilisation of Transfer Learning (TL) to acquire contextual information and complex patterns. Therefore, the amalgamation of pre-trained models has great importance. The major purpose of this related classification is to effectively handle irregular input and improve the overall efficiency of pre-trained models. The analysis of eHyPRETo involves the use of a carefully annotated biological dataset focused on Indian Mosquito Vector Biocontrol Agents. Results: the eHyPRETo model has remarkable stability and effectiveness in categorising, as evidenced by its continuously high accuracy of 98,73 % achieved during training and evaluation throughout several epochs. The eHyPRETo model&#39;s domain applicability was assessed. The obtained p-value of 0,06 indicates that the model is successful and adaptable across many domains. Conclusion: the suggested hybrid technique has great promise for practical applications such as medical diagnosis, financial fraud detection, climate change analysis, targeted marketing campaigns, and self-driving automobile navigation, among others. The eHyPRETo model has been developed in response to the challenges in RC, representing a significant advancement in the fields of linguistics and artificial intelligence.


INTRODUCTION
RC facilitates the extraction of correct question answers, structured data, and sentiment interpretation from unstructured text.Information retrieval, knowledge graph construction, sentiment analysis, named entity identification, question-answering systems, and other applications depend on accurate and efficient relation categorization.The integration of transfer learning-based techniques, the advent of hybrid approaches, and the ongoing development of pre-trained models have all contributed to significant advancements in relation categorization.The relation classification enhancements are considered to be the overall objective of those developments.
Actively evolving BERT models, (RoBERTa) (Y.Liu et al., 2019), ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) (Clark et al., 2020), and Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) were the pre-trained models for enhancing the NLP systems significantly, and also revolutionizing the classification efficiency.Additional data and optimisation approaches are utilized by RoBERTa throughout FT (Fine-Tuning) for the classification efficiency optimization.Then, ELECTRA provides alternative applicable synonyms in the particular tokens corpus in the input text, thereby enhancing the efficiency of the pre-trained procedure also maintaining the high accuracy.The other PTMs, considerably subsidized for the relation classification: DistilBERT ( (Yi & Hu, 2019), etc.To create a Personality Trait Classification model from text, the advantages of CNN and LSTM have been combined (Ahmad et al., 2021) for capturing contextual data.The hybrid pre-trained models based on TL make use of pre-trained and FT data for the enhancement of the accuracy of classification.Employing generalised language understanding, models like SpanBERT (Joshi et al., 2020) and ERNIE (Zhang et al., 2020) provides the notable accuracy in RC.
Upgraded Hybrid PRE-trained and TL-RC Model, associates the advantages of ELECTRA, RoBERTa, and Bi-LSTM was suggested in this research.The pre-training phase of ELECTRA's token replacement and the TL phase of RoBERTa's FT are united, and Bi-LSTM is utilized for enhancing the contextual information capture, sequential dependency management, and RC model's accuracy.

Related work
PTM took advantage through the substantial attention that classes have received through the course of time.Figure 2 illustrates the techniques used for relation classification over time.Machine learning algorithms such as Support Vector Machines (SVMs), Random Forest, Logistic Regression, Gradient Boosting, and Hidden Markov Models (HMMs), etc. were used for developing the relation classification model (Formisano et al., 2008).For continuous improvement, deep learning neural networks such as CNN, Recurrent Neural Networks (RNN), Transformerbased models, etc. were used to learn the features automatically (Itamar Arel, Derek C. Rose, 2010) (He et al., 2010).These methods, however, often struggled with capturing complex linguistic patterns and generalizing them across various domains.To address this limitation, the word embeddings such as Word2Vec, GloVe, FastText, and ELMo, etc, and syntactic features such as Dependency Parse Trees, Part-of-Speech (POS) tags were explored to enhance classification accuracy through semantic understanding and contextual information (Xu et al., 2015) (Agarwal et al., 2015).Furthermore, the utilization of word embeddings and syntactic features helps to make significant improvements in accuracy and the ability to handle diverse datasets.However, recent advancements in the field have witnessed a paradigm shift towards the utilization of large-scale transformer-based PTM, such as GPT, BERT, RoBERTa, ELECTRA, MarianMT and so on (S.Yu et al., 2019) (Du et al., 2022).Their outstanding capacity to use contextual information made them the preferred choices for RC.
An evident focus on multitask learning and domain adaptation named innovative TL approaches are presented in this review, and it is considered as critical to enhance the model performance and extending RC abilities.Also, novel trainings indicate the significance.Including pretrained models in technical NN layers because they give greater outcomes and leads to the development of more consistent and accurate hybrid models for diverse applications (Aydoğan & Karci, 2020).Many transfer learning hybrid models, including BERT-BiLSTM, BERT-BiLSTM-CRF, BERT-BiLSTM-LSTM, BERT-BiGRU, BERT-BiGRU-CRF, RoBERTa-BiLSTM-CRF, etc., have been developed for a number of applications and have improved relation classification performance (Q.Yu et al., 2021) (Qin et al., 2021).These models combine pre-trained models and deep neural networks.The hybrid pre-trained models for text categorization that researchers built using transfer learning are listed in table 1.In order to handle noisy words and increase the network's resilience, Tianyi et al. employed transfer learning to initialise the network with prior information gained via entity categorization (T.Liu et al., 2018).
The eHyPRETo, the integration of ELECTRA, RoBERTa, and Bi-LSTM is proposed to address the limitation of existing hybrid models in capturing fine-grained contextual information and complex sequential dependencies, especially in the context and specialized relation classification tasks.Section 3 explores the proposed model eHyPRETo, detailing its working architecture.Section 4 describes the results and discussion, and Section 5 provides the conclusion.The proposed eHyPRETo suggests a novel approach that synergizes their unique capabilities, resulting in understanding and classifying relationships in text across various contexts and improving accuracy.The eHyPRETo consists of various components such as ontology embedded dataset for training and testing, data encoding for embedded vector transformation, eHyPRETo architecture for relation classification model building, Training and Fine-tuning to optimize parameters, Classification for categorizing relations, and Evaluation for the model's performance assessment.Figure 3 illustrates the eHyPRETo model components.

Ontology Embedded Dataset
The input dataset comprises 632 manually annotated sentences related to mosquito vectors and biocontrol agents which are collected from research articles in PubMed database.During annotation, six distinct relation labels were assigned to capture different relations between entities.The domain-specific concepts such as mosquito name, and biocontrol agents are embedded using MVAOnt, Mosquito Vector Association Ontology1.The entity pairs <e1></e1> and <e2></e2> are used for incorporating concepts.The dataset is partitioned into subsets such as for training, validation, and testing to ensure a balanced distribution of relation labels for eHyPRETo evaluation.

Data Encoding
An essential step in getting input data ready for the eHyPRETo model is data encoding.It entails converting input phrases that are ontology embedded into numerical embedded vectors useful for model processing.During this procedure, the model's embeddings are used to translate tokenized sentences where each token represents a word or subword to numerical representations.From the sentence "The mosquito is a vector," which is encoded into ["The", "mosquito", "is", "a", and "vector"), a vector with numerical values, such as [12,345,45,7, 89], is captured.Each number is an index of tokens.The attention masks make it easier to focus on significant concepts and ignore unnecessary ones.The encoded vectors and attention mask together make up the model's input, permitting for managing information forRC effectively.
eHyPRETo Architecture eHyPRETo model integrates the advantages of RoBERTa, BiLSTM, and ELECTRA for incrementing RC performance and contextual knowledge.While RoBERTa supports in comprehending the general semantics of phrases, ELECTRA assists in understanding the way of words that fit together.This special combination makes use of RoBERTa embeddings to capture the wider context, ELECTRA embeddings for FT, and Bi-LSTM for capturing the bidirectional context.By integrating ELECTRA, RoBERTa, and Bi-LSTM is a crucial enhancement that greatly progresses the model's contextual comprehension of ideas and RC.

JG, et al
In order to capture bidirectional context, the eHyPRETo Bi-LSTM layer includes an input layer that receives tokenized and contextual embeddings from ELECTRA and RoBERTa.Setting input and output dimensions, configuring hidden units, and taking advantage of the LSTM's bidirectional nature to process input sequences in a bidirectional manner are all part of the layered design.Long-range dependencies are captured, and contextual awareness is improved.More RC accuracy is possible with the eHyPRETo model thanks to dynamic contextual awareness.The combined contextual embedding vectors of ELECTRA, RoBERTa, and Bi-LSTM are the outcomes of the eHyPRETo.This technique improves knowledge of the way of ideas are related to one another, which in turn progresses the model's RC ability.The comprehensive eHyPRETo requirements for RC tasks are presented in table 2.

Training and Fine-tuning
In the Training and Fine-tuning, eHyPRETo iteratively refines parameters related to the weights and biases of its neural network layers for optimization.This optimization involves minimizing the cross-entropy loss function by Adam optimizer enhancing the model's capacity to learn and generalize from training data.Consequently this process enhances complex relationship capturing and ensuring better performance in relation classification.

Classification
The classification layer processes the contextual embeddings obtained from the eHyPRETo model.It takes the encoded tokenized sentences, which have been enriched with ELECTRA, RoBERTa, and Bi-LSTM, and uses rich contextual information to classify the type of relationship between the concepts in the input sentence more accurately.The classification layer outputs a set of probabilities, indicating model's confidence in each possible relation label, thereby facilitating accurate relation classification within the given text.The high probability is the appropriate classified relation.For example, if the probability values generated for "<e1>Pinus kesiya</ e1>leaf essential oil is against <e2>Aedes aegypti</e2>" for six relations is [0,730, 0,135, 0,025, 0,020, 0,045, 0,45], then the relation label corresponding to the highest probability is the classified output.

Evaluation
In the evaluation phase, the eHyPRETo model's predictions are compared with the test dataset.It assesses the performance by statistical measuers such as accuracy, precision, recall, F1-score, and Cohen's Kappa.Evaluation shows how well the model is performing in terms of relation classification.After evaluation, the numerical predictions or probabilities generated by the eHyPRETo model for the raw data are transformed into human-readable and semantically meaningful relation labels.This step connects model's output with real-world understanding by mapping numerical representations to linguistic expressions, such as "is_a," "biocontrol", "cause_effect", etc.The resulting structured output provides a clear representation of the relationships between concepts within the input text to facilitate tasks like knowledge extraction, information retrieval, and decision support based on the context.
The next section delivers the comprehensive results and discussion of eHyPRETo relation classification performance in mosquito vectors and biocontrol agents context.

RESULTS AND DISCUSSION
The eHyPRETo performs remarkably in relation classification by enhancing contextual understanding and capturing complex relationships.The eHyPRETo model is developed using Python.The eHyPRETo model is assessed using a manually annotated mosquito vectors and biocontrol agents dataset, categorized into six distinct relation labels.This model exhibits enhanced performance in effectively classifying and distinguishing between these relation labels, particularly in the context of the unbalanced dataset.Table 3 shows the eHyPRETo performance over 10 epochs.Even after 10 epochs, there has been no noticeable improvement in accuracy.Figure 4 illustrates the graph depicting epoch and accuracy.Further, the eHyPRETo is assessed with traditional machine learning models and PTM.Table 4 shows the eHyPRETocomparision with BERT and its variants.eHyPRETo outperforms all the models, achieve high values for each metrics.Figure 5 illustrates the corresponding comparision chart.The outstanding performance of eHyPRETo suggests the model effectiveness in accurately classifying complex relationships in the unbalanced dataset.After assessing and refining the model's performance, the final classified relation in terms of probabilities is mapped into human-readable relation labels for understanding as shown in figure 6.

CONCLUSION
In conclusion, relation classification in the field of mosquito vector biocontrol agents was successfully carried out using the eHyPRETo, a unique hybrid pre-trained model combining ELECTRA, RoBERTa, and Bi-LSTM.Through precise classification of interactions among different entities in this setting, the model shows promise for greatly improving our knowledge of vector-borne illnesses and biocontrol agents.Using a variety of metrics, the eHyPRETo model demonstrates the significance of relation categorization.The significance test also demonstrates the domain applicability of the model.The created model can be useful in a number of fields, including agriculture, healthcare, the military, the air force, and more.Overall, the combination of Salud, Ciencia y Tecnología -Serie de Conferencias.2024; 3:758 8 Sanh et al., 2019), A Lite BERT (ALBERT) (Lan et al., 2020), eXtreme Learning with Transformers (XLNet) (Yang et al., 2019), Enhanced Representation through kNowledge IntEgration (ERNIE) (Zhang et al., 2020), and TAPAS (Tabular Pre-trained Language Model (Herzig et al., 2020).For the relation classification, numerous pre-trained models are employed and it is presented in figure 1.Through the utilization of large-scale datasets, rich contextual embeddings are obtained for attaining textual semantic meanings.These pre-trained models are trained on several datasets can be obtained by FT such models.Researchers have modified them for the relation classification tasks thereby attaining superior outcomes.

Figure 1 .
Figure 1.Pre-trained Models Combination of several NN (Neural Networks) models by the hybrid models for enhancing the RC accuracy.Researches have suggested numerous hybrid models as it integrates Convolutional Neural Networks (CNN) (Hui et al., 2020), Long Short-Term Memory (LSTM), Bidirectional Long Short-Term Memory (Bi-LSTM) (Harnoune

Figure 4 .
Figure 4. Epoch Vs Accuracy The eHyPREto model's performance across ten epochs, consistently demonstrates high accuracy and precision, indicating the model's effectiveness in making accurate predictions.The accuracy increases from 92,86 % in the first epoch to 98,73 % in subsequent epochs, suggesting the model's learning and adaptation over time.Precision, Recall, and F1score also maintain high values, with minimal variations across epochs, emphasizing the model's strength and stability.Cohen's Kappa score demonstrates strong agreement between predicted and actual values, with values ranging from 0,8507 to 0,9761 across the ten epochs.The constant performance across each epoch is a positive reflection of the eHyPRETo model's stability.Further, the eHyPRETo is assessed with traditional machine learning models and PTM.Table4shows the eHyPRETocomparision with BERT and its variants.eHyPRETo outperforms all the models, achieve high values for each metrics.Figure5illustrates the corresponding comparision chart.The outstanding performance of eHyPRETo suggests the model effectiveness in accurately classifying complex relationships in the unbalanced dataset.After assessing and refining the model's performance, the final classified relation in terms of probabilities is mapped into human-readable relation labels for understanding as shown in figure6.

Figure 6 .
Figure 6.Illustration of Model's Classified Relation Finally, the eHyPRETo's performance is examined across different domains by Mosquito Vector Biocontrol Agents as the source domain and SemEval 2010 Task 8 dataset as the target domain.The combined data is used for training, and the target data is used for testing and evaluating model's performance.The p-value (0,06) that is obtained by t-test indicates the model's ability to adapt in diverse domains, exposing its effectiveness beyond specific contexts.Figure 7 visualizes the domain adaptability performance of eHyPRETo model in terms of accuracy.
Figure 7 visualizes the domain adaptability performance of eHyPRETo model in terms of accuracy.

Figure 7 .
Figure 7. eHyPRETo's Domain Adaptability Visualization The eHyPRETo model's performance highlights its substantial potential in decisionmaking, knowledge extraction, question-answering systems, semantic search engines, and more applications related to mosquito vectors and biocontrol agents.The model has demonstrated effectiveness in the specific domain of mosquito vectors and biocontrol agents.