Accuracy is the holy grail of any machine learning model. It’s the metric that tells you how well your model is performing and how close it is to making accurate predictions. But what exactly is model accuracy and how can you improve it? In this comprehensive guide, we’ll explore the ins and outs of model accuracy, what it means, and how to improve it. We’ll cover topics such as the different types of accuracy, how to calculate it, and how to interpret the results. You’ll also learn about the different techniques and strategies that can help you improve your model’s accuracy and achieve better results. So, let’s dive in and get started on your journey to understanding and improving model accuracy!
What is Model Accuracy?
Definition and Importance
Model accuracy refers to the degree of correctness or accuracy of a model’s predictions or outputs. It is a measure of how well a model performs on a specific task, such as classification, regression, or clustering. In simpler terms, it indicates how close a model’s predictions are to the actual values or labels.
Accuracy is a crucial aspect of model development because it determines the usefulness and reliability of a model in real-world applications. A model with high accuracy is more likely to provide accurate predictions, while a model with low accuracy may lead to incorrect decisions, wasted resources, and potential harm. Therefore, it is essential to understand and improve model accuracy to ensure that the model is reliable and effective in its intended use.
How to Measure Model Accuracy
Model accuracy is a critical metric in the field of machine learning that measures how well a model can make predictions. In order to evaluate the performance of a model, it is important to measure its accuracy. There are several ways to measure model accuracy, including:
- Accuracy Score: This is the most commonly used metric to measure model accuracy. It is the ratio of the number of correct predictions made by the model to the total number of predictions made. It is calculated as follows:
accuracy = (number of correct predictions) / (total number of predictions)
- Precision Score: This metric measures the proportion of true positive predictions out of all positive predictions made by the model. It is a useful metric when the cost of false positives is high. It is calculated as follows:
precision = (number of true positive predictions) / (number of positive predictions) - Recall Score: This metric measures the proportion of true positive predictions out of all actual positive instances in the dataset. It is a useful metric when the cost of false negatives is high. It is calculated as follows:
recall = (number of true positive predictions) / (number of actual positive instances) - F1 Score: This is the harmonic mean of precision and recall scores. It is a useful metric when both precision and recall are important. It is calculated as follows:
f1 = 2 * (precision * recall) / (precision + recall) - Confusion Matrix: This is a table that summarizes the performance of a classification model by separating true positive, true negative, false positive, and false negative predictions. It provides a comprehensive view of the model’s performance and can be used to calculate other accuracy metrics such as accuracy score, precision score, recall score, and F1 score.
It is important to note that the choice of accuracy metric depends on the specific problem being solved and the cost of different types of errors. In some cases, accuracy may not be the best metric to use, and other metrics such as precision or recall may be more appropriate. Additionally, it is important to keep in mind that accuracy alone does not provide a complete picture of a model’s performance, and other metrics such as training time, computational resources, and interpretability should also be considered.
Factors Affecting Model Accuracy
Data Quality
The accuracy of a model is heavily dependent on the quality of the data it is trained on. Poor quality data can lead to biased, inaccurate or overfitting models. In this section, we will explore the different aspects of data quality that can impact model accuracy.
Missing or Incomplete Data
Missing or incomplete data can be a major issue for machine learning models. It can lead to biased models, or even cause the model to fail entirely. There are several techniques that can be used to handle missing data, such as imputation, where the missing values are estimated based on the other values in the dataset, or deletion, where the rows with missing values are removed from the dataset.
Noise and Outliers
Noise and outliers can also have a significant impact on model accuracy. Noise refers to random variations in the data that do not have any meaningful relationship with the target variable. Outliers, on the other hand, are extreme values that are significantly different from the other values in the dataset. Both noise and outliers can lead to overfitting, where the model becomes too complex and starts to fit the noise in the data instead of the underlying patterns. There are several techniques that can be used to handle noise and outliers, such as normalization, where the data is scaled to a common range, or robust regression, where the model is trained to be more resistant to outliers.
Imbalanced Data
Imbalanced data is a situation where the dataset contains significantly more examples of one class than another. For example, in a disease diagnosis dataset, there may be many more examples of healthy patients than patients with the disease. This can lead to biased models, where the model is too skewed towards the majority class. There are several techniques that can be used to handle imbalanced data, such as oversampling the minority class, where additional examples of the minority class are generated, or undersampling the majority class, where examples of the majority class are removed.
Data Curation
Finally, data curation is an important aspect of data quality. Data curation involves cleaning, transforming, and preprocessing the data to ensure that it is in a suitable format for the machine learning model. This can include tasks such as removing irrelevant features, handling categorical variables, and normalizing the data. Good data curation can greatly improve model accuracy by ensuring that the model is trained on high-quality data that is easy to interpret and understand.
Model Complexity
The Influence of Model Complexity on Accuracy
In the context of machine learning, model complexity refers to the intricacy of the algorithm or model used to make predictions. This can include the number of layers in a neural network, the size of the model, or the complexity of the mathematical equations employed. The relationship between model complexity and accuracy is multifaceted and depends on several factors.
The Trade-off between Model Complexity and Overfitting
One of the primary considerations when assessing model complexity is the risk of overfitting. Overfitting occurs when a model is too complex and fits the training data too closely, leading to poor generalization performance on unseen data. A simple model may not capture the nuances of the data, while an overly complex model may memorize noise in the training data, resulting in poor performance on new data.
Selecting the Optimal Model Complexity
Determining the optimal level of model complexity requires careful evaluation of the data and the problem at hand. It often involves finding a balance between capturing the underlying patterns in the data and avoiding overfitting.
Methods for Reducing Model Complexity
There are several techniques for reducing model complexity while maintaining or even improving accuracy. These include:
- Principled methods: Regularization techniques, such as L1 and L2 regularization, or early stopping, can help prevent overfitting by adding a penalty term to the loss function or by stopping the training process when the performance on a validation set stops improving.
- Data-driven methods: Dimensionality reduction techniques, such as principal component analysis (PCA) or independent component analysis (ICA), can be used to reduce the number of features without losing too much information.
- Model selection: Cross-validation techniques, such as k-fold cross-validation, can be used to compare the performance of different models with varying levels of complexity and select the best-performing model.
Conclusion
In conclusion, model complexity plays a crucial role in determining the accuracy of a machine learning model. While a more complex model may provide better performance on the training data, it can also lead to overfitting and poor generalization. Careful consideration and evaluation of the data, as well as the use of appropriate techniques to reduce complexity, are essential for achieving high accuracy and robust performance in real-world applications.
Overfitting and Underfitting
Overfitting
Overfitting occurs when a model becomes too complex and learns the noise in the training data, resulting in poor performance on new, unseen data.
- Causes:
- High model complexity
- Insufficient training data
- Inappropriate model architecture
- Symptoms:
- High accuracy on training data
- Low accuracy on validation or test data
- Over-optimization of model parameters
- Consequences:
- Model cannot generalize well to new data
- Over-optimized model may not be scalable or interpretable
Underfitting
Underfitting occurs when a model is too simple and cannot capture the underlying patterns in the training data, resulting in poor performance on both training and new data.
+ Insufficient model complexity
+ Low accuracy on both training and new data
+ Large performance gap between training and new data
+ Model cannot capture underlying patterns in data
+ Model may not be able to learn from new data
+ Model may not be able to generalize well to new data
Strategies to Address Overfitting and Underfitting
- Regularization: Techniques such as L1 and L2 regularization, dropout, and early stopping can help prevent overfitting by adding a penalty term to the loss function or by removing or masking some of the model’s output during training.
- Model selection: Selecting an appropriate model architecture that is neither too simple nor too complex can help prevent underfitting.
- Data augmentation: Increasing the size of the training data can help prevent underfitting by providing the model with more examples to learn from.
- Cross-validation: Using techniques such as k-fold cross-validation can help assess the model’s performance on new data and prevent overfitting by ensuring that the model is not over-optimized on the training data.
Feature Engineering
Feature Engineering:
- Definition:
Feature engineering is the process of selecting, transforming, and creating new features from raw data to improve the performance of machine learning models. - Importance:
It plays a crucial role in improving model accuracy by transforming raw data into a format that can be easily understood by machine learning algorithms. - Techniques:
Common techniques in feature engineering include:- Data cleaning: This involves removing or correcting errors, missing values, and inconsistencies in the data.
- Feature scaling: This technique rescales the data to a standard range, such as 0 to 1 or -1 to 1, to ensure that all features are on the same scale.
- Feature selection: This involves selecting the most relevant features for the model based on statistical or domain knowledge.
- Feature creation: This involves creating new features from existing ones, such as combining two features or deriving a new feature from existing ones.
- Challenges:
One of the main challenges in feature engineering is overfitting, where the model becomes too complex and performs well on the training data but poorly on new data. - Best practices:
- Start with a small set of features and gradually add more as needed.
- Use domain knowledge to guide the feature selection process.
- Use techniques such as cross-validation and regularization to prevent overfitting.
- Continuously evaluate the performance of the model on new data.
Strategies for Improving Model Accuracy
Preprocessing Techniques
Effective preprocessing techniques play a crucial role in enhancing model accuracy. These techniques aim to transform raw data into a format that is more suitable for analysis and modeling. The following are some key preprocessing techniques that can be employed to improve model accuracy:
- Data Cleaning: This involves identifying and handling missing values, outliers, and inconsistencies in the data. Imputing missing values, winrowing outliers, and data normalization are some common techniques used in data cleaning.
- Feature Scaling: This technique ensures that all features are on the same scale, making it easier for the model to compare them. Common scaling techniques include standardization and normalization.
- Feature Selection: This involves selecting the most relevant features for the model, reducing noise and irrelevant information. Wrapper methods, filter methods, and embedded methods are popular techniques used in feature selection.
- Data Reduction: This technique reduces the dimensionality of the data, making it easier for the model to learn. Techniques such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) can be used for data reduction.
- Data Augmentation: This technique involves creating new data samples by applying transformations to the existing data. Data augmentation can be used to increase the size of the training dataset and improve the model’s generalization ability.
By employing these preprocessing techniques, data scientists can prepare their data for modeling, ultimately leading to improved model accuracy.
Feature Selection and Engineering
Introduction to Feature Selection and Engineering
In the realm of machine learning, the process of selecting and engineering features plays a pivotal role in enhancing the accuracy of a model. This is particularly crucial when dealing with large datasets that may contain a multitude of irrelevant or redundant features. The practice of feature selection involves identifying the most relevant features that contribute to the model’s performance, while feature engineering entails creating new features from existing ones to improve the model’s predictive capabilities.
Methods for Feature Selection
- Filter Methods:
- Correlation-based feature selection: Identifies features with the highest correlation with the target variable.
- Mutual information: Measures the dependency between features and the target variable.
- Relief: Selects features based on the attribute’s importance in reducing the error rate.
- Wrapper Methods:
- Forward selection: Incrementally adds the best feature to the model.
- Backward elimination: Removes the least important feature from the model.
- Recursive feature elimination: An iterative approach that removes the least important feature in each iteration.
- Embedded Methods:
- LASSO regularization: Regularizes the model by adding a penalty term to the objective function, thereby shrinking the coefficients of irrelevant features.
- Ridge regression: Penalizes large coefficients in the model, thus reducing the impact of irrelevant features.
Techniques for Feature Engineering
- Scaling and Normalization:
- Min-max scaling: Rescales the data to a fixed range, usually between 0 and 1.
- Z-score normalization: Centers the data around 0, with a standard deviation of 1.
- Aggregation and Disaggregation:
- Aggregation: Combining multiple features to create a single feature, such as computing the average or sum of values.
- Disaggregation: Breaking down a single feature into multiple sub-features, which can help reveal hidden patterns.
- Binning and Discretization:
- Binning: Grouping continuous data into discrete intervals or bins, which can simplify the data and reduce noise.
- Discretization: Converting continuous data into discrete values, which can improve the interpretability of the model.
- Categorical Encoding:
- One-hot encoding: Converting categorical data into binary features, where each category is represented by a 0 or 1.
- Label encoding: Encoding categorical data into numerical values, where each category is assigned a unique numerical value.
- Polynomial and Interaction Features:
- Polynomial features: Creating new features by raising the original features to a power higher than 1, which can capture non-linear relationships.
- Interaction features: Creating new features by multiplying pairs of original features, which can capture interaction effects between variables.
By employing feature selection and engineering techniques, practitioners can optimize their models and achieve higher accuracy on their tasks.
Hyperparameter Tuning
Hyperparameter tuning is a crucial process in improving the accuracy of machine learning models. Hyperparameters are the parameters that are set before the model is trained, and they determine how the model learns and makes predictions.
Importance of Hyperparameter Tuning
Hyperparameter tuning can significantly impact the performance of a machine learning model. By adjusting the hyperparameters, it is possible to optimize the model’s performance and improve its accuracy.
Common Hyperparameters
There are several hyperparameters that are commonly tuned in machine learning models, including:
- Learning rate
- Regularization strength
- Number of hidden layers and neurons in a neural network
- Dropout rate
- Batch size
Hyperparameter Tuning Techniques
There are several techniques for hyperparameter tuning, including:
- Grid Search
- Random Search
- Bayesian Optimization
- Evolutionary Algorithms
Grid Search
Grid Search is a brute-force approach to hyperparameter tuning. It involves specifying a range of values for each hyperparameter and then training the model with all possible combinations of hyperparameters. The combination that results in the best performance is then selected.
Random Search
Random Search is similar to Grid Search, but instead of trying all possible combinations, it selects a random subset of hyperparameters to train the model.
Bayesian Optimization
Bayesian Optimization is a more sophisticated approach to hyperparameter tuning. It uses a probabilistic model to predict the performance of the model based on the hyperparameters. The model is then optimized using an algorithm that selects the hyperparameters that are most likely to result in the best performance.
Evolutionary Algorithms
Evolutionary Algorithms, such as Genetic Algorithms and Particle Swarm Optimization, are inspired by natural evolution and are used to optimize complex systems. They involve creating a population of potential solutions and evolving them over generations to find the best solution.
Conclusion
Hyperparameter tuning is a critical process in improving the accuracy of machine learning models. By adjusting the hyperparameters, it is possible to optimize the model’s performance and improve its accuracy. There are several techniques for hyperparameter tuning, including Grid Search, Random Search, Bayesian Optimization, and Evolutionary Algorithms. The choice of technique depends on the complexity of the model and the size of the dataset.
Ensemble Methods
Ensemble methods are a popular approach for improving model accuracy in machine learning. Ensemble methods involve combining multiple weaker models to create a stronger, more accurate model. There are several different types of ensemble methods, including:
- Bagging (Bootstrap Aggregating): This method involves training multiple models on different subsets of the data and then combining their predictions to create a final prediction.
- Boosting: This method involves training multiple models sequentially, with each model focusing on the instances that were misclassified by the previous model. The final prediction is made by combining the predictions of all the models.
- Stacking: This method involves training multiple models and then using their predictions as input to a final “meta-model” that makes the final prediction.
Ensemble methods have been shown to be highly effective in improving model accuracy, particularly in cases where the individual models are weak or highly variable. However, ensemble methods can also be computationally expensive and may require a large amount of data to be effective.
One key advantage of ensemble methods is that they can often be used with a wide range of different machine learning algorithms, including decision trees, neural networks, and support vector machines. This makes ensemble methods a versatile and powerful tool for improving model accuracy in a variety of applications.
Best Practices for Model Accuracy
Model Interpretability
In order to improve model accuracy, it is essential to ensure that the model is interpretable. Model interpretability refers to the ability to understand and explain how the model works. This is important because it allows developers to identify and fix errors in the model, as well as to ensure that the model is making accurate predictions.
There are several ways to improve model interpretability, including:
- Feature Importance Analysis: This involves analyzing the importance of each feature in the model. By understanding which features are most important, developers can focus on improving those features to improve the overall accuracy of the model.
- Decision Trees: Decision trees are a popular method for improving model interpretability. They provide a visual representation of the model’s decision-making process, making it easier to understand how the model is making predictions.
- Learning to Interpret Models: Developers can also improve model interpretability by learning how to interpret the model’s output. This involves understanding the different types of errors that the model can make, as well as how to identify and correct those errors.
Overall, improving model interpretability is a critical aspect of developing accurate models. By understanding how the model works and identifying errors, developers can improve the accuracy of the model and ensure that it is making accurate predictions.
Model Validation and Testing
Effective model validation and testing are crucial steps in ensuring that your machine learning models are accurate and reliable. In this section, we will discuss the importance of model validation and testing, as well as the various techniques and best practices that can be used to improve model accuracy.
Importance of Model Validation and Testing
Model validation and testing are essential components of the machine learning process, as they help to ensure that the model is accurate and reliable. Model validation involves checking the model’s performance on a separate dataset that was not used during training, while model testing involves checking the model’s performance on a completely separate dataset.
Validating and testing the model are important because they allow you to evaluate the model’s performance and identify any issues or errors that may have been introduced during the training process. By catching these issues early on, you can make necessary adjustments to the model and improve its accuracy.
Techniques for Model Validation and Testing
There are several techniques that can be used to validate and test a machine learning model, including:
- Cross-validation: This technique involves dividing the dataset into multiple subsets and training the model on each subset, while using the remaining subset for validation. This helps to ensure that the model is not overfitting to any particular subset of the data.
- Holdout validation: This technique involves setting aside a portion of the dataset as a test set, while using the remaining portion as the training set. The model is then trained on the training set and tested on the test set.
- K-fold cross-validation: This technique involves dividing the dataset into k subsets and training the model on k-1 of the subsets, while using the remaining subset for validation. This is repeated k times, with a different subset being used for validation each time. The final accuracy is then calculated as the average of the k validation accuracies.
Best Practices for Model Validation and Testing
In addition to using these techniques, there are several best practices that can help to improve model accuracy through validation and testing:
- Use a large, diverse dataset: A large, diverse dataset can help to ensure that the model is not overfitting to any particular subset of the data.
- Regularly evaluate the model’s performance: It is important to regularly evaluate the model’s performance on the validation and test sets, and make adjustments as necessary.
- Use appropriate evaluation metrics: Depending on the problem being solved, different evaluation metrics may be more appropriate than others. For example, accuracy may be a good metric for classification problems, while F1 score may be a better metric for imbalanced datasets.
- Consider the model’s performance on edge cases: It is important to consider the model’s performance on edge cases, as these can often be the most challenging cases for the model to handle.
By following these best practices, you can improve the accuracy of your machine learning models and ensure that they are reliable and effective.
Model Deployment and Monitoring
Effective model deployment and monitoring are crucial for maintaining high accuracy in machine learning models. In this section, we will discuss best practices for deploying models in production environments and monitoring their performance over time.
Model Deployment
Once a model has been trained and validated, it’s time to deploy it to a production environment. Here are some best practices for model deployment:
- Scalability: Ensure that the model can scale to handle increasing amounts of data and traffic. This may involve deploying the model on a cloud-based infrastructure or using a distributed computing framework like Apache Spark.
- Performance: Optimize the model for performance by minimizing latency and reducing the number of requests required to process a single task. This may involve optimizing the model’s code or using a caching mechanism to reduce the number of times the model needs to be executed.
- Security: Ensure that the model is secure and cannot be easily accessed or tampered with by unauthorized users. This may involve implementing authentication and authorization mechanisms or encrypting sensitive data.
Model Monitoring
Once a model is deployed, it’s important to monitor its performance over time to ensure that it continues to provide accurate predictions. Here are some best practices for model monitoring:
- Performance Metrics: Establish a set of performance metrics to track over time, such as accuracy, precision, recall, and F1 score. These metrics should be tracked at both the individual model level and the system level to ensure that the model is providing value to the overall system.
- Data Drift: Monitor the model for data drift, which occurs when the distribution of the input data changes over time. Data drift can lead to decreased accuracy and should be detected and addressed as soon as possible.
- User Feedback: Collect user feedback on the model’s performance to ensure that it is meeting the needs of its users. This feedback can be used to improve the model’s accuracy and address any issues that users may be experiencing.
By following these best practices for model deployment and monitoring, you can ensure that your machine learning models continue to provide accurate predictions over time and deliver value to your organization.
Real-World Examples of Model Accuracy
Industry Use Cases
Model accuracy plays a crucial role in various industries, from healthcare to finance. Here are some real-world examples of how model accuracy impacts different sectors:
Healthcare
In healthcare, accurate models are essential for predicting patient outcomes, diagnosing diseases, and recommending treatments. For instance, machine learning models can help predict the risk of readmission for patients with heart failure, allowing healthcare providers to take preventive measures and improve patient care.
Finance
Accurate models are critical in the finance industry for fraud detection, credit scoring, and risk assessment. Banks and financial institutions use machine learning models to analyze customer data and identify potential fraud, reducing the risk of financial losses. Additionally, credit scoring models help determine the creditworthiness of borrowers, enabling lenders to make informed decisions.
Retail
Retailers use machine learning models to analyze customer data and predict purchasing behavior. By analyzing customer preferences and past purchases, retailers can personalize marketing campaigns and recommend products that are more likely to be purchased. This results in increased sales and customer satisfaction.
Manufacturing
In the manufacturing industry, accurate models are used for predictive maintenance, quality control, and supply chain optimization. By analyzing data from machines and sensors, manufacturers can predict when equipment is likely to fail, reducing downtime and improving efficiency. Additionally, machine learning models can help identify quality issues during the production process, ensuring that products meet customer specifications.
Transportation
Accurate models are also critical in the transportation industry for route optimization, demand forecasting, and predictive maintenance. Transportation companies use machine learning models to optimize routes, reducing travel time and fuel consumption. Additionally, models can help forecast demand for transportation services, allowing companies to allocate resources more efficiently.
Overall, model accuracy plays a vital role in various industries, enabling businesses to make informed decisions, reduce costs, and improve customer satisfaction.
Success Stories and Lessons Learned
- Case Study 1: Amazon’s AI-based recommendation system
- Amazon’s recommendation system leverages collaborative filtering and content-based filtering techniques to suggest products to customers based on their browsing and purchase history.
- By utilizing a combination of machine learning algorithms, the system has achieved impressive results, increasing customer satisfaction and driving sales.
- Lessons learned:
- Personalization is key to enhancing user experience and increasing customer loyalty.
- The use of multiple filtering techniques can lead to more accurate recommendations.
- Key takeaway: Combining multiple techniques and continuously refining the model can lead to significant improvements in accuracy and business outcomes.
- Case Study 2: Google’s image recognition technology
- Google’s image recognition technology uses deep learning models like Convolutional Neural Networks (CNNs) to classify images in various applications, such as Google Photos and Google Images.
- The system’s high accuracy has enabled Google to provide superior image search results and tagging capabilities.
- Deep learning models can achieve state-of-the-art performance in image recognition tasks.
- Continuous optimization and updating of the model with new data is crucial for maintaining high accuracy.
- Key takeaway: Investing in advanced model architectures and continuous learning can lead to breakthrough improvements in accuracy and application performance.
- Case Study 3: Netflix’s movie and TV show recommendation system
- Netflix’s recommendation system utilizes a hybrid approach that combines collaborative filtering, content-based filtering, and matric factorization techniques to provide personalized recommendations to users.
- The system has been instrumental in enhancing user engagement and driving subscription growth for the streaming giant.
- A hybrid approach that combines multiple filtering techniques can lead to more accurate recommendations.
- Regularly updating the model with new data is essential for maintaining high accuracy.
- Key takeaway: Adopting a multi-faceted approach and regularly updating the model can significantly improve accuracy and user satisfaction.
FAQs
1. What is model accuracy?
Model accuracy refers to the measure of how well a machine learning model performs on a specific task. It is a quantitative evaluation of the model’s ability to make accurate predictions or classifications based on the input data.
2. How is model accuracy calculated?
Model accuracy is typically calculated by comparing the predicted output of the model with the actual output or ground truth. The most commonly used metric for accuracy is the percentage of correct predictions, also known as the accuracy rate or the classification accuracy.
3. What is a good model accuracy?
A good model accuracy depends on the specific task and the data being used. In general, a higher accuracy rate is desirable, but it is important to consider other factors such as the complexity of the model, the size of the dataset, and the cost of computation. In some cases, a model with a lower accuracy rate may be preferred over a more complex model that requires more computational resources.
4. How can I improve the accuracy of my model?
Improving the accuracy of a model requires a systematic approach that involves understanding the limitations of the current model, collecting more data, selecting the right features, and fine-tuning the model parameters. Additionally, using more advanced algorithms, incorporating external data sources, and applying techniques such as cross-validation and hyperparameter tuning can also improve the accuracy of the model.
5. What is the difference between accuracy and precision?
Accuracy and precision are two common metrics used to evaluate the performance of a machine learning model. Accuracy measures the proportion of correct predictions, while precision measures the proportion of true positive predictions out of all positive predictions. A model with high accuracy may have a high false positive rate, while a model with high precision may have a low recall rate. Therefore, it is important to consider both metrics when evaluating the performance of a model.
