2nd K-AI Manufacturing Competition (3)

▤ 목차

Model Evaluation and Results

728x90

반응형

Loss Function and Training Progress

• Loss Function:
  • The model was trained using the binary cross-entropy loss function, which is well-suited for binary classification tasks. It calculates the divergence between the predicted probabilities and the true labels, guiding the model's optimization.
  • Binary cross-entropy ensures that the model penalizes incorrect predictions proportionally to their confidence, encouraging more accurate probability outputs.
• Training Progress:
  • During training, both the training loss and validation loss were monitored over epochs. The following observations were made:
    1. Loss values decreased steadily during the initial epochs, indicating effective learning.
    2. After approximately 13 epochs, validation loss began to stabilize and occasionally increase, suggesting the onset of overfitting.
    3. Early stopping terminated training at the optimal point, typically around 20 epochs, to ensure the model achieved peak performance without overfitting.
• Visual Insight:
  • Loss and accuracy curves were plotted to visualize the training process. The graphs demonstrated a consistent improvement in training and validation during the early epochs, followed by stabilization. Early stopping effectively prevented significant divergence between the two curves.

01

Loss & accuracy curves

Evaluation Metrics

To assess the model's performance comprehensively, several evaluation metrics were employed:

1. Confusion Matrix:
   • The confusion matrix highlights the model's prediction outcomes across four categories:
     • True Negatives (TN): Samples correctly identified as defective (NG).
     • False Positives (FP): Samples incorrectly identified as acceptable (OK).
     • False Negatives (FN): Samples incorrectly identified as defective (NG).
     • True Positives (TP): Samples correctly identified as acceptable (OK).
   • Results from the test dataset showed:
     • TN: 38,185
     • FP: 17,762
     • FN: 22,240
     • TP: 172,363

       

2. Accuracy:
   • Accuracy measures the overall correctness of predictions: $\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{Total Samples}}$
   • The model achieved an accuracy of 84.03%, indicating strong performance on the test dataset.
3. F1-Score:
   • The F1-score considers both precision and recall, providing a balanced measure of the model's classification performance: $\text{F1-Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$
   • The F1-score for the model was 0.896, demonstrating its effectiveness in handling imbalanced data.
4. AUC (Area Under the ROC Curve):
   • The AUC measures the model's ability to distinguish between classes across all threshold values. A higher AUC indicates better discriminatory power.
   • The model achieved an AUC of 0.871, reflecting its strong capacity to differentiate between defective and acceptable products.

 

Results and Interpretation

• Model Strengths:
  • High accuracy and F1-score indicate that the model performs well in both identifying acceptable products and detecting defects.
  • The high AUC score further confirms the model's reliability in distinguishing between the two classes.
• Challenges:
  • Despite the overall success, a moderate number of false negatives (22,240) were observed, where defective products were incorrectly classified as acceptable. This may be due to subtle patterns in the data that the model struggled to capture.
  • Further refinements, such as adding more features or fine-tuning the GRU layers, could help reduce false negatives.
• Visualizations:
  • The ROC curve and confusion matrix were visualized to understand the model's performance better. These plots reinforced the model’s robustness while highlighting areas for potential improvement.

Key Takeaway:

The GRU-based predictive model achieved a strong balance between accuracy, precision, and recall, demonstrating its utility in real-time quality assessment during the melting process. While effective, the model's ability to reduce false negatives could be further optimized to enhance defect detection reliability.

Implications for Manufacturing SMEs

Application in Real-World Manufacturing Processes

The proposed predictive model addresses key challenges in current manufacturing environments, particularly the reliance on manual interventions and operator expertise. Its integration into manufacturing processes has several practical benefits:

1. Enhanced Process Monitoring and Control:
   • Real-time predictions of product quality allow for proactive adjustments to operational parameters (e.g., temperature, stirring speed).
   • Operators can rely on data-driven insights rather than subjective evaluations, reducing variability caused by human error.
2. Improved Decision-Making in the Absence of Skilled Operators:
   • The model provides actionable insights for less experienced personnel, ensuring consistent product quality even in the absence of skilled operators.
   • By analyzing live data from PLCs or DBMS systems, the model minimizes dependency on operator expertise, bridging skill gaps in the workforce.
3. Facilitating Automation and Smart Factory Integration:
   • In advanced manufacturing setups, the model can be integrated into a fully automated smart factory ecosystem.
   • By linking real-time predictions with control systems, process adjustments can be automated, enabling consistent production quality without manual intervention.

Scalability to Other Processes and Sectors

While the current study focuses on the melting process in powdered cream production, the model's principles are transferable to other manufacturing processes and industries:

1. Adaptability to New Products and Raw Materials:
   • The model can be retrained with data from different raw materials, processes, or product specifications, making it flexible for various manufacturing environments.
   • Custom weights can be assigned to critical variables based on the unique requirements of other processes.
2. Incorporation of External Variables:
   • External factors, such as environmental conditions (e.g., temperature, humidity), can be integrated into the model to enhance prediction accuracy further.
   • Collaboration with field experts can help identify additional relevant variables, expanding the model’s applicability.
3. Cross-Industry Potential:
   • Beyond food manufacturing, the model can be applied in industries such as chemicals, pharmaceuticals, and electronics, where maintaining precise operational conditions is critical to product quality.

Key Takeaway:

By implementing this predictive model, SMEs can transition from reactive to proactive process management, improving product quality, reducing waste, and enabling scalability to more automated and data-driven operations. The model's adaptability to other processes and industries offers a pathway for widespread adoption and long-term value creation.