Abstract

The progression of multimodal machine learning (MML), a pivotal aspect of the artificial intelligence (AI) revolution, greatly enhances the analysis and comprehension of video data by providing insights across several modalities, including text, audio, and visual formats. MML models have extensive applications in entertainment, healthcare, and autonomous systems; nevertheless, when trained on expansive video datasets that encompass a diverse cultural, ethnic, and linguistic spectrum, they encounter significant challenges related to fairness and prejudice. This work presents a comprehensive investigation of bias reduction and fairness in MML, addressing issues arising from the intricacies of large-scale video data. This study (1) identifies the origins and mechanisms of bias in MML systems, (2) introduces advanced methodologies to enhance model fairness, and (3) offers a thorough framework for assessing fairness in large-scale video datasets. We offer a framework that integrates bias-aware pre-processing, fairness-aware modeling across multimodal settings, and scalable assessment metrics. Specifically, we employ balanced sampling, GANs for synthetic data augmentation, and adversarial debiasing to provide equitable representation and prediction. We validate our methods on extensive benchmark datasets (YouTube-8M, Activity Net, and Ego4D), demonstrating substantial enhancements in performance and fairness. The experimental findings indicate that the multimodal model surpasses both the modal models and the state-of-the-art techniques, achieving an accuracy of 90.5% and an F1 score of 91.0%. Ultimately, we enhanced fairness measurements, specifically differential impact and equalized chances, by 32.3% and 17.9%, respectively, demonstrating the efficacy of our bias mitigation strategies. However, comparative assessments reveal that our technique delivers state-of-the-art performance on the trade-off between predictive accuracy and ethical fairness, making it a feasible option for real-life contexts where equity is critical. Qualitative study corroborates the alleviation of demographic bias in model predictions, especially on sensitive tasks such as emotion detection and demographic classification. Our research enhances the ethical use of MML systems, guaranteeing that these models are resilient and equitable among diverse population subgroups while establishing a foundation for future advancements in multimodal fusion methodologies and task-specific fairness metrics.

References

• Brynjolfsson, E., & McElheran, K. (2019). Data in Al-Zoghby, A., Al-Awadly, E., Ebada, A. I., & Awad, W. A. E. K. (2024). Overview of Multimodal Machine Learning. ACM Transactions on Asian and Low-Resource Language Information Processing.
• Barua, A., Ahmed, M. U., & Begum, S. (2023). A systematic literature review on multimodal machine learning: Applications, challenges, gaps and future directions. IEEE Access, 11, 14804-14831.
• Ren, Y., Xu, N., Ling, M., & Geng, X. (2022). Label distribution for multimodal machine learning. Frontiers of Computer Science, 16, 1-11.
• Ouenniche, K. (2023). Multimodal deep learning for audiovisual production (Doctoral dissertation, Institut Polytechnique de Paris).
• Haouhat, A., Bellaouar, S., Nehar, A., & Cherroun, H. (2023). Modality Influence in Multimodal Machine Learning. arXiv preprint arXiv:2306.06476.
• Zhang, H. (2022). Multimodal learning for quality of experience in video-to-retail applications: algorithms and deployment.
• Caton, S., & Haas, C. (2024). Fairness in machine learning: A survey. ACM Computing Surveys, 56(7), 1-38.
• Jui, T. D., & Rivas, P. (2024). Fairness issues, current approaches, and challenges in machine learning models. International Journal of Machine Learning and Cybernetics, 1-31.
• Oluwaseyi, J. (2024). Bias and Fairness in Machine Learning Algorithms: Detection, Mitigation, and Accountability. Data Science.
• Kheya, T. A., Bouadjenek, M. R., & Aryal, S. (2024). The Pursuit of Fairness in Artificial Intelligence Models: A Survey. arXiv preprint arXiv:2403.17333.
• Mavrogiorgos, K., Kiourtis, A., Mavrogiorgou, A., Menychtas, A., & Kyriazis, D. (2024). Bias in Machine Learning: A Literature Review. Applied Sciences, 14(19), 8860.
• Balayn, A., Lofi, C., & Houben, G. J. (2021). Managing bias and unfairness in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systems. The VLDB Journal, 30(5), 739-768.
• Balayn, A., Lofi, C., & Houben, G. J. (2021). Managing bias and unfairness in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systems. The VLDB Journal, 30(5), 739-768.
• Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., & Busch, C. (2020). Demographic bias in biometrics: A survey on an emerging challenge. IEEE Transactions on Technology and Society, 1(2), 89-103.
• Nandan Prasad, A. (2024). Ethical Implications and Bias Mitigation. In Introduction to Data Governance for Machine Learning Systems: Fundamental Principles, Critical Practices, and Future Trends (pp. 307-382). Berkeley, CA: Apress.
• Shrestha, S. (2023). Design, Determination, and Evaluation of Gender-Based Bias Mitigation Techniques for Music Recommender Systems (Master's thesis, University of Denver).
• Salem, J., Desai, D., & Gupta, S. (2022, June). Don’t let Ricci v. DeStefano hold you back: A bias-aware legal solution to the hiring paradox. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 651-666).
• Dash, S. (2023). Fairness in Image Search: A Study of Occupational Stereotyping in Image Retrieval and its Debiasing. arXiv preprint arXiv:2305.03881.
• Zhao, C., Chen, F., & Thuraisingham, B. (2021, August). Fairness-aware online meta-learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 2294-2304).
• Chen, A., Rossi, R. A., Park, N., Trivedi, P., Wang, Y., Yu, T., ... & Ahmed, N. K. (2024). Fairness-aware graph neural networks: A survey. ACM Transactions on Knowledge Discovery from Data, 18(6), 1-23.
• Yang, J., Jiang, J., Sun, Z., & Chen, J. (2024, September). A large-scale empirical study on improving the fairness of image classification models. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 210-222).
• Hosseini, R., Zhang, L., Garg, B., & Xie, P. (2023, July). Fair and accurate decision making through group-aware learning. In International Conference on Machine Learning (pp. 13254-13269). PMLR.
• Zhao, C. (2021). Fairness-Aware Multi-Task and Meta Learning (Doctoral dissertation).
• Sun, Y. (2022). Fairness-Aware Data-Driven Building Models (DDBMs) and Their Application in Model Predictive Controller (MPC) (Doctoral dissertation, Concordia University).
• Jain, B., Huber, M., & Elmasri, R. (2024). Fairness for deep learning predictions using bias parity score based loss function regularization. International Journal on Artificial Intelligence Tools, 33(03), 2460003.
• Sarridis, I., Koutlis, C., Papadopoulos, S., & Diou, C. (2024). Flac: Fairness-aware representation learning by suppressing attribute-class associations. IEEE Transactions on Pattern Analysis and Machine Intelligence.
• Li, Y., Yang, Y., Cao, J., Liu, S., Tang, H., & Xu, G. (2024, August). Toward Structure Fairness in Dynamic Graph Embedding: A Trend-aware Dual Debiasing Approach. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 1701-1712).
• Kim, D., Woo, H., & Lee, Y. (2024). Addressing Bias and Fairness Using Fair Federated Learning: A Synthetic Review. Electronics, 13(23), 4664.
• Dehdashtian, S., He, R., Li, Y., Balakrishnan, G., Vasconcelos, N., Ordonez, V., & Boddeti, V. N. (2024). Fairness and Bias Mitigation in Computer Vision: A Survey. arXiv preprint arXiv:2408.02464.
• Adewumi, T., Alkhaled, L., Gurung, N., van Boven, G., & Pagliai, I. (2024). Fairness and bias in multimodal ai: A survey. arXiv preprint arXiv:2406.19097.
• Goyal, P., Duval, Q., Seessel, I., Caron, M., Misra, I., Sagun, L., ... & Bojanowski, P. (2022). Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint arXiv:2202.08360.
• Luk, M. (2023). Generative AI: Overview, economic impact, and applications in asset management. Economic Impact, and Applications in Asset Management (September 18, 2023).
• Ma, Q., Xue, X., Zhou, D., Yu, X., Liu, D., Zhang, X., ... & Ma, W. (2024). Computational experiments meet large language model based agents: A survey and perspective. arXiv preprint arXiv:2402.00262.
• Pulapaka, S., Godavarthi, S., & Ding, S. Empowering the Public Sector with Generative AI.
• Li, Z. (2021). Learning Geometry, Appearance and Motion in the Wild. Cornell University.
• Eibeck, A., Shaocong, Z., Mei Qi, L., & Kraft, M. (2024). Research data supporting" A Simple and Efficient Approach to Unsupervised Instance Matching and its Application to Linked Data of Power Plants".
• Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021). Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963.
• Greene, T., Shmueli, G., & Ray, S. (2023). Taking the person seriously: Ethically aware IS research in the era of reinforcement learning-based personalization. Journal of the Association for Information Systems, 24(6), 1527-1561.
• Lavian, T. I. (2006). Lambda Data Grid: Communications Architecture in Support of Grid Computing (Doctoral dissertation, University of California, Berkeley).
• Mayer, C. B. (2005). Quality-based replication of freshness-differentiated web applications and their back-end databases. Arizona State University.