Knowledge Documentation Framework for AI Initiatives: Development and Validation Across Organizational AI Contexts
Downloads
Despite the growing organizational reliance on artificial intelligence (AI) systems, knowledge documentation (KD) practices in AI initiatives remain largely ad hoc, unstandardized, and disconnected from project lifecycle management. This study addressed this gap by proposing and validating the Knowledge Documentation Framework for AI Initiatives (KDF-AI), a twelve-component, five-phase, maturity-tiered framework synthesized through a literature review of peer-reviewed studies on KD practices in organizations implementing AI. Using Design Science Research (DSR) as the methodological paradigm, the study completed two iterative cycles: a literature-based synthesis that produced KDF-AI v1 and an expert evaluation cycle that resulted in the refined KDF-AI v2. Expert content validation was conducted with three domain validators representing academic, governance, and AI practitioner perspectives using a mixed-method approach that combined quantitative Content Validity Index (CVI) assessment with deductive thematic analysis of semi-structured interviews. The results showed that 57 of 66 items (86.4%) achieved universal inter-rater agreement, producing S-CVI/UA = 0.864 and S-CVI/Ave = 0.955, both exceeding the recommended threshold of 0.80. The nine items that did not meet the threshold consistently reflected issues of clarity rather than relevance, indicating strong conceptual acceptance of the framework while highlighting the need for more operationally specific articulation in several Advanced-tier components. Nine targeted revisions resulted in the development of KDF-AI v2. The study contributed: (1) a validated lifecycle-integrated KD framework for AI initiatives; (2) a taxonomy of ten systematically identified gaps in current AI KD practices; and (3) a methodological demonstration of mixed-method CVI validation for framework development in information systems research.
Almanasreh, E., Moles, R. J., & Chen, T. F. (2022). A practical approach to the assessment and quantification of content validity. In Contemporary research methods in pharmacy and health services (pp. 583–599). Elsevier.
Braun, V., Clarke, V., & Hayfield, N. (2022). ‘A starting point for your journey, not a map’: Nikki Hayfield in conversation with Virginia Braun and Victoria Clarke about thematic analysis. Qualitative Research in Psychology, 19(2), 424–445.
Jarrahi, M. H., Askay, D., Eshraghi, A., & Smith, P. (2023). Artificial intelligence and knowledge management: A partnership between human and AI. Business Horizons, 66(1), 87–99.
Kova?i?, M., Mutavdžija, M., Buntak, K., & Pus, I. (2022). Using artificial intelligence for creating and managing organizational knowledge. Tehni?ki Vjesnik, 29(4), 1413–1418.
Olan, F., Arakpogun, E. O., Suklan, J., Nakpodia, F., Damij, N., & Jayawickrama, U. (2022). Artificial intelligence and knowledge sharing: Contributing factors to organizational performance. Journal of Business Research, 145, 605–615.
Oyekunle, D., & Boohene, D. (2024). Digital transformation potential: The role of artificial intelligence in business. International Journal of Professional Business Review: Int. J. Prof. Bus. Rev., 9(3), 1.
Song, Y., Qiu, X., & Liu, J. (2025). The impact of artificial intelligence adoption on organizational decision-making: An empirical study based on the technology acceptance model in business management. Systems, 13(8), 683.
Cha, I., Oh, J., Park, C. Y., Han, J., & Lee, H. (2023). Unlocking the tacit knowledge of data work in machine learning. In Extended abstracts of the 2023 CHI conference on human factors in computing systems (CHI EA '23). ACM.
Chang, J., & Custis, C. (2022). Understanding implementation challenges in machine learning documentation. In Proceedings of the 2nd ACM conference on equity and access in algorithms, mechanisms, and optimization (EAAMO '22). ACM.
Gelashvili-Luik, T., Vihma, P., & Pappel, I. (2025). Navigating the AI revolution: Challenges and opportunities for integrating emerging technologies into knowledge management systems. Frontiers in Artificial Intelligence, 8, Article 1518744.
Gerlach, J. P., & Lange, D. (2026). Fading memories: The role of machine learning in organizational knowledge depreciation. Academy of Management Review. Advance online publication.
Gundersen, O. E., & Kjensmo, S. (2018). State of the art: Reproducibility in artificial intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).
Harfouche, A., Kalika, M., Mouakket, S., & Ralyte, J. (2022). A framework for AI-supported knowledge creation and the KAM model. Information Systems Frontiers, 24, 1545–1562.
Heger, A. K., Marquis, L. B., & Vorvoreanu, M. (2022). Understanding machine learning practitioners' data documentation perceptions, needs, challenges, and desiderata. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2).
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.
Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), Article 100804.
Konigstorfer, F. (2024). A comprehensive review of techniques for documenting artificial intelligence. Digital Policy, Regulation and Governance. Advance online publication.
Konigstorfer, F., & Thalmann, S. (2021). Software documentation is not enough! Requirements for the documentation of AI. Digital Policy, Regulation and Governance, 23(3), 245–260.
Olan, F., Ogiemwonyi Arakpogun, E., Suklan, J., Nakpodia, F., Damij, N., & Jayawickrama, U. (2022). Artificial intelligence and knowledge sharing: Contributing factors to organizational performance. Journal of Business Research, 145, 605–615.
Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45–77.
Raji, I. D., Smart, A., White, R. N., & Mitchell, M. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 ACM Conference on Fairness, Accountability, and Transparency (FAT '20)*. ACM.
Taherdoost, H., & Madanchian, M. (2023). Artificial intelligence and knowledge management: Impacts, benefits, and implementation. Computers, 12(4), Article 72.
Copyright (c) 2026 Fitria Handayani, Finannisa Zhafira, Dana Indra Sensuse, Sofian Lusa
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA). that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
