







Metadata management plays a critical role in data governance, resource discovery, and decision-making in the data-driven era. While traditional metadata approaches have primarily focused on organization, classification, and resource reuse, the integration of modern artificial intelligence (AI) technologies has significantly transformed these processes. This paper investigates both traditional and AI-driven metadata approaches by examining open-source solutions, commercial tools, and research initiatives. A comparative analysis of traditional and AI-driven metadata management methods is provided, highlighting existing challenges and their impact on next-generation datasets. The paper also presents an innovative AI-assisted metadata management framework designed to address these challenges. This framework leverages more advanced modern AI technologies to automate metadata generation, enhance governance, and improve the accessibility and usability of modern datasets. Finally, the paper outlines future directions for research and development, proposing opportunities to further advance metadata management in the context of AI-driven innovation and complex datasets.
Metadata Integration: A Systematic Review on Methods, Challenges and Applications Across Multiple Domains
Metadata contributes to ensuring semantic interoperability, data reuse, and efficient information retrieval across heterogeneous systems. However, existing studies present fragmented approaches and lack a consolidated view of the strategies adopted in different domains. This fragmentation creates a gap in understanding the current state of the art and identifying cross-domain trends and challenges. This systematic review aims to address this gap by analyzing metadata integration practices across multiple domains, including Health and Medicine, Smart Cities and IoT, Data Science, Geosciences, Cultural Heritage, Library and Information Science. This review follows the Kitchenham structure, encompassing the planning, execution, and reporting phases. A total of 81 peer-reviewed articles published between 2014 and 2025 were selected from five major databases, based on the defined inclusion and exclusion criteria and a structured quality assessment process. The results reveal a predominance of the use of ontology, followed by metadata-driven language, standards, processes, and common metadata schema with an emerging trend towards automation through the adoption of AI-based approaches. The identified challenges include semantic heterogeneity, lack of standardization, limited automation, and usability issues in the current tools and platforms. We provide a comprehensive synthesis of existing approaches, highlighting domain-specific and cross-domain trends, and identifying research opportunities to advance metadata integration through automation, semantic enrichment, and standardized practices.
(PDF) Understanding the Nature of Metadata: Systematic Review
PDF | Background Metadata are created to describe the corresponding data in a detailed and unambiguous way and is used for various applications in... | Find, read and cite all the research you need on ResearchGate

DSWB | Metadata Standards: Selecting and Applying Standards in Research | Webinar Recording
Guest Post — Scholarly AI Search Shortcomings and the Need for Better Metadata - The Scholarly Kitchen
AI scholarly search tools often miss important literature due to incomplete metadata. Better full-text-derived metadata could significantly improve discovery.

Databricks: Leading Data and AI Platform for Enterprises
Databricks offers a unified platform for data, analytics and AI. Build better AI with a data-centric approach. Simplify ETL, data warehousing, governance and AI on the Data Intelligence Platform.

Manuscript submission systems and metadata completeness in Crossref: Patterns and associations
The importance of open research information, particularly publication metadata, is widely recognised. Crossref is one of the most important infrastructures for registering open metadata as part of DOI record registration. It is widely known, however, that the metadata of many publications is far from complete, with many publishers making certain metadata openly available, but failing to do so for other metadata elements. Publishers’ ability to register this metadata with Crossref depends on their capacity to capture and retain this data in their production workflows. Manuscript submission systems are an important, yet largely overlooked, factor in the extent to which publishers make metadata available through Crossref. In this paper, we present the results of an analysis investigating the relation between the level of metadata that publishers deposit with Crossref and the submission systems that they deploy for their journals. We have looked at the 153 publishers with the largest amounts of publications in Crossref and concentrate on the four most commonly used systems: Editorial Manager, ScholarOne, Open Journal Systems (OJS) and eJournalPress. We show that some submission systems appear better suited to capturing certain metadata elements. However, there are always cases where publishers using the same system differ widely in the level of metadata they register, suggesting that technology is not the only prohibiting factor and other considerations are at play.
A large-scale audit of dataset licensing and attribution in AI
The race to train language models on vast, diverse and inconsistently documented datasets raises pressing legal and ethical concerns. To improve data transparency and understanding, we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace more than 1,800 text datasets. We develop tools and standards to trace the lineage of these datasets, including their source, creators, licences and subsequent use. Our landscape analysis highlights sharp divides in the composition and focus of data licenced for commercial use. Important categories including low-resource languages, creative tasks and new synthetic data all tend to be restrictively licenced. We observe frequent miscategorization of licences on popular dataset hosting sites, with licence omission rates of more than 70% and error rates of more than 50%. This highlights a crisis in misattribution and informed use of popular datasets driving many recent breakthroughs. Our analysis of data sources also explains the application of copyright law and fair use to finetuning data. As a contribution to continuing improvements in dataset transparency and responsible use, we release our audit, with an interactive user interface, the Data Provenance Explorer, to enable practitioners to trace and filter on data provenance for the most popular finetuning data collections: www.dataprovenance.org.

How are library cataloging metadata used differently over time: a large-scale quantitative analysis of the Library of Congress catalog
Purpose. The library catalog is not only a useful tool for patrons to find and access library collections but also a valuable dataset for various types of quantitative analysis on our cultural assets. This research aims to develop the latter potential of library metadata by conducting a large-scale analysis of how the metadata fields encoded in the Machine-Readable Cataloging (MARC) standards.Design/methodology/approach. We examined more than 6 million book records from 1980 to 2018 in the catalog of the Library of Congress. We particularly focused on how bibliographic records have changed over time and whether these changes correlate with the introduction of new cataloging policies.Findings. Our results show that more than 200 unique fields and 1,300 unique subfields have been used in the MARC format, though the majority of them are used in fewer than 1% of all records. At the same time, bibliographic records have become increasingly complex, with more fields and subfields per record over the past 40 years. Additionally, there are clear, although sometimes asynchronous, parallel developments between MARC tags and cataloging standards.Originality/value. This study represents the first large-scale quantitative analysis of the history of library metadata, revealing significant and interesting changes over time and highlighting challenges for the meaningful use of library metadata in the current data environment.

In Pursuit of Ideal Data: Epistemic Enchantment and the Unintended Consequences of Datafying for Artificial Intelligence
The increasing datafication of work has become a pervasive theme within organizations, particularly with the rise of artificial intelligence (AI) technologies that rely on data and machine learning to generate insights and decisions. Prior research has shown that datafication practices aimed at administrative control and accountability can transform organizations in profound and often unintended ways. Our research moves beyond investigating datafication for administrative purposes to explore how it unfolds when actors intend to extract knowledge through AI predictive modeling. Drawing on a three-year ethnography of a human resources (HR) department introducing AI for candidate screening, we find that datafication expanded far beyond the technology’s initially envisioned scope. We explain this scope expansion by introducing the concept of epistemic enchantment : a collective process through which organizational members become captivated and mobilized by AI’s promise to generate predictive, objective, and continuously improving insights from data, while overlooking the costs and risks involved. Our study reveals how datafication practices can expand endlessly as organizational members, enchanted by AI’s epistemic promises, pursue the notion of ideal data as an ever-desired but never reachable goal.

Semantic alignment of the German Human Genome-Phenome Archive metadata model in Europe’s genomics field
Legal and technical developments drive data sharing via federated infrastructures, especially in the field of human omics. This requires interoperability across technical, syntactic, organizational, and semantic layers. The German Human Genome-Phenome Archive (GHGA) has been building a national, federated infrastructure for secure sharing of human omics data. As part of its mission to enhance interoperability and to promote reliable data sharing, a detailed crosswalk analysis was conducted comparing the GHGA metadata model with four other domain-relevant standards and metadata models: EGA (Submission API and model draft), FAIR Genomes and ISA-tab. The analysis aimed at identifying semantic consensus fields to define datasets in the context of human omics by forward mapping (GHGA model to external models). Backward mapping (external models to GHGA) focused on spotting gaps in GHGA’s semantic metadata representation. Forward mapping showed overall similar property coverage across models, aligning with MINSEQE. Backward mapping showed greater model heterogeneity. None of the identified information gaps spanned across all models. These findings highlight the detail and adaptability of the GHGA metadata model.

GitHub - Responsible-Dataset-Sharing/easy-dataset-share: A CLI tool that helps AI researchers share datasets responsibly.
A CLI tool that helps AI researchers share datasets responsibly. - Responsible-Dataset-Sharing/easy-dataset-share
Karpathy's LLM Wiki: What It Means & How to Build One
Crossref: The sustainable source of community-owned scholarly metadata
This paper describes the scholarly metadata collected and made available by Crossref, as well as its importance in the scholarly research ecosystem. Containing over 106 million records and expanding at an average rate of 11% a year, Crossref’s metadata has become one of the major sources of scholarly data for publishers, authors, librarians, funders, and researchers. The metadata set consists of 13 content types, including not only traditional types, such as journals and conference papers, but also data sets, reports, preprints, peer reviews, and grants. The metadata is not limited to basic publication metadata, but can also include abstracts and links to full text, funding and license information, citation links, and the information about corrections, updates, retractions, etc. This scale and breadth make Crossref a valuable source for research in scientometrics, including measuring the growth and impact of science and understanding new trends in scholarly communications. The metadata is available through a number of APIs, including REST API and OAI-PMH. In this paper, we describe the kind of metadata that Crossref provides and how it is collected and curated. We also look at Crossref’s role in the research ecosystem and trends in metadata curation over the years, including the evolution of its citation data provision. We summarize the research used in Crossref’s metadata and describe plans that will improve metadata quality and retrieval in the future.

InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents
Data analysis has become an indispensable part of scientific research. To discover the latent knowledge and insights hidden within massive datasets, we need to perform deep exploratory analysis to realize their full value. With the advent of large language models (LLMs) and multi-agent systems, more and more researchers are making use of these technologies for insight discovery. However, there are few benchmarks for evaluating insight discovery capabilities. As one of the most comprehensive existing frameworks, InsightBench also suffers from many critical flaws: format inconsistencies, poorly conceived objectives, and redundant insights. These issues may significantly affect the quality of data and the evaluation of agents. To address these issues, we thoroughly investigate shortcomings in InsightBench and propose essential criteria for a high-quality insight benchmark. Regarding this, we develop a data-curation pipeline to construct a new dataset named InsightEval. We further introduce a novel metric to measure the exploratory performance of agents. Through extensive experiments on InsightEval, we highlight prevailing challenges in automated insight discovery and raise some key findings to guide future research in this promising direction.

OpenMetadata: #1 Open Source Context Layer
OpenMetadata is the open-source AI context layer — powered by metadata, semantics, and memory that give people and AI a shared, rich understanding of your data. Built by the founders of Apache Hadoop, Apache Atlas, and Uber Databook. The unified platform for data cataloging, discovery, quality, observability, governance, lineage, collaboration & more.
Wikidata Launches Free Vector Database as Open Alternative to Closed AI Systems
Wikidata has launched something big.
