Frontiers in Massive Data Analysis

by National Research Council, Division on Engineering and Physical Sciences, Board on Mathematical Sciences and Their Applications

Data mining of massive data sets is transforming the way we think about crisis response, marketing, entertainment, cybersecurity and national intelligence. Collections of documents, images, videos, and networks are being thought of not merely as bit strings to be stored, indexed, and retrieved, but as potential sources of discovery and knowledge, requiring sophisticated analysis techniques that go far beyond classical indexing and keyword counting, aiming to find relational and semantic interpretations of the phenomena underlying the data. Frontiers in Massive Data Analysis examines the frontier of analyzing massive amounts of data, whether in a static database or streaming through a system. Data at that scale-terabytes and petabytes-is increasingly common in science (e.g., particle physics, remote sensing, genomics), Internet commerce, business analytics, national security, communications, and elsewhere. The tools that work to infer knowledge from data at smaller scales do not necessarily work, or work well, at such massive scale. New tools, skills, and approaches are necessary, and this report identifies many of them, plus promising research directions to explore. Frontiers in Massive Data Analysis discusses pitfalls in trying to infer knowledge from massive data, and it characterizes seven major classes of computation that are common in the analysis of massive data. Overall, this report illustrates the cross-disciplinary knowledge-from computer science, statistics, machine learning, and application disciplines-that must be brought to bear to make useful inferences from massive data.

190 pages, Paperback

First published September 17, 2013

About the author

The National Research Council (NRC) functions under the auspices of the National Academy of Sciences (NAS), the National Academy of Engineering (NAE), and the Institute of Medicine (IOM). The NAS, NAE, IOM, and NRC are part of a private, nonprofit institution that provides science, technology and health policy advice under a congressional charter signed by President Abraham Lincoln that was originally granted to the NAS in 1863. Under this charter, the NRC was established in 1916, the NAE in 1964, and the IOM in 1970. The four organizations are collectively referred to as the National Academies.

The mission of the NRC is to improve government decision making and public policy, increase public education and understanding, and promote the acquisition and dissemination of knowledge in matters involving science, engineering, technology, and health. The institution takes this charge seriously and works to inform policies and actions that have the power to improve the lives of people in the U.S. and around the world.

The NRC is committed to providing elected leaders, policy makers, and the public with expert advice based on sound scientific evidence. The NRC does not receive direct federal appropriations for its work. Individual projects are funded by federal agencies, foundations, other governmental and private sources, and the institution’s endowment. The work is made possible by 6,000 of the world’s top scientists, engineers, and other professionals who volunteer their time without compensation to serve on committees and participate in activities. The NRC is administered jointly by the NAS, NAE, and the IOM through the NRC Governing Board.

The core services involve collecting, analyzing, and sharing information and knowledge. The independence of the institution, combined with its unique ability to convene experts, allows it to be responsive to a host of requests.

The portfolio of activities includes:

* Consensus Studies: These comprehensive reports focus on major policy issues and provide recommendations for solving complex problems.
* Expert Meetings and Workshops: By convening symposia, workshops, meetings, and roundtables, the NRC connects professionals as well as the interested public and stimulates dialogue on diverse matters.
* Program and Research Management: At the request of state and federal agencies, the NRC manages and evaluates research programs, conducts program assessments, and reviews proposals.
* Fellowships: The NRC administers several postdoctoral fellowship programs.

Free Scientific Information: Publishing more than 200 reports and related publications each year, the institution is one of the largest providers of free scientific and technical information in the world. Most of it is now on the Web at www.nap.edu.