North American News Text, Complete

Data

The table below contains a breakdown of the sources, epochs and word counts for the data in the North American News Text releases:

The New York Times and the Los Angeles Times/Washington Post services include a range of other newspaper sources in their syndicated newswires. The Los Angeles Times/Washington Post material in this corpus includes some news text from the following sources:

• Newsday
• The Baltimore Sun
• The Hartford Courant

The New York Times material in this corpus contains some data from the following sources, although New York Times articles predominate:

• Bloomberg Business News
• The Boston Globe
• Los Angeles Daily News
• Fort Worth Star-Telegram
• Newsweek
• Cox News Service
• The Arizona Republic
• Seattle Post-Intelligencer
• San Francisco Examiner
• Houston Chronicle
• San Francisco Chronicle
• Economist Newspaper Ltd.
• Hearst Newspapers

The text content of each data file (following uncompression with the GNU-unzip utility) consists of plain ASCII character data with SGML tags to indicate article boundaries and organization of information within each article.

There are differences among the five primary newswire sources in terms of the number and types of SGML tags used in the text, but the following tag structure is common to all data sets:

-- start of a new article ... -- some variety of "header" tags appears here -- start of the text content of the article

-- all paragraph boundaries are marked by this tag ... -- text data as it is provided by the newswire service

In general, the differences in format among the various newswire sources will be found in the SGML tags that appear between and , and those that appear between and . The actual text content of articles (the region between and ) is consistent in format across sources, except for some uses of the SGML "&..;" notation to represent special characters in the data. For example, "&MD;" is used in the "latwp" material to represent the "em-dash" character, which is typically used to separate the "dateline" from the opening sentence in the first paragraph of each article. There may also be differences in how quotation marks are rendered.

As this re-release is intended to complement the BLLIP North American News Text releases, the directory structure of this corpus is identical to that of the BLLIP publications.