Corpus Analysis

Contents

• What is a Corpus
• Corpus Creation
• Corpus Annotation
• Corpus Analysis Tools
• Corpus Encoding Standards
• Markup Languages: SGML/XML
• Corpus Search
  • Corpus Search Tools

• Repositories of Corpora

• Corpus-based Lexicography

• The use of WordNet in Text Analysis (coming soon)

What is a Corpus

Corpus Creation

• Electronic form: adaptation of material already in electronic form, scanning, keyboarding.

• Permissions: copyright, safeguard against exploitation and piracy.

• Design: types and proportions of material in it (spoken and written language, from formal to informal, from literary to ordinary, etc).

• Characteristics:
  • quantity => large
  • quality => authentic
  • simplicity => plain text
  • documented => documented

Corpus Annotation

• Plain corpus: it appears in its existing raw state of plain text.
  • Project Gutenberg.

• Corpus marked up for formatting attributes e.g. page breaks, paragraphs, font sizes, etc..

• Corpus annotated with identifying information, such as title, author, genre, register, edition date etc..

• Corpus annotated with linguistic information:
  • Sentence boundary disambiguation. The simple "full_stop_space_capital_letter" approach works well for simple texts but is rather unreliable for texts with many proper names and abbreviations at the end of sentences such as newspaper articles. In these texts, for exemple, a full stop can act as the end of sentence or it can be part of an abbreviation, but when abbreviation is the last word in a sentence, the full stop plays a dual role.
  • Tokenization: lexical units are identified. For some tasks simply assuming that objects bounded with white spaces are tokens is enough, but this is certainly not true generally. For example, multi-words elements are sometimes seen as tokens. The tokeniser identifies words and multi-word units.
  • Lemmatisation: involves the reduction of the words in a corpus to their respective lexemes. Lemmatisation allows the researcher to extract and examine all the variants of a particular lexeme without having to input all the possible variants, and to produce frequency and distribution information for the lexeme.
  • POS tagging (grammatical tagging): aims at assigning to each lexical unit in the text a code indicating its part of speech. It is the most basic type of linguistic corpus annotation and forms an essential foundation for further forms of analysis.
  • Parsing (treebanking): involves the procedure of bringing basic morphosyntactic categories into high-level syntactic relationships with one another.
  • Semantic tagging:
    • Marking of word senses (sense resolution).
    • Marking of semantic relationships between items in the text, for example the agents or patients of particular actions.
    • Marking with semantic categories.
  • Discourse annotation: is the marking of the text with tags such as "apologies" e.g. sorry, excuse me; "greetings" e.g. hello; "politeness" e.g. please; "responses" e.g. really, that's right.
  • Anaphoric annotation: is the marking of pronoun reference. This task can only be carried out by human analysts.
    • XANADU (UCREL)

• Problem-oriented tagging: is the phenomenon whereby users will take a corpus, either already annotated, or unannotated, and add to it their own form of annotation, oriented particularly towards their own research goal.

• Phonetic transcription (only in spoken language corpora): this is a very time consuming task because it needs to be carried out by humans which have to be well skilled in the perception and transcription of speech sounds.

• Prosody annotation (only in spoken language corpora): prosody refers to all aspects of the sound system above the level of segmental sounds e.g. stress, intonation and rhythm. The annotations in prosodically annotated corpora typically follow widely accepted descriptive frameworks.

Corpus Analysis Tools

• LTG Helpdesk FAQ. These are some of the most frequent questions addressed to the Language Software Helpdesk at the Language Technology Group of Edinburgh University. The Helpdesk is intended as a general service to researchers and developers looking for information on a wide range of Natural Language software, tools, and resources. The FAQ information can also be accessed via a list of key indices.

• Michael Barlow's Corpus Linguistics website. The section of the page devoted to the different available taggers.

• BNC-Corpus Analysis Tools. The section on corpus analysis tools in the British National Corpus website on English language corpora and corpus resources.

• Linguistic Annotation.The LDC page describing tools and formats for creating and managing linguistic annotations. "Linguistic annotation" covers any descriptive or analytic notations applied to raw language data. The focus is on tools which have been widely used for constructing annotated linguistic databases, and on the formats commonly adopted by such tools and databases. There is also an Italian version of this page.

• Computer-assisted text analysis resources in WWW. The website is maintained by ZUMA, the Center for Survey Research and Methodology of GESIS, an infrastructural association which supplies fundamental social science services on a national and international level, both in terms of theory and practice. Especially useful for Content Analysis.

• LTG Software. The Language Technology Group of the University of Edinburgh makes available various software packages. For research purposes, these are often available for free to academic research groups and for a small fee to industrial R&D groups.

• Corpus Annotation. Corpus analysis software available from UCREL (University Centre for Computer Corpus Research on Language) of Lancaster University.

• Corpus Software. Corpus analysis software available from the Corpus Research Group of Birmingham University .

• UCL - Tools. The page of UCL (Université Catholique de Louvain) dealing with corpus analysis software.

• POS Taggers. An annotated list of POS Taggers in the corpus-based computational linguistics website maintained by Christopher Manning from SULTRY (Sydney University Language Technology Research Laboratory).

Corpus Encoding Standards

• TEI (Text Encoding Initiative). TEI is an international project to develop guidelines for the preparation and interchange of electronic texts for scholarly research, and to satisfy a broad range of uses by the language industries more generally. The TEI is sponsored by the Association for Computers and the Humanities (ACH), the Association for Computational Linguistics (ACL), and the Association for Literary and Linguistic Computing (ALLC).
  • TEI Lite.The document describes a manageable selection from the extensive set of SGML elements defined by the full TEI scheme. The italian version of this document can be found here.

• CES (Corpus Encoding Standard). This document is the first version of the CES, which is a part of the EAGLES Guidelines. The CES is designed to be optimally suited for use in language engineering research and applications, in order to serve as a widely accepted set of encoding standards for corpus-based work in natural language processing applications. The CES is an application of SGML compliant with the specifications of the TEI Guidelines.

• MULTEXT (Multilingual Text Tools and Corpora). Multext encompasses a series of projects whose goals are to develop standards and specifications for the encoding and processing of linguistic corpora, and to develop tools, corpora and linguistic resources embodying these standards. Multext is developing tools, corpora, and linguistic resources for a wide variety of languages. Where appropriate, work has been done in collaboration with larger umbrella standardization efforts, such as EAGLES and TEI.

Markup Languages: SGML/XML

• The SGML/XML Web Page. The SGML/XML Web Page, by Robin Cover, is a comprehensive online database containing reference information and software pertaining to the Standard Generalized Markup Language (SGML) and its subset, the Extensible Markup Language (XML). The Page is currently sponsored by OASIS (Organization for the Advancement of Structured Information Standards) and three OASIS Members: ISOGEN International Corp, Sun Microsystems, and webMethods.

• wdvl-XML. The Web Developer's Virtual Library page on XML.

• W3C-XML. The W3C (World Wide Web Consortium) XML recommendation.

• Microsoft-XML. A Microsoft introduction to XML, with examples and tutorials.

• XML.ORG. The XML industry portal sponsored by IBM, Oracle, Sun Microsystems and others.

• Free XML Tools. The page, maintained by Lars Marius Garshol from Oslo University, contains a detailed index of free XML Tools.

• W3C-XSL. The W3C XSL (Extensible StyleSheet Language) recommendation.

• Microsoft-XSL. A Microsoft introduction to XSL.

• XML Mailing list. Archives of XML-L@LISTSERV.HEANET.IE mailing list.

Italian Websites:

• XML-IT Mailing list. The website contains a list of the most important XML mailing lists, forum and newsgroups in English and in other languages.

• XML. The page, maintained by a student of the University of Venice, is a collection of (mainly italian) documents concerning XML and JDBC.

• Gruppo XML Italia Mailing list. Archives of XML@XML.IT, the mailing list of XML Italia group.

Corpus Search

• Concordances: is an alphabetical listing of the words contained in a text, given together with the contexts in which they appear. A concordance is particularly useful for studying a piece of literature when thinking in terms of a particular word, phrase or theme. It will show exactly how often a word occurs, or even if it does not occur and so can be extremely helpful in building up some idea of how different themes recur.
  • The Web Concordances

  Corpus Search Tools

  • UCREL Corpus Search Tools. The list of tools developed by UCREL(University Centre for Computer Corpus Research on Language) of Lancaster University.
  • Michael Barlow's Corpus Linguistics website. The section of the page devoted to different concordance programs.
  • Text Analysis Tools and Techniques. A list of concordance programs in the CTI (Computers in Teaching Initiative-Centre for Textual Studies) Guide to Digital Resources. CTI belongs to HCU (Humanities Computing Unit), a special service of OUCS (Oxford University Computing Services).
  • Euralex - Concordancing Programs. Euralex (European Association for Lexicography) list of concordancing programs.

• Content analysis: is a research tool used to determine the presence of certain words or concepts within texts or sets of texts. Researchers quantify and analyze the presence, meaning(s) and relationship(s) of such words and concepts, then make inferences about the messages within the text(s), the writer(s), the audience, and even the culture and time of which these are a part. In fact, computer content analysis provides another approach to measuring aspects of social structure, the basic assumption being that different social positions imply different social perspectives that are evident in language.
  • Software for content analysis
  • Harald Klein's content analysis page