TOBAR NA GAEDHILGE 1.6

• What the results may look like

We may begin by finding sentences containing a selected word, and we will do one example each from the Munster, Connacht and Scottish texts. Thereafter, we will draw our examples from the Ulster texts, which form by far the largest part of the material stored.

Figure 1: We looked for examples in the Munster texts of the word cábóg (a country person). We found two examples in Pádraig Ua Maoileoin, Na hAird Ó Thuaidh, and we show the second example here. Page and line reference is given to the published book.

The navigation panel (at upper right) allows us to move around the retrieved examples. The panel at the lower right allows the form in which the sentence is displayed to be modified. All the options are described under Figure 6 below.

Figure 2: We looked for examples in the Connacht texts of the sequence of words lúb gaoil (blood relationship), and we found two examples, both in Séamus Mag Uidhir, Fánaidheacht i gConndae Mhuigheo. We show the first sentence here.

Figure 3: We looked for examples in the Scottish texts of words beginning with càr. With a request like this for words matching a general pattern, clearly more than one word may match — we refer to this situation as matching a disjunction of words. The sentences for each matching word are presented (over all the relevant books) before presenting the sentences for the next matching word. Here, we show a sentence containing the matching word càraich (fix).

Besides viewing complete sentences as above, two other ways of viewing results  are provided, which are more compact for large quantities of text. Output from these methods is shown in the next two figures.

Figure 4: Uses of the word saoghal (life) shown as its frequency in the various Ulster texts.

Scrolling may be necessary to reveal the information for all the texts. If the request matched a disjunction of words, a menu option AthFhocal (next word) will proceed to the next matching word. A menu option Réidh (finished) leaves the displayed results and returns to the search screen.

Figure 5: A keyword-in-context index of the word-form athair (father) in Séamus 'ac Grianna, Thiar i dTír Chonaill. The first batch of 43 occurrences is shown.

The navigation panel at the upper right allows several options. First of all, we may move up and down through the concordance lines, which are in screenfuls of a size which depends on the size of the window (43 lines in this illustration). We may move down a screenful (Síos), up a screenful (Suas), to start of text (Suas go bárr), to end of text (Síos go bun). However, these options will not take us into another text, nor (if the request matched several word in a disjunction) will they take us into another matching word; they move only within the current text, for the current matching word.

Second, we may move, forwards only, through the entire collection: AthThlámán (next batch), AthLeabhar (next book), AthFhocal (next matching word, if the request matched several in disjunction), Réidh (finished). If we wish to examine all the examples from the beginning again, we need only choose Réidh and then click OK without altering our previous choices. This second set of options are provided with keyboard shortcuts, which may prove convenient for repeated use.

Cóipeáil (copy) copies the current display to a textfile, which by default, is called samplaí.txt and is placed in the My Documents folder, and the copied material is appended to it. Comhad Cóipeála (copy file) allows the name and location of the file to be changed, and also the mode from append to overwrite (but it will revert to appending after overwriting once).

Réidh (finished) leaves the displayed results and returns to the search screen.

Figure 6: Returning now to display by sentences, we give an example of the word oidhreógach (ice) from the Ulster texts. The example shown is from Seosamh 'ac Grianna, Pádraic Ó Conaire agus Aistí Eile.

The panel at the lower right allows the text of the sentence to be shown in a choice of ways: plain and uncorrected (Foillsighthe); plain but corrected (Lom); including mark-up (Marcáilte); or as a list of the words by which it is indexed (Innéacsáilte). The Innéacsáilte option allows you to see which index terms will fetch this sentence. This will allow you to investigate our failure to achieve consistent indexing over this body of text! The panel also allows the sentence to requested to appear in other languages, when this is possible (see later) — any language with its name shown in in italics is unavailable for this sentence.

The navigation panel at the upper right allows several options.

First of all, we may move around the examples. We may move down a sentence (Síos), up a sentence (Suas), to start of text (Suas go bárr), to end of text (Síos go bun). However, these options will not take us into another text, nor (if the request matched several words in a disjunction) will they take us into another matching word; they move only within the current text, for the current matching word.

Second, we may move, forwards only, through the whole collection: AthAbairt (next sentence), AthLeabhar (next book), AthFhocal (next matching word, if the request matched several in disjunction), Réidh (finished). If we wish to examine all the examples from the beginning again, we need only choose Réidh and then click OK without altering our previous choices. This second set of options are provided with keyboard shortcuts, which may prove convenient for repeated use.

Cóipeáil (copy) copies the current display to a textfile, which by default, is called samplaí.txt and is placed in the My Documents folder, and the copied material is appended to it. Comhad Cóipeála (copy file) allows the name and location of the file to be changed, and also the mode from append to overwrite (but it will revert to appending after the first copy). The darker elongated panel shown above is the result of clicking Comhad Cóipeála.

Réidh (finished) leaves the displayed results and returns to the search screen.

If you accidentally lose the navigation panel from the screen while examining the results, a reminder labelled Preab-liosta! will appear from underneath it.  Clicking on this reminder will recover the navigation panel.

Multi-word retrievals may require the creation of workfiles. These will be placed in the temporary workspace folder, if possible. In the unlikely event that this is not possible, you will be prompted for the name of a folder to hold workfiles; you could, for example, place them on a memory stick.

• What texts can be searched

Gaelic is found in several slightly different forms, and the texts are organized into collections to reflect this and to keep each collection fairly homogeneous in language.  The five collections supplied are (as of 2012/09/24):

• Ulaidh (Ulster)           Index: Gaedhilg. 17 authors;  56 books;  48,212 word-forms;  3,711,513 word-tokens
• Connachta                 Index: Gaedhilge.  3 authors;   3 books;  10,066 word-forms;    139,184 word-tokens
• Mumhain (Munster)  Index: Gaolainn.   2 authors;   2 books;   6,755 word-forms;    101,386 word-tokens
• Alba (Scotland)          Index: Gàidhlig.    4 authors;   4 books;    7,407 word-forms;    99,047 word-tokens
• Oirthear (Eastern)     Index: Gaodhlag.   1 region;    1 book;    3,446 word-forms;   48,571 word-tokens

At the present stage of development, the Ulaidh collection is much larger than any of the others.

A different division into collections would, of course, be possible.

The identities of the texts in all collections are listed in full below.

Searching may be restricted to any chosen subset of the texts of a collection, by deselecting temporarily individual texts or authors.

Each collection has a pre-compiled index associated with it, made from the words found in the relevant books, and in which requests for words are looked up. The statistics just given for the collections refer to these indexes — indexes of words found in the Gaelic texts. The Ulaidh collection has several additional indexes — Béarla (English) and Fraincis (French) indexes of words made from English and French translations or originals of some of the books in the Ulaidh collection. Further, from version 1.4, a rough and ready lemmatization has been applied to the English and French texts, so that the Béarla and Francis indexes are offered in terms of lemmata (lemma-types) as well as in terms of foirmeacha (word-forms). (There is also, from version 1.6, a Gearmáinis (German) index, which is based on one translation, and which should be regarded as purely experimental.)

The statistics for the additional indexes to the Ulaidh collection are as follows, at 2012/09/24:

• Ulaidh (Ulster)           Index: Béarla (foirmeacha). 6 Gaelic writers; 14 books;     35,454 word-forms;    1,425,921 word-tokens
•                                   Index: Béarla (lemmata).  6 Gaelic writers;   14 books;   28,238 lemma-forms;    1,424,830 lemma-tokens
•                                   Index: Fraincis (foirmeacha). 3 Gaelic writers;  6 books;    28,176 word-forms;    616,692 word-tokens
•                                   Index: Fraincis (lemmata).  3 Gaelic writers;   6 books;    14,017 lemma-forms;    616,331 lemma-tokens
•                                   Index: Gearmáinis (foirmeacha). 1 Gaelic writer;  1 book;    8,703 word-forms;    52,073 word-tokens
•                                   Index: Gearmáinis (lemmata).  1 Gaelic writer;   1 book;    6,132 lemma-forms;    52,015 lemma-tokens

(There are several reasons for the discrepancies between the word-token counts and the lemma-token counts for the same books. Some words, such as English won't or can't count as one word-token but as two lemma-tokens: will not, can not. Also, foreign words are excluded from lemma indexes.)

Much more detail will be given later about what may be found in the indexes. But it is well to say at the outset that a Gaelic (for example) index means an index to everything that is found in the Gaelic versions of books. It does not mean an index containing exclusively Gaelic words — rather, each index is, to an extent, multi-lingual.  The same applies to English and French indexes — but, as we have just stated, lemma indexes are made more nearly monolingual than word indexes.

Figure 7: This is the program's opening screen, and the first task is to choose the desired collection and index.

The program should show a list of the available collections, as in blue above — together with some statistics of the highlighted collection (below) and a pick list of its index (on the right). If there are no collections listed, you may be in the wrong folder, and you can browse (using Cuirtear Lorg) to a different folder. The indexes available to the highlighted collection are shown in brown. When you have marked the desired collection and chosen your index to it, click on Isteach to enter the collection/index.

Amach is to exit the program.

Treoir is for help. The help file supplied with Tobar na Gaedhilge is in Windows Help format (.hlp). Support for Windows Help File format was discontinued by Microsoft in Windows Vista and later. In order to use Windows Help files in these versions of Windows, download and install the Microsoft upgrade of WinHlp32.exe for your version of Windows from http://support.microsoft.com/kb/917607

• Requesting words

When a collection has been selected, and Isteach clicked, the display changes to that shown in the next figure, which allows you to type words, among other things.

Figure 8: Requesting words.

Before entering our own words, however, first notice that this screen allows you to go back and change to a different collection and index, by using the Athruigh button on the Cnuasacht panel. And also, that you may see which authors and books are included in the current collection and index by using the Athruigh button on the Leabharthaí panel, and you may choose to select temporarily a subset of those books. (Note that selecting a subset of a collection is not reflected in the pick-list of words, which remains that for the whole collection.)

From the Radharc panel, you may choose your display mode for the results: frequencies (Minicidheacht), a keyword-in-context concordance (KWIC) or sentences (Abairteacha). Samples of each form of results have already been shown above.

And now we come to the Focal panel, where the desired word or words may be typed into the box provided, or may be inserted there by double-clicking them on the pick list, which is a displayed segment of the collection's index. The pick list accommodates itself to the existing contents of the box, as a guide to what words are available.

As an alternative to typing it into the box, a search word may also be chosen by double-clicking it on the pick list.  The word will be appended to anything already in the box (and not highlighted).

In the word-form box you may put:
• a word, such as oidhreógach or saoghal or athair, as used in our previous examples
• two or more words occurring together, either consecutively or within the same sentence (eg. lúb gaoil)
• any word may contain a wild-card (*), that is, an asterisk which matches any number of letters, including none.  For example, all words with a particular stem may be sought (eg. beir*), or all words with a particular termination (eg. *stin).

If the words box contains more than one word (ie. there is a space within it), you are asked to choose between seeking the words directly adjacent and in the given order; or within the same sentence in any order.

It is even possible to give one or more of the words as simply the asterisk (*), which matches any word; the search is then assumed to be a consecutive one. (But avoid giving * as the final word, as the search will be slow.) As we will see below (under demutation) a hyphen is, in most circumstances, counted as a separate word, so search for sean-bhean as three words: sean - bean (as well as sean bean and seanbhean to cover any unhyphenated instances).

You may tick the Gan beinn ar an tsíneadh fhada checkbox if you want to include words which differ from that requested only by the presence or absence of an accent, eg. comhradh with this checkbox ticked will match comhrádh, cómhradh and cómhrádh as well. Ticking this box also includes words which differ from that requested by the addition of a hyphen, apostrophe or period — though most Gaelic words containing apostrophes or hyphens are indexed as two or more separate words anyway, as explained under decompounding below.

Actually, the function of this box would be more correctly described as to include among the retrieved words any which differ from the requested word only (1) by the presence or absence of any accent, not just the acute (2) by case differences, or (3) by the intrusion of non-alphabetic non-punctuation characters such as £, %, period (indicating abbreviation), apostrophe (indicating elision), hyphen (indicating anonymisation), etc. But indexes already have some of these distinctions removed, eg. word indexes are lowercased (except German); and all indexes have punctuation removed; so the main effect of this option is to neutralize differences of accents, and (for lemma indexes, see later) differences of case.

Word indexes, except for German, are always reduced to lower-case; and any letters you enter in the search word will also be so reduced (for lemmata, see later). For the German word-index, the initial letter of both stored and requested forms is exempted from lower-casing.

To type accented vowels, use your normal method of doing this under Windows. For information about keying accented letters under Windows, look here, or see the section "Keyboards layouts" near the end of this file. (But you will not require the support for dotted consonants offered by these keyboards, as lenition is always indicated by suffixing the letter h in Tobar na Gaedhilge.)

When all this is complete, you may click the OK button to produce the results.

Further hints on the selection of search words will come in the next section of this document.

• More about what to search for

Here are some pointers regarding what kinds of words are worth requesting.

When a word is requested, it is matched against a pre-compiled index of words from the chosen collection.  For Gaelic, this index consists of words which are aggregated in a number of ways to increase coverage:

• lowercased: the words in the index have been converted to lowercase by replacing any capital letters by small letters; this even applies to proper names. Any capital letters you include in your request will also be so converted. Keep an eye on the scrolling alphabetic list for guidance on what words are available.

• decliticised: common enclitics, such as d' in d'ól, or 's in 'seadh, or -sa in agam-sa, are treated as separate words in the index (d' + ól; 's + eadh; agam + - + sa), and should also be detached in your request. Enclitics are normally signalled in running text by a hyphen or an apostrophe. But when there is no overt signal in similar cases (eg. agamsa, seadh), the splitting in the index will have been performed manually and is unlikely to have been exhaustive.

A number of common contracted words have been indexed under their parts, e.g. 'na (from ina) under ' and a; 'na (from chun an) under 'n and a'; 'na (from chun na) under ' and na; ab (from a ba) under a and b, or under a and b'; gurab under gur and a and b; and many other similar cases. This aspect is to be made more rigorous.

• decompounded: very few words containing a hyphen have been admitted to the indexes — a list of these can be obtained by searching for *-*. Rather, most hyphenated words have been treated atomistically in the indexes, and are found by seeking their parts, including the hyphen, eg. leith-phighinn by seeking the three items leith and - and pighinn, with checking of the "consecutive" option ("Díreach i ndiaidh a chéile").

• demutated: initial mutations are removed from words in the index; so, for example, fear, fhear and bhfear are all indexed as fear, while t-olc, n-olc and holc are all indexed as olc—but, where the mutation is permanent, it is retained, e.g. chugam, thart (in one of its senses), (go) dtí. You may have noticed the benefits of demutation and decliticisation in our athair example above. An initial mutation does not leave any trace in the index; and this is also true of any hyphen which is nothing but part of an initial mutation. When typing words of Gaelic to be searched for, remember to remove initial mutations, unless they are a permanent part of the word. Removal of initial mutations may seem counter-intuitive when requesting a sequence of words (eg. ár athair), but it is nonetheless required.

But the words in a Gaelic index are not lemmatized, i.e. terminally inflected forms, such as fear, fir, feara, must be searched for separately—although the wild card may often be used to advantage to retrieve the several related forms.

You may choose (by ticking Gan beinn ar an tsíneadh fhada) to make the search insensitive to accents. A suitable example might be searching together for words like comhartha and cómhartha. Both words are help separately in the index, and with this option selected a request for either of them will fetch both.

Finally, note that, wherever the text is clearly in error, we may silently correct the indexed form of a word, but we never correct the text itself, that is, the original text form—right or wrong—will be displayed in the contexts (e.g. the misprint comhhartha is indexed as comhartha, but remains comhhartha in contexts).

For English, French and German indexes, broadly similar considerations apply. Word-indexes are lower-cased, except for German, where the initial letter is exempted from lower-casing. Enclitics are separated (eg. English 's, 've, n't, French l', m', qu'). Hyphenated words are generally decomponded, eg. French garde-robe; but this policy has not been consistently applied to English, where eg. decompounded water-tight is found as well as unitary water-tight and watertight. A decision on decompounding the German indexes (in the absence of hyphens) has still to be made. (For lemmatization in the English, French and German indexes, see the section below titled "Lemmatization".)

• More languages!

It happens that a minority of the texts in the Ulaidh collection also exist in translated (or original) English forms, and some too in French. When a Gaelic sentence is being viewed, then, it is possible to request the display of the English and/or French equivalent sentences too, and they will be shown if they are available. You may have noticed buttons, on the sentence-level displays above, labelled Béarla and Fraincis — this is their purpose. When a translation is not available for the sentence being displayed, the name of the unavailable language is italicised.

Figure 9: Sentences containing the word creafadaigh (shaking); the first of two examples from Seosamh 'ac Grianna, Seideán Bruithne/Amy Foster. English and French translations are available and are shown.

In this and the following examples, remember that the sentence may be shown in a choice of ways in the first language: plain (Lom); including mark-up (Marcáilte); as a list of the word-forms by which it is indexed (Innéacsáilte); or plain and uncorrected exactly as printed (Foillsighte). In other languages it is shown plain.

Likewise, when we use the Béarla or Francis word indexes to the Ulaidh collection, we may view Gaelic sentences which serve to translate a particular word of English or French. Like the Gaelic word-form indexes, the Béarla and Fraincis word indexes are lowercased and decliticised, but unlike Gaelic, both lemmatized and unlemmatized indexes are now available — see below. For the Gearmáinis word index — still in development — the initial letter is exempted from lowercasing, and some degree of decompounding is under consideration.

Figure 10: Search of the English index of the Ulaidh collection for the word bunch. An example is shown from Ben-Hur, and the English and Gaelic and French of the sentence is displayed.

Figure 11: Search of the French index of the Ulaidh collection for the word accroché. An example is shown from Iascaire Inse Tuile, and the French and Gaelic of the sentence is displayed.

• Lemmatization

Alongside the indexes of words, we now offer lemmatized Béarla and Fraincis indexes to parts of the Ulaidh collection. (Lemmatized Irish indexes are not yet in sight.) Thus, a request for the English lemma man will match the words man or men; while a request for the French lemma homme will match the words homme or hommes.

Figure 12: A KWIC list of the examples of the lemma listen in Gadaidheacht le Láimh Láidir, according to our English lemmatized index. The corresponding Gaelic material may be inspected, one example at a time, in the sentence display mode.

Figure 13: A KWIC list of the examples of the lemma abandonner (to abandon) in Ben-Hur, according to our lemmatized French index. The corresponding Gaelic material may be inspected, one example at a time, in the sentence display mode.

It is important to understand, however, that our lemmatization of English and French is severely limited. It has been performed automatically, using the Stuttgart Tree Tagger and, as with all statistical operations, a percentage of errors is inevitable, despite much manual checking.

In our lemmatized indexes, an initial capital has been retained in some words, mostly names. Therefore it will make a difference to the results whether the letters you type in your request to a lemmatized index are small letters or capitals. Keep an eye on the scrolling alphabetic list for guidance on what lemmas are available and when you should use a capital letter.

Lemmatized indexes exclude foreign words, ie. words not in the main language of the index.

If you wish to include in your results lemmas which differ from your request only by an accent, put a check mark on Gan beinn ar an tsíneadh fhada just as with word indexes.

When displaying the results from a lemmatized index, the Innéacsáilte display option shows the sentence in the first language as a list of the lemmas with which it is indexed.

A problem arises in retrieving only the desired sense of an ambiguous English lemma such as pack or stamp or well or lie or bound or back in English, or of an ambiguous French lemma such as pas or tendre or vague or fin. At version 1.4, all senses of such a lemma were simply displayed together. At version 1.5, we have tried to separate the senses by part of speech, using four broad categories of N (noun), V (verb), J (adjective) and Z (other). Thus for example a request for English lemma well will be asked to choose between N, V and Z; a request for French lemma vague will be asked to choose between J and N. This may help in many cases, but not in others; for the English lemma lie, for example, a more useful division would be into recline and untruth, rather then into noun and verb. Further changes in this area may be expected in future versions. The discrimination by parts of speech has somewhat increased the lemma counts, compared with version 1.4.

• Translation equivalents

A related innovation is the calculation of translation equivalents. Given a word in the source language, this consists of a listing of the relatively most common words in the corresponding segments of the target language. This will clearly be more effective using lemmas than using words, so it is offered only with lemmatized English or lemmatized French as the source index. At present the target produced is a list of words (Gaelic, English or French), though again lemmas would be preferable but are not yet available. This technique has potential, but is limited at present by the amount of text available (English/Gaelic: about 1.25 million words; French/Gaelic: about 350,000 words; English/French: about 300,000 words). Results with the present amounts of text will vary from useful to comical.

Where a lemmatized source index (English or French) is in use, translation equivalents may be chosen as a fourth output display mode, named Freagar-fhocla. The calculation, for a selected source-language lemma, may take a few moments. The resulting display is a list of target-language words, each accompanied by a score, and sorted on these scores (the user may have it re-sorted alphabetically on the words themselves). The scores — which are not raw word counts — may range from 99,999,999 down to 100,000. They measure how many times as common the target-language word is in the neighbourhood of the source-language lemma than in the target language corpus on average.

Figure 14: Search for Gaelic words collocating with the English lemma child, in the Ulster texts.

The chosen source-language lemma defines a set of sentences in the source-language corpus — those sentences in which it occurs — and a corresponding set of sentences in the target-language corpus — those sentences which translate them. This "select part" of the target-language corpus is studied, looking for words (freagar-fhocla, word-equivalents) which are more frequent there than in the target-language corpus on average.

If the source-language word is uncommon (read: selects less than one-thousandth part of the source language corpus), a warning is issued that the results may not be statistically useful, but no impediment is placed on calculating them.

If a target-language word turns out to be equally frequent in the select part as on average, it is given a score of 100,000; if it turns out to be twice as frequent in the select part as on average, it is assigned 200,000; and so on. Words less frequent in the select part than on average are discarded as uninteresting, so that 100,000 is the minimum score among those retained. At the other end of the range, the score 99,999,999 is assigned to any word which is 100 times or more as common in the select part as on average.

Even if a word falls within the range 10000..99999999, it is still omitted from the displayed list if its absolute frequency is small. This is intended to overcome "accidental" collocations, which will disappear naturally as more text is added, but may mask more significant data while they remain. A suitable empirical lower cutoff for absolute frequency of a word-equivalent is found to be the square root of one-tenth of the frequency of the source-language word.

Results are still poor enough with the amount of text available, but will improve as the quantity increases. Even at the present time, however, it may be of interest to input English lemmas from the following list, and to compare the results with the content of existing English–Irish dictionaries, noting what is found in the dictionaries but absent in the corpus, as well as what relevant equivalents are found in the corpus but not in the dictionaries: smoke, minute, also, yet, dog, ice, bee, garden, help, interest, gravel, cave, busy, cell, kitchen, open.