Überarbeitung der Wortliste

Moderator: Moderatoren

Didier
Posts: 3
Joined: Fri Nov 04 2011 14:10

Post by Didier »

@Eryniel "(...), but it does look rather familiar...":

In all honesty, I wonder what you mean by "looking familiar", but as our favorite author once wrote "Both rings are round and that's where the comparison ends". Admittedly though, there's one pretty nice design idea retained from Maike's Ithildin application: using small country flags to switch between languages - cool and simple, if one puts the good flags (I feel ashamed again). But I don't want to be off-topic here, so don't hesitate sending me a PM if you want more details.

@Roman:

My announce on Elfling was delayed by a week or so through moderation, but yesterday evening I just received a mail from Lőrinczi Gábor who is building a Sindarin-Hungarian dictionary. He had already done a simple word-list back in 2006, which may still be found on its website, but he sent me a sample page of his new dictionary on which he worked during 3 years, and it looks very very impressive. Anyhow, and that's my point posting here, he also kindly sent me a list of ~50 entries missing or differing in my dictionary. For now, I just had a quick glance at it, and except just a few items you had already noticed here (dufui, harwed... etc. a few posts above) and that I have therefore already fixed too, there remains a good list of items that neither you nor I seem to have. I don't know exactly how to proceed here, but would you be interested in getting the list too? If so, I'll share it with you and perhaps we can have a discussion?

Didier.
P.S. Regarding Ladon's Breath: since you are what I would call "advanced users", did you have a look at the editor in the latest revision? (again, PM me, if this is felt off-topic for this forum.)
User avatar
Roman
Admin
Posts: 2049
Joined: Tue Mar 13 2007 13:27
Contact:

Post by Roman »

For now, I just had a quick glance at it, and except just a few items you had already noticed here (dufui, harwed... etc. a few posts above) and that I have therefore already fixed too, there remains a good list of items that neither you nor I seem to have. I don't know exactly how to proceed here, but would you be interested in getting the list too? If so, I'll share it with you and perhaps we can have a discussion?
Absolutely! And he is of course welcome to collaborate with us if he wants Hungarian to be added to Lúthien's application.
P.S. Regarding Ladon's Breath: since you are what I would call "advanced users", did you have a look at the editor in the latest revision? (again, PM me, if this is felt off-topic for this forum.)
I think it's right on topic. When I tried it out, the changes I in Ladon's Heads weren't carried over into Ladon's Breath and were reverted after a restart of Firefox. Also, it's kind of annoying that the window closes after the update of one entry. I like it that you can actually edit the XML entry, however - although syntax highlighting would be nice, of course. :-) Also, I found no way of adding new entries..
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

Ah hello Didier, my, it's a small world isn't it :)

- - - - - -

For those interested, I finished the database this weekend. I did some minor changes on the data model as well: some of those things only become apparent if you are actually stuffing the data in it and something falls out, or something like that.
Anyhow, it is all in there now - including all the edited entries and the "wordlist_komm" data.

If anyone wishes to have a look at it, let me know via a PM message, and I can give you either a MySQL dump or a set of SQL scripts.

I'll now create a first version of the viewer app so that the contents can be easily read and searched.

Here's the final (well, quite absolutely final) data model:


Image
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

... and I can give you either a MySQL dump or a set of SQL scripts.
oh, and of course as a SQLite db file (size ~ 2Mb for the whole thing including indexes). This is the format that I want to use for the application.
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

there's an updated version of the database available.

While working on the application it turned out that it greatly helps performance to change some foreign key relationships, so I changed that.

You can pm me for a copy.

Image

It is now really fast: even a select over all tables including a wildcard match with the wildcard at the front (*abc) such as this one:

select distinct e1.gloss, e2.gloss, d.name, m.text
from ENTRY e1, ENTRY e2, DATACLASS d, METADATA m, TRANSLATION t, TR_MD tm
where e1.id = t.sind_entry_id
and e2.id = t.mod_entry_id
and e2.language_id = 100
and e2.gloss like '%nsch%'
and tm.translation_id = t.id
and tm.metadata_id = m.id
and m.dataclass_id = d.id

(yielding results such as 'Mensch', 'Wunsch', 'Menschenvolk' 'Gemeinschaft' etcetera)

only takes 37 millliseconds :)
This is a decisive thing if you use the app to search for, say, rhyme words in creating poetry - if it takes a second to give you a new result set that's positively annoying.
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

Next step .. it's a bit of a technical story for now, but I'll post it in case anyone is interested (and in order to keep you in the loop)

Now the database is complete, or in any case complete enough for the time being, the next thing is to build something (an application) around it to make it usable.
For this, it is necessary to make the program code "talk to the database", which is a matter of installing the right database drivers. This is usually straightforward, but for this combination of java (for the application) + sqlite (database) the driver that I had found at http://www.zentus.com/sqlitejdbc/ was slightly incapacitated.
Luckily, it turned out that someone has just written a shiny new one (http://www.xerial.org/trac/Xerial/wiki/SQLiteJDBC) that worked like a charm.

So now I can hook the code up to the DB:

Image
(this is a panel showing just the ENTRY table)

This is, admittedly, not a very useful application. But to me this is like a milestone: getting that database complete and into the right form was by large the hardest part. And now the broken jdbc driver hurdle has been taken, it starts to feel like a downhill bicycle ride :)
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

hi,

I think this approach of using SQLite is working really well.
I've tried to find the best (fastest) way to load the data into the application and what I have now is really fast: here's a little screen movie of the list of English entries, which is filtered in real time by the letters that you type in the search area above it. The idea is (just as with the older version that I made last year) that when you click on an entry in the (filtered) list, the translation is displayed in a panel next to it (not displayed yet in this screen movie)

I don't know if it is possible to embed a youtube movie on this forum, but this is the link: http://youtu.be/oVnlOCBhyws
Faerphin
Posts: 59
Joined: Sun Jul 04 2010 13:45

Post by Faerphin »

First I want to say that I love what you're doing, and that I very much look forward to the final result. What you are working on is actually a Sindarin-dream come true.

Secondly I would like to ask a complete laymans question: Would it be possible to add ones own annotations to words (or even new words entirely), either directly through the interface, or via some small simple other program ?
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

Faerphin wrote:First I want to say that I love what you're doing, and that I very much look forward to the final result. What you are working on is actually a Sindarin-dream come true.
Thank you! :D That's great to hear!

Actually: today I got a working application that displays the translations. It's still a bit crude, just displays the English entries and does not yet give the metadata, but it is working.
(in the final version you can search for either English, German or Sindarin entries - I don't have the French entries for this dataset atm, but of course we can add those as well)

Here's a short screen movie demonstrating the prototype application:


http://youtu.be/VZF0_0iDJh0

Faerphin wrote:Secondly I would like to ask a complete laymans question: Would it be possible to add ones own annotations to words (or even new words entirely), either directly through the interface, or via some small simple other program ?
That's a very good question. I was discussing this point with Eryniel last week, and I think that there has also been some discussion about this point previously in this thread. As far as I am concerned - but this is not for me to decide because it is and remains a group project - it would be indeed good to be able to add your own annotations and indeed also entries to your own copy of the data-set. Technically spoken it isn't a big deal.

There has been some discussion about maintaining a reference dataset on a central place, perhaps that some people here would be willing to act as a sort of editorial team that decides whether or not some new (reconstructed) entries should make it to the reference dataset. But you can always change your own private dataset, and maybe we could build in an option to either "reset" the dataset to the reference set, or possibly merge it?

I already have thought about how to implement that "change" functionality. It's probably going to be an "edit" function within the application, but the details will have to crystalize out yet a bit :)
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

now I got the whole set of data associated with a particular (English) entry: translation(s), Sindarin entry, possible child entries (alternatives, plurals), special case mutations, pronunciation and metadata (word-sort, references, etc.)

see: http://youtu.be/z6C23Qpfans

Next: switching between German and English entries

I also solved something with the SQLite database that was possibly troublesome: I could not get the jdbc driver to accept a relative path. The problem with that could be that while the application could work on my own computer, it would not on any other's, because you would not have the same disk / folder layout. But someone pointed me in the right direction at
http://stackoverflow.com/questions/8202 ... e-url-path and it seems to work fine now.
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

Image

8)
User avatar
Roman
Admin
Posts: 2049
Joined: Tue Mar 13 2007 13:27
Contact:

Post by Roman »

Secondly I would like to ask a complete laymans question: Would it be possible to add ones own annotations to words (or even new words entirely), either directly through the interface, or via some small simple other program ?
The idea is that everything should be editable, so that everyone can contribute to the dictionary. When a new publication comes out, anyone who has the time can input those words. Also, if mistakes are spotted, they can be corrected quickly this way. Apart from that, it increases the fitness of the dictionary, so that should it be the case that the people involved in the work now cannot do it anymore at some time in the future, others may take over easily.
This requires a central database to be maintained somewhere, however, so that the others will just have to get the newest version. I'm not sure yet how to organize this.
I guess one could add a 'private column', i.e. a place for personal comments for the words which will not be overwritten in the update, if this is desired. But the idea actually is that if you feel an annotation is necessary for a word, then you add this annotation into the 'official version' for everyone to benefit from.


Btw, I have a nice silme icon to offer (in white on green and black on grey) which can pass as the Sindarin flag. ;-)
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

sorry for the posting spree, but if things start rolling, they start rolling ...

now it is possible to search in either German and English, and to switch between those languages (I still have to arrange things so that the selection in either language will persist if you switch to and fro). Also, the application is now embedded in a proper framework which allows for enabling of menu items, as is visible on the screen movie (though there is nothing in there yet).

Next: searching and switching to and fro from "modern" (German / English) languages and Sindarin. I can also add the possibility to display the results in more than one one language.

I'm actually quite happy that I pondered the data model so thoroughly, because it now pays off in the ease of use: as you can see in the screen movie, the German and English metadata labels such as "verb" and "noun" are all automatically switched, because all these data are in the object graph that is selected from the original search entry (which is coupled to a language). In the previous incarnation of this application last year, I still had to set all this on several places, which was much more difficult to keep synced (and possibly also negatively effecting the performance).

Here's the most recent screencast: http://youtu.be/d8R1sf7yYEI
User avatar
Lúthien Meliel
Posts: 79
Joined: Thu Jun 26 2008 21:58

Post by Lúthien Meliel »

Roman wrote:Btw, I have a nice silme icon to offer (in white on green and black on grey) which can pass as the Sindarin flag. ;-)
Oh, I'd love to use that. I have improvised something (just a tiny star in a dark blue field) last year, but that was just, well, improvised.

-- As for the annotations: I can imagine that people would also feel a need to just use their private annotations? I was thinking along the lines of adding a "private comment" column as well - there is already a "comment" column at present.

But this is indeed a complex issue .. we need to think this through really well.

@ Roman: do you think that it would be possible to form a little group (say, two or three persons) that would want to act as editors of the central word / dataset?
Roman wrote: The idea is that everything should be editable, so that everyone can contribute to the dictionary. When a new publication comes out, anyone who has the time can input those words. Also, if mistakes are spotted, they can be corrected quickly this way. Apart from that, it increases the fitness of the dictionary, so that should it be the case that the people involved in the work now cannot do it anymore at some time in the future, others may take over easily.
I also love the idea of "everything should be editable" - but my experience with collaborative environments, even when using these advanced version control systems like SVN or what have you, is that this can quickly become a nightmare of entangled changes that no-one can ever sort out anymore.
Therefore I don't think that it is doable to make things so that "every user can upload any change to the central database" without any form of moderation. Such a system would also be very vulnerable to both vandalism and errors.

I think that it IS doable to make it so that any user can change their own database, add words and annotations, and that they can then send those changes to the "editors", who can then decide to add those changes to the central database or not.
User avatar
Roman
Admin
Posts: 2049
Joined: Tue Mar 13 2007 13:27
Contact:

Post by Roman »

Therefore I don't think that it is doable to make things so that "every user can upload any change to the central database" without any form of moderation. Such a system would also be very vulnerable to both vandalism and errors.
I think that it IS doable to make it so that any user can change their own database, add words and annotations, and that they can then send those changes to the "editors", who can then decide to add those changes to the central database or not.
Yes, some of the problems with Wikipedia come from that (for example, at one point the German article on energy was flooded by esoterics who insisted that their views were just as good and overwhelmed the physicists by sheer numbers), but I don't like the concept of an 'approval team' at all, as it tends to lead to the formation of cliques who gang up against any outsiders. For this reason, we actually have rotary moderators on this forum, each half a year, and every user can volunteer.
I would like to see something along the lines of a forum, i.e. being able to register and make changes right away, but with a possibility to be banned if vandalism is committed. But maybe the others should also say what they think.
Post Reply