Lexicon.dic (Redevelopment)

[Tethys]

TC and I have been talking about ways to condense and consolidate in BotF. One of the most overlooked areas is the lexicon.dic

Many entries are duplicates, some are differing only slightly and can be merged. We have a rough estimate of about 150 lexicon entries that can be freed up to use in projects. This is on the high side. Realistically we may want to keep some diversity (most of the entries would come from Race Descriptions for the 5 majors = 35x4 = 140, and then some here and there). This section may be updated by anyone, but please limit posts to workable code or associated lexicon locations. Irrelevant posts or erroneous info may be removed

Here is an example of a (working) patch that will change duplicate lexicon entry "Echo Papa 607", index 1389 (56Dh) and instead use lexicon entry "Echo Papa 607", index 257 (101h)

Code: Select all

NAME: EP607 Duplicate Lexicon Patch
AUTHOR: Tethys/thunderchero
DESC: This patch will adjust a duplicate Echo Papa 607 lexicon index to use a single index (257), freeing up lexicon entry (1389).
DESC: 
DESC: patch file name: EP607_Duplicate_Lex.patch
URL: https://www.armadafleetcommand.com/onscreen/botf/
TAG: 

# >>  = current value

# <<  = new value

>> 0x04f1fc B8 6D 05 00 00

<< 0x04f1fc B8 01 01 00 00

Unused Lexicon Entry List:

Code: Select all

1350     >     They
1351     >     They break trade treaty

[Tethys]

Here I will begin documenting the unused lexicon entries I find. Others are welcome to share info, but please verify the entry is not present in trek.exe
I will update the list as info is added.

I have just identified 2 unused entry

1350 and 1351

[Flocke]

Eh, I am about to update UE on what I found. I recently got side tracked on the lexicon.dic and found it worth to not map the vanilla entries by number any longer but by some readable identifiers. lol
not sure what list there exist already but I ran the app and compared screens and mapped most of the labels and lots of other entries and grouped the entries that belong together but are spread throught the many entries
I'm mostly done with it and will upload if not today then on weekend
but I'm happy to merge what you have already

edit: actually I was not looking for unused entries, but the other way around, to give each entry some proper label. So it's worth to keep searching for unused entries and match with my list lateron

[Tethys]

The way I identified was to find a questionable looking entry and search for the entry in IDA for example entry 1350 I would search text string 546h for a nearby Lexicon Term call, and I found none for those two entry. But Thunderchero advised me that some lexicon term are identified by index pointer? So care should be taken in this regard as well.

[Flocke]

Many of those index values are computed for sure. A simple search for matching values doesn't help here. That's already cause some text seems to be split to multiple values, and the compiler likely compiled it to relative refences of one entry to another. The best chance you have is to look for key variable references of the lexicon and check that you found all of the text values used ingame.

[Spocks-cuddly-tribble]

There is many entries for unused (e.g. alien artifacts, transferred dilithium, food, energy) and semi implemented (e.g. tactical cloak) features. Some of them are referenced in my modding posts (you can search them with the keyword 'lexicon').

Not sure how you guys want to deal with those entries.

[Tethys]

Not sure how you guys want to deal with those entries.

I doubt we can remove the entries from the dic without having to alter 90% lexicon indexes in trek.exe

So I'd assume a list of indexes be created (first post) of known unused indexes.

[Tethys]

I cracked open lexicon.dic in the hex editor and first of all: its weird. There is some table at the end which I don't really understand well. The values do not match potential lexicon entry indexes. I have tried every conceivable combination of value types, 2 byte, 4 byte, 4 and 8 byte floats, little and big endian.. there are just no ID's and its weird... although I did only test the very last entry (1459)

Testing is being done on a vanilla lexicon.dic with only one or 2 entry edited. Although, this may throw off values if UE has any hand (which it does obviously)

First 2 bytes equals DOUBLE (2919) the lexicon list entry numbers (1459) plus 1. I am about to do some testing and see if an additional lexicon entry can be added by adding provisions for 2 slots.

My assumption is that lexicon.dic counts the entry itself as a slot (lexicon index ID), as well as the text associated with the entry. I will report back shortly.

Edit: offset 10b6b (B4 05) total lexicon entry ID plus one
offset 13905 (33 0B 01 00) [68403 decimal] + offset -redacted info (its the next 4 bytes)- (34 00 00 00) [52 decimal] = size of lexicon index block from 0x00 [68455]

this should be enough information to at least attempt to add a new entry and see if it can show up in UE

I did not have success adding entry, there is probably a trek.exe reference to the entry ID count which also need to be adjusted.

[Flocke]

To understand the lexicon.dic file, best have a look at the UE source code.
Here I noted what's going on there in ue.edit.res.stbof.dic.Lexicon.java:

ue_lexicon_load.png (125.79 KiB) Viewed 2121 times

https://gitlab.com/stbotf/ultimate-edit ... xicon.java

So there is one unknown variable that however likely does not matter.
I will have a look whether I can add some simple add/remove buttons to UE.

[Spocks-cuddly-tribble]

there is one unknown variable that however likely does not matter.

Correct, no indication for any asm usage of dword [adrmap+0]. Even entry length data seems ignored, just a read until null.

No hard coded limits of any kind.

[Flocke]

I just pushed changes to https://gitlab.com/stbotf/ultimate-edit ... _detection for adding & removing lexicon entries:

lexicon add & remove.jpg (36.26 KiB) Viewed 2081 times

I gave it a short try and my extended lexicon file loaded just fine in BotF.
You however need to clone or download the source and build it yourself till next release

[Tethys]

I will have a look whether I can add some simple add/remove buttons to UE.

Nice!

When I added my entry, it was in haste and I probably messed up; I did not have time to attempt to fix it. Good to see someone can did it right

Adding and removing entry yes, but perhaps disallowing (grey out the remove button) for any entry below 1459

[Flocke]

Adding and removing entry yes, but perhaps disallowing (grey out the remove button) for any entry below 1459

There already is a check on that, raising a confirmation box.
But I am going to do some further tweaking, and also check on moved entries below the loaded lexicon size.

[Flocke]

I pushed another few updates to the lexicon editor:

Lexicon editor improvements.jpg (71.32 KiB) Viewed 1995 times

Adding the Ikonli library was a bit challenging, but I think those icons are a great addition.
Further I improved layout, added some entry shift checks and some other really helpful features I think.
With the lexicon I'm done for now, unless you find something useful I missed.

[Tethys]

I missed this post, sorry Flocke! The UE looks top notch I just downloaded the latest release 0.9.0 but I have not loaded GALM yet with it. Loaded Vanilla first just to check it out, because I recall somewhere you saying that it "fixes" something with GALM. I just don't know if that "fix" would break features or require further adjustments. I will have to make a backup first, and I am not quite ready to make a backup because I am still making additions.

Thank you for your accurate work sir