Lexicon.dic (Redevelopment)

Other Projects; support/discussion/questions

Moderator: thunderchero

User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2392
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Lexicon.dic (Redevelopment)

Post by Tethys »

TC and I have been talking about ways to condense and consolidate in BotF. One of the most overlooked areas is the lexicon.dic

Many entries are duplicates, some are differing only slightly and can be merged. We have a rough estimate of about 150 lexicon entries that can be freed up to use in projects. This is on the high side. Realistically we may want to keep some diversity (most of the entries would come from Race Descriptions for the 5 majors = 35x4 = 140, and then some here and there). This section may be updated by anyone, but please limit posts to workable code or associated lexicon locations. :!: Irrelevant posts or erroneous info may be removed :!:

Here is an example of a (working) patch that will change duplicate lexicon entry "Echo Papa 607", index 1389 (56Dh) and instead use lexicon entry "Echo Papa 607", index 257 (101h)

Code: Select all

NAME: EP607 Duplicate Lexicon Patch
AUTHOR: Tethys/thunderchero
DESC: This patch will adjust a duplicate Echo Papa 607 lexicon index to use a single index (257), freeing up lexicon entry (1389).
DESC: 
DESC: patch file name: EP607_Duplicate_Lex.patch
URL: https://www.armadafleetcommand.com/onscreen/botf/
TAG: 

# >>  = current value

# <<  = new value

>> 0x04f1fc B8 6D 05 00 00

<< 0x04f1fc B8 01 01 00 00
Unused Lexicon Entry List:

Code: Select all

1350     >     They
1351     >     They break trade treaty
Last edited by Tethys on Thu Jan 27, 2022 4:23 am, edited 1 time in total.
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2392
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Tethys »

Here I will begin documenting the unused lexicon entries I find. Others are welcome to share info, but please verify the entry is not present in trek.exe
I will update the list as info is added.

I have just identified 2 unused entry

1350 and 1351
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3196
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Flocke »

Eh, I am about to update UE on what I found. I recently got side tracked on the lexicon.dic and found it worth to not map the vanilla entries by number any longer but by some readable identifiers. lol
not sure what list there exist already but I ran the app and compared screens and mapped most of the labels and lots of other entries and grouped the entries that belong together but are spread throught the many entries
I'm mostly done with it and will upload if not today then on weekend
but I'm happy to merge what you have already :D

edit: actually I was not looking for unused entries, but the other way around, to give each entry some proper label. So it's worth to keep searching for unused entries and match with my list lateron ;)
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2392
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Tethys »

The way I identified was to find a questionable looking entry and search for the entry in IDA for example entry 1350 I would search text string 546h for a nearby Lexicon Term call, and I found none for those two entry. But Thunderchero advised me that some lexicon term are identified by index pointer? So care should be taken in this regard as well.
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3196
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Flocke »

Many of those index values are computed for sure. A simple search for matching values doesn't help here. That's already cause some text seems to be split to multiple values, and the compiler likely compiled it to relative refences of one entry to another. The best chance you have is to look for key variable references of the lexicon and check that you found all of the text values used ingame.
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 1883
Joined: Sun Apr 27, 2008 2:00 am

Re: Lexicon.dic (Redevelopment)

Post by Spocks-cuddly-tribble »

There is many entries for unused (e.g. alien artifacts, transferred dilithium, food, energy) and semi implemented (e.g. tactical cloak) features. Some of them are referenced in my modding posts (you can search them with the keyword 'lexicon').

Not sure how you guys want to deal with those entries.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2392
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Tethys »

Spocks-cuddly-tribble wrote: Fri Jan 28, 2022 10:31 am Not sure how you guys want to deal with those entries.
I doubt we can remove the entries from the dic without having to alter 90% lexicon indexes in trek.exe

So I'd assume a list of indexes be created (first post) of known unused indexes.
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2392
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Tethys »

I cracked open lexicon.dic in the hex editor and first of all: its weird. There is some table at the end which I don't really understand well. The values do not match potential lexicon entry indexes. I have tried every conceivable combination of value types, 2 byte, 4 byte, 4 and 8 byte floats, little and big endian.. there are just no ID's and its weird... although I did only test the very last entry (1459)

Testing is being done on a vanilla lexicon.dic with only one or 2 entry edited. Although, this may throw off values if UE has any hand (which it does obviously)

First 2 bytes equals DOUBLE (2919) the lexicon list entry numbers (1459) plus 1. I am about to do some testing and see if an additional lexicon entry can be added by adding provisions for 2 slots.

My assumption is that lexicon.dic counts the entry itself as a slot (lexicon index ID), as well as the text associated with the entry. I will report back shortly.

Edit: offset 10b6b (B4 05) total lexicon entry ID plus one
offset 13905 (33 0B 01 00) [68403 decimal] + offset -redacted info (its the next 4 bytes)- (34 00 00 00) [52 decimal] = size of lexicon index block from 0x00 [68455]

this should be enough information to at least attempt to add a new entry and see if it can show up in UE

I did not have success adding entry, there is probably a trek.exe reference to the entry ID count which also need to be adjusted.
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3196
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Flocke »

To understand the lexicon.dic file, best have a look at the UE source code. :razz:
Here I noted what's going on there in ue.edit.res.stbof.dic.Lexicon.java:
ue_lexicon_load.png
ue_lexicon_load.png (125.79 KiB) Viewed 2686 times
https://gitlab.com/stbotf/ultimate-edit ... xicon.java

So there is one unknown variable that however likely does not matter.
I will have a look whether I can add some simple add/remove buttons to UE. :cool:
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 1883
Joined: Sun Apr 27, 2008 2:00 am

Re: Lexicon.dic (Redevelopment)

Post by Spocks-cuddly-tribble »

Flocke wrote: Fri Feb 04, 2022 12:27 pmthere is one unknown variable that however likely does not matter.
Correct, no indication for any asm usage of dword [adrmap+0]. Even entry length data seems ignored, just a read until null.

No hard coded limits of any kind.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3196
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Flocke »

I just pushed changes to https://gitlab.com/stbotf/ultimate-edit ... _detection for adding & removing lexicon entries:
lexicon add & remove.jpg
lexicon add & remove.jpg (36.26 KiB) Viewed 2646 times
I gave it a short try and my extended lexicon file loaded just fine in BotF.
You however need to clone or download the source and build it yourself till next release ;)
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2392
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Tethys »

Flocke wrote: Fri Feb 04, 2022 12:27 pm I will have a look whether I can add some simple add/remove buttons to UE. :cool:
Nice!

When I added my entry, it was in haste and I probably messed up; I did not have time to attempt to fix it. Good to see someone can did it right :lol:

Adding and removing entry yes, but perhaps disallowing (grey out the remove button) for any entry below 1459 ;)
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3196
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Flocke »

Tethys wrote: Sat Feb 05, 2022 12:44 pm Adding and removing entry yes, but perhaps disallowing (grey out the remove button) for any entry below 1459 ;)
There already is a check on that, raising a confirmation box. ;)
But I am going to do some further tweaking, and also check on moved entries below the loaded lexicon size.
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3196
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Flocke »

I pushed another few updates to the lexicon editor:
Lexicon editor improvements.jpg
Lexicon editor improvements.jpg (71.32 KiB) Viewed 2560 times
Adding the Ikonli library was a bit challenging, but I think those icons are a great addition.
Further I improved layout, added some entry shift checks and some other really helpful features I think.
With the lexicon I'm done for now, unless you find something useful I missed. :cool:
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2392
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Lexicon.dic (Redevelopment)

Post by Tethys »

I missed this post, sorry Flocke! The UE looks top notch I just downloaded the latest release 0.9.0 but I have not loaded GALM yet with it. Loaded Vanilla first just to check it out, because I recall somewhere you saying that it "fixes" something with GALM. I just don't know if that "fix" would break features or require further adjustments. I will have to make a backup first, and I am not quite ready to make a backup because I am still making additions.

Thank you for your accurate work sir :)
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
Post Reply

Return to “Other Projects”