Shipname Appendix Fix & Adding new name groups

Shipname Appendix Fix & Adding new name groups; support/discussion/questions

Moderator: thunderchero

User avatar
EnPhreg
Lieutenant-Commander
Lieutenant-Commander
Posts: 130
Joined: Thu Jul 10, 2008 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by EnPhreg »

Spocks-cuddly-tribble wrote: Sat Dec 29, 2018 3:48 pm In theory the following two changes of the patch 'Adding new shipname groups - extending shipname.bin & shipname.dat' might work with the 'one byte per group in shipname.bin' change by DCER.

Code: Select all

at 0x74910 change 8B 16 83 C6 04 -> 0F B6 16 46 90

at 0x749C6 reset 0x13 bytes to 8B B3 F8 00 00 00 C1 E7 02 8B 4C 24 2C 01 FE 89 E7 01 CE


-asm-

AUTO:00475510
movzx edx, byte ptr[esi]
inc esi
nop

restore original code sequence at:

AUTO:004755C6                 mov     esi, [ebx+0F8h]
AUTO:004755CC                 shl     edi, 2
AUTO:004755CF                 mov     ecx, [esp+2Ch]
AUTO:004755D3                 add     esi, edi
AUTO:004755D5                 mov     edi, esp
AUTO:004755D7                 add     esi, ecx
Happy testing. :smile:
nice!
i will give it a try asap.

hi SCT, greetings,
would it be possible to give an answer to my questions in this post? viewtopic.php?p=49396#p49396
or do you have already read it and you're out of business to long to have useful answers for me?
User avatar
EnPhreg
Lieutenant-Commander
Lieutenant-Commander
Posts: 130
Joined: Thu Jul 10, 2008 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by EnPhreg »

so, first results for testing:
implemented new code and used a shipname.bin with one additional name group and card strike cruiser model I + II was given the new name group 62 (3E) via manually shiplist.sst edit.
within game, the names of all ship types are not set correctly anymore.
within UE, can't edit ship stats, cause UE has still a problem that a ship model is using a name group higher than 61,
but ship name edit in UE works again, cause shipname.bin has the 'one byte per group' structure again.
but the game still can't handle the 'one byte per group' structure of shipname.bin.
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6632
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Shipname Appendix Fix & Adding new name groups

Post by thunderchero »

EnPhreg wrote: Sun Dec 30, 2018 7:17 am
thunderchero wrote: Fri Dec 28, 2018 6:27 pm If I saw the shipname.bin I might be able to explain error.
i have attached the shipname.bin file. at the end of the list is group 62 (132 new bytes - each name 44 bytes) (ID 3E - starting at (hexadecimal) 0xF4B8), three names copied and altered from yridian name list (group 61 - 3D).
at position 0xF8 4 new bytes for number of names in new group 62.
at position 0xFC 4 bytes with new pointer for start of the name group list.

shipname.bin is attached within a zip-file. wasn't able to attach the single file.
The only difference I see is you used the german version shipname.bin. but this should not matter.
Spocks-cuddly-tribble wrote: Sat Dec 29, 2018 3:48 pm Happy testing. :smile:
I had same results as EnPhreg :sad:

EDIT;
EnPhreg wrote: Sun Dec 30, 2018 7:52 am within UE, can't edit ship stats, cause UE has still a problem that a ship model is using a name group higher than 61,
playing with UE source code I was able to increase limit to allow selection above 61, but it is a fixed value. from my limited experience with java it should be possible to tell it to read current groups in shipname.bin for max to display.
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 909
Joined: Sun Apr 27, 2008 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by Spocks-cuddly-tribble »

1.) If UE doesn't support adding more name groups, even apart of the 'one byte glitch', the whole thing seems kinda pointless.

2.) If you solve #1 via altering UEs java code then better revert the 'one byte glitch' as well than messing with the asm code.

3.) Possibility of asm-fixing depends on the way the UE glitch 'corrupts' shipname.bin. I assumed the unused of the 248 name groups are represented by zero bytes with the pointer for start of name list, regardless of number of used name groups, always at vanilla offset and default. But your test results indicate this isn't the case?
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6632
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Shipname Appendix Fix & Adding new name groups

Post by thunderchero »

Spocks-cuddly-tribble wrote: Sun Dec 30, 2018 5:41 pm 3.) Possibility of asm-fixing depends on the way the UE glitch 'corrupts' shipname.bin. I assumed the unused of the 248 name groups are represented by zero bytes with the pointer for start of name list, regardless of number of used name groups, always at vanilla offset and default. But your test results indicate this isn't the case?
your new code and current way UE sets up shipname.bin will result in CTD on start of game some times. when game does start all ship names will start at wrong point of ship name list. you may get last 1 or 2 letters of a ship name or name maybe blank

UE writes shipname.bin as follows;
0x00 -> 0xf7 preset possible 248 groups
0xf8 -> 0xfb preset FC 00 00 00 for start of name list
0xfc -> to end of file ship names

note test 2; your new code and shipname.bin edited after groups added by UE
0x00 -> 0x3E 62 groups
0x3F -> 0x42 new offset for start of name list starting at end of used groups
0x43 -> to end of file ship names

game will start in all my tests, but all ship names will start at wrong point of ship name list. you may get last 1 or 2 letters of a ship name or name maybe blank.
User avatar
EnPhreg
Lieutenant-Commander
Lieutenant-Commander
Posts: 130
Joined: Thu Jul 10, 2008 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by EnPhreg »

thunderchero wrote: Sun Dec 30, 2018 10:55 am The only difference I see is you used the german version shipname.bin. but this should not matter.
how can you identify my shipname.bin as german? what's the difference?
i thought they were 100% identical?!
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6632
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Shipname Appendix Fix & Adding new name groups

Post by thunderchero »

EnPhreg wrote: Mon Dec 31, 2018 11:11 am
thunderchero wrote: Sun Dec 30, 2018 10:55 am The only difference I see is you used the german version shipname.bin. but this should not matter.
how can you identify my shipname.bin as german? what's the difference?
1. spelling of "outpost" and "starbase" are in German
2. second difference of your file some ship names start with 1B 00 instead of FF FF starting at 0xee5e -> 0xef39 all 6 ship names in group 56 (38 hex)
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 909
Joined: Sun Apr 27, 2008 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by Spocks-cuddly-tribble »

This UE file handling is far worse than expected.

So I kindly recommend:

1.) Cleaning up this topic via deleting all post after my fix codes one page 1 i.e. beginning here:

viewtopic.php?f=212&t=1622#p47330

2.) Adding a warning that the corresponding UE feature is bugged, rendering the game unusable, and shouldn't be used until further notice.

3.) When reworking UE code only note info given in my main fix post.


Some elaborations just to make it hardcore pakled proof:

1.) The code works for both (extended and vanilla) shipname.bin files as long as the file structure is setup correctly

2.) The file structure is 4-byte entries before start of name list and 44-byte entries beginning with start of name list (the FFFF part matters!)

3.) As told twice in my post the pointer for start of name list can be ignored/overwritten/removed since start of name list is auto-located (EU doesn't need to calculate this!) Since I expected that no one cares about reading my instructions properly, I setup auto-location code to work with all possible options (i.e. pointer deleted, adjusted or just overwritten, all the same, BotF won't care as long as #2 is given!)

4.) It follows from #3 that it doesn't matter whether unused name groups are represented by zero valued four byte entries before start of name list or not, giving maximum freedom to setup UE code as long as #2 is given
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6632
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Shipname Appendix Fix & Adding new name groups

Post by thunderchero »

Spocks-cuddly-tribble wrote: Tue Jan 01, 2019 2:02 pm 1.) Cleaning up this topic via deleting all post after my fix codes one page 1 i.e. beginning here:

viewtopic.php?f=212&t=1622#p47330
I try not to delete any posts, but I will split topic once/if a solution is found for UE issue.
Spocks-cuddly-tribble wrote: Tue Jan 01, 2019 2:02 pm 2.) Adding a warning that the corresponding UE feature is bugged, rendering the game unusable, and shouldn't be used until further notice.
Done
Spocks-cuddly-tribble wrote: Tue Jan 01, 2019 2:02 pm 3.) When reworking UE code only note info given in my main fix post.
Thanks to your pakled proof explanation I now understand what is exactly required :grin:

Edit; just on a side note. I noticed that groups 8, 13, 18, 27, 29, 43, 47 have no ships assigned to those groups in vanilla.
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 2641
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Shipname Appendix Fix & Adding new name groups

Post by Flocke »

I had a look on the UE code now and checked on the shipname.bin structure.
From my stand point you completely messed this topic and DCER implemented exactly what he has been asked for.

The file has a fixed header size of 248 bytes, followed by a useless 4 byte pointer to the entry list, which is 44 bytes each. Each entry then has 2 bytes for the id, 2 bytes set to 'FF FF' for 32bit alignment of a short value I guess, followed by the name.

The patch was said to "support up to 248 shipname groups i.e. IDs 0-D7", with the asm changes barely documented, which lead to the assumption that each group now is identified by a single byte to fit the fixed header size of 248 bytes.

If that is false and instead the exe was changed to read 248 x 4 = 992 bytes instead, this is really easy to fix.
Have a look at Shipname.java:
shipname_fix.jpg
shipname_fix.jpg (50.7 KiB) Viewed 248 times
What is happening here:
In line 115 you see an if check whether there are more than 62 groups to store (group0-61).
If smaller than 62, it stores in the old format, iterating all the groups in the first for loop and converting the entry count "arr.size()" in line 120 to a byte array (32bit int = 4 bytes) and writing it out. The next for loop in line 124 starts with "i = size" of where the previous loop stopped and fills the remaining group slots with zeros to match the fixed header size.

When you now check the else clause below of it, you see it mostly does the same but iterates up to 248 values and always writes a single byte instead. See line 136, there it reads first byte of the converted address with "b[0]" and only writes that single byte instead of the full 4 byte array. And in the for loop below, at line 139 it iterates up to 248 groups instead of the previous 62 ones and then in line 141 it writes a single 0 byte instead of the 0 byte array like in the if clause.

So how to fix:
  • remove the if check in line 115 as well as the full else block.
  • increase the 62 group limit in line 124 to 248.
  • clean up the indention and remove the now redundant brackets from line 116 and 129 if you like
  • for the read part in line 54 ofc also fix to skip 992 bytes instead of 248
  • fix the size calculation at line 159
  • cleanup the "StopSign" naming, that is not a stop sign but the broken dynamic list offset cause they messed it by placing it in a dynamic header
I however wonder why with the asm patch the limit was set to 248. Why not fix the file format and add a group counter to the front or move the entry pointer? Why exactly 248 group entries and not 1000 or something?

---
edit:
Spocks-cuddly-tribble wrote: Tue Jan 01, 2019 2:02 pm 2.) The file structure is 4-byte entries before start of name list and 44-byte entries beginning with start of name list (the FFFF part matters!)

4.) It follows from #3 that it doesn't matter whether unused name groups are represented by zero valued four byte entries before start of name list or not, giving maximum freedom to setup UE code as long as #2 is given
If I'm interpreting that correct, that is a flawed implementation. Sure, the entry counts never reach FFFF I guess, but each programmer will hate you to place a redundant check just to deviate from the explicit file format but do your own rolling. :wink:

That also counts when you read the group count from last entry index instead. When you compare to other files it is convention to read the count from first entry or list offset pointers in the header when there is different data that follows.

I guess here the developers wanted to implement a dynamic group list but messed it up when they added the entry list pointer to the header, forgetting on that new groups in the header of course resize the header and thereby break the location pointer. They should have added a counter to the front like they did with other files.

---
edit2:

further fixes:
  • on the Ship Stats side in Ship.java line 378 you find an error check that you might want to update.
  • in ShipnameGUI.java line 336 you might want to change the patch warning and remove to set the ShipnameCode action, given in my test trek.exe is not modified by UE and it is good it doesn't enforce so - alternatively remove the return in line 346 to allow ignore that warning. If removing this option, the btnREM_GROUP listenner in line 368 should be removed as well
  • in ShipGUI.java line 404 fix the group selection count - it should better not be hard coded but read from the Shipname class instead, which could be loaded along with the other data in line 205 by "SHIPNAME = (Shipname)STBOF.getInternalFile(Stbof.fileSHIPNAME_BIN, true);" if I got that right.
That should be all. With a text search I check all the source file matches on hard-coded "61", "62", "63" and "248" occurences and also investigated the Shipname, ShipStat and their gui files.
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 909
Joined: Sun Apr 27, 2008 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by Spocks-cuddly-tribble »

Hi Flocke,

first of all, you are a great guy and we all love you.

Now to your points:
Flocke wrote: Fri Apr 16, 2021 7:37 pmFrom my stand point you completely messed this topic and DCER implemented exactly what he has been asked for.
You lack the context to assess the situation:

- at this time (10+ years ago) Peter1984 and Tethys demanded so much support for their countless (and sometimes pointless) projects to extend many stbof.res files, that it bugged even you at some point
- changing the shipname.bin header structure (dword to byte) would require rewriting almost the entire asm code and even worse, when used with a not matching trek.exe code, can cause not only the game to crash or freeze with no crashlog, but also can freeze the entire OS (due to the very unfortune coding of the memory allocation of shipname.dat)
- due to the complex, multifactorial nature of the problem wrt shipname.bin vs. shipname.dat, I stepped in and provided a solution, delaying more important projects
- thunderchero and Peter1984 mistakenly claimed my code doesn't read namegroups 63+ in game, it turned out they used UE to add their manually hex-edited, extended shipname.bin files to stbof.res, and UE auto removed groups 63+ when doing so
- Dcer (at this time already being out of business) was just asked to keep UE from doing this, but Peter1984 thronged Dcer to implement the extension feature, providing a wrong link detailing my structure changes of shipname.dat (dword -> byte)
- Dcer never again posted after this and he probably never read this topic, not sure the patch info wasn't even posted by the time of his last UE update attempt
- thunderchero and Peter1984 were able to extend shipname.bin using my patch instructions i.e. they understood the header structure is still dword and the byte change refered to shipname.dat

- shipname.bin file size (and header size) is variable, it's data structure not (dword before start of names list and 44 byte name entries)
- you CAN use a fixed position for start of name list with no issues (e.g. 4*248) if you don't want to check for number of used name groups

Flocke wrote: Fri Apr 16, 2021 7:37 pmI however wonder why with the asm patch the limit was set to 248. Why not fix the file format and add a group counter to the front or move the entry pointer? Why exactly 248 group entries and not 1000 or something?
Again, this limit is determined by shipname.dat, not shipname.bin itself. Today you could just create a new dynamic area and relocate all references to it, at this time we did not know how to do this (even with your and QDs support).


Flocke wrote: Fri Apr 16, 2021 7:37 pmIf I'm interpreting that correct, that is a flawed implementation. Sure, the entry counts never reach FFFF I guess, but each programmer will hate you to place a redundant check just to deviate from the explicit file format
More precisely, it's the only workaround for the flawed implementation you describe below, which works for default and extended shipname.bin files.

Flocke wrote: Fri Apr 16, 2021 7:37 pmI guess here the developers wanted to implement a dynamic group list but messed it up when they added the entry list pointer to the header, forgetting on that new groups in the header of course resize the header and thereby break the location pointer.
Correct, and that's only the shipname.bin part of this mess (which my patch ignores now). See my notes above about potential issues with the messy default memory allocation of shipname.dat. :shock:
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 2641
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Shipname Appendix Fix & Adding new name groups

Post by Flocke »

Oh well, I hoped for an easy fix on the file structure, but then it''ll be the fixed 992byte header. No problem to that.
Regarding DCER I easily could have failed on the header adjustment just as well. Without having looked into the asm code, I would have assumed it was rather easy to change the group size but not the fixed header size. And by the number, I had no clue that there is some other reason to make it exactly the number of the header size.

I know how hard one sometimes needs to battle the asm code. The patch for 248 naming groups is a great patch already and offers ALOT of space for namings! :)
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 909
Joined: Sun Apr 27, 2008 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by Spocks-cuddly-tribble »

Flocke wrote: Sat Apr 17, 2021 8:34 pm 248 naming groups ... offers ALOT of space for namings! :)
You could it even limit to 128 groups in UE to have a smaller and cleaner header. Given the BotF limit of 255 ships in shiplist.sst (including monster, upgrades and no-name groups) no mod should ever reach this.

One last piece of advice, I'd add a note in the UE shipname.bin GUI that EVERY change to shipnames (and/or it's code) requires starting new games. Otherwise people will claim UE and/or exe-code is broken....

Flocke wrote: Sat Apr 17, 2021 8:34 pmRegarding DCER I easily could have failed on the header adjustment just as well.
He didn't fail. If at all, his mistakes would be 1. being too nice and not rejecting unfounded, demanding requests and 2. trusting technical info from the wrong people.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
Lathon
Lieutenant-Commander
Lieutenant-Commander
Posts: 112
Joined: Thu Oct 15, 2009 2:00 am

Re: Shipname Appendix Fix & Adding new name groups

Post by Lathon »

Spocks-cuddly-tribble wrote: Mon Apr 19, 2021 1:31 pm
Flocke wrote: Sat Apr 17, 2021 8:34 pm 248 naming groups ... offers ALOT of space for namings! :)
You could it even limit to 128 groups in UE to have a smaller and cleaner header. Given the BotF limit of 255 ships in shiplist.sst (including monster, upgrades and no-name groups) no mod should ever reach this.

One last piece of advice, I'd add a note in the UE shipname.bin GUI that EVERY change to shipnames (and/or it's code) requires starting new games. Otherwise people will claim UE and/or exe-code is broken....

Flocke wrote: Sat Apr 17, 2021 8:34 pmRegarding DCER I easily could have failed on the header adjustment just as well.
He didn't fail. If at all, his mistakes would be 1. being too nice and not rejecting unfounded, demanding requests and 2. trusting technical info from the wrong people.
Or we could just have a general warning after saving that states "Most changes will require a new game." That way it covers anything else as well
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 2641
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Shipname Appendix Fix & Adding new name groups

Post by Flocke »

Lathon wrote: Mon Apr 19, 2021 5:42 pm Or we could just have a general warning after saving that states "Most changes will require a new game." That way it covers anything else as well
Yeah I thought the same. :up:
But better warn prior to having it saved. LOL :lol:
Spocks-cuddly-tribble wrote: Mon Apr 19, 2021 1:31 pm He didn't fail. If at all, his mistakes would be 1. being too nice and not rejecting unfounded, demanding requests and 2. trusting technical info from the wrong people.
I only can make assumptions, and with my programming view I likely would have failed by your initial post. And I have enough confidence in DCER that he did not blindly follow 3rd party instructions without having at least taken a look into what the author of this patch posted.

But misunderstandings happen all the time and this doesn't matter much now. I have much respect to all of you. And ofc DCER simply could have declined the request, there is no point in demanding anything or in blaming someone for not having it fixed, especially not by me. And now we'll have that fixed soon, finally. ;)
Post Reply

Return to “Shipname Appendix Fix & Adding new name groups”