Extending the shiplist

Extending the shiplist; support/discussion/questions

Moderator: thunderchero

User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2556
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Extending the shiplist

Post by Tethys »

Flocke wrote: Sun Mar 24, 2024 4:17 pm
Spocks-cuddly-tribble wrote: Sun Mar 24, 2024 3:50 pm Indeed, good thinking. There is two cases where it might fail:

- removing all minor/monster ships from shiplist (have the monsters shiptech.sst entries?)
- using "building minor ships after membership" patch

BUT, easiest way to make it work would be just one extra single byte edit: :wink:
0046F81C  jz   short loc_46F85D (inc ship ID check next or exit) -> loc_46F86B EXIT
Nice trick to depend on
46F970         mov     ax, ds:shipNum_Max_ID
As a programmer I however learned that true while loops are bad coding style. :razz:
I'm happy with the current patch I implemented. :cool:
Not related directly but related to original topic:
shipNum_Max_ID 5 locations but I think shiplist can be extended up to 65535 max ships.

Code: Select all

Replace:

asm
0046FB81 66 A3 3C 54 5B 00    mov ds:005B543C, ax
0046FB87 66 3D 7D 00          cmp ax, 7Dh
With:

asm
0046FB81 A3 3C 54 5B 00       mov ds:005B543C, eax
0046FB86 3D 7D 00 00 00       cmp eax, 7Dh

This changes a 10-byte block into a 10-byte block again — so no size mismatch, and everything stays aligned.
This is one subroutine out of 5 where shipNum_Max_ID is read or written. All should be changed. Also:

Code: Select all

005B543C ?? ??                   shipNum_Max_ID_word_5B543C dw ?                   ; DATA XREF: shipTech_sst_entry_shipIdEAX_to_edx+30r
005B543C                                                                           ; read_shiplist_sst_46F9C0+30r ...
005B543E ?? ??                           align 10h
align 10h can be used because critical section falls on a 16 bit divisible address by default: no performance hit, no crashes (assuming all code referencing critical section ACTUALLY falls on the 16 bit divisible address and not 2 bytes before it). Change dw to dd. Further adjustments may be necessary for example 0046F808 in build_shiplist_7D sub. That's all I have been able to figure out for now, I will revisit later. UNTESTED

Code: Select all

0046F7FE 0F 84 71 00 00 00                   jz      loc_46F875

becomes

0046F7FE 74 75 90                    jz      loc_46F875
89 44 24 10                         mov     [esp+10h], eax
3D 7D 00 00 00                         cmp     eax, 7Dh  
Why has nobody suggested changing the encoding of the signed value from 2 to 4 bytes? There is enough space, and it looks like it can be done without using relocations. At the very least we should see 32,768 shiplist entries :eek: Good luck hitting that limit :lol:

Code: Select all

0046F866 83 FA 7D                            cmp     edx, 7Dh        ; Compare Two Operands
0046F869 7C A2                               jl      short loc_46F80D ; Jump if Less (SF!=OF)
The above is the only one yet found which would need relocation, 0F bytes needed 0046F80B and 0046F82D can be adjusted plus 2 bytes, move conditional jl 0046F869, rewrite 0046F86B to EB jump about 4-5 bytes forward to earlier exit.

Code: Select all

00479836 81 C5 F4 01 00 00                   add     ebp, 1F4h       ; 500 unused combat.bin entry size
Is this truly unused? Can these bytes be utilized for extension?

Off topic: why is starname bin with a 125 limit? Is this 125 scans of the map to place names or something?
004B78A4 83 F9 7D cmp ecx, 7Dh ; Compare Two Operands
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2556
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Extending the shiplist

Post by Tethys »

I think I see now that al is the problem, with it being so small and as SCT said is used about 150x for ship list. But I think not all of them need to be changed because some of them use ax (enough bytes for intended goal)
Spocks-cuddly-tribble wrote: Fri Apr 02, 2010 7:08 am On this occasion, I also had another look at extending the shiplist phase II.


Word instead of single-byte registers - not very complicated (excepting possible issues with ship-ID "FF") - but all in all ~200 code sequences would have to be rewritten:

sub_46F940 (read_shiptech_sst) -> 15 calls
sub_46F9C0 (read_shiplist_sst) -> 155 calls

So even with a very optimistic assumption of 5 minutes per code location...

Thereby the following deliberations become more or less moot. :(

UPDATE: I have inspected all the reference subroutines for
0046F940 read_shipTech_sst_entry_shipIdEAX_to_edx
There looks to be quite a few places where al, cl, and dl should all be extended. Of course there is no room natively to add the preceding opcode, and we cant just use the full eax, ecx, and edx for all locations because of preceding data (garbage data left in the register) and also because most importantly there are ONLY 2 bytes available in shiplist.sst file @ offset 0x66h. At 0x68h there is other data which would mess up a full register (eax or similar) move of the 4 bytes here (garbage in the lower half). So ax is required in these cases (which also require relocating some preceding datas to make room for extra opcodes). Data at 68h appears to be "ship function +68h"; alternatively, I believe movsx/zx can be used and might save some codespace :?: when using eax

I am currently building a list of locations and trying to discern which type is needed (if eax is used with movsx/zx or if we must use ax in certain situations).
List 16 locations for read_shipTech_sst: (location follow by ? need further investigations)
► Show Spoiler
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2556
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Extending the shiplist

Post by Tethys »

List 268 locations for read_shiplist_sst_46F9C0 (might be more or less, some parts not well understood)
► Show Spoiler
Several locations in this last section above require relocation due to operand size. I think a mini-subroutine can be written for most of them, some will need to be single jump out and jump back in though. For example mov al, [mem+52h] is followed many times by a mov edx, esp; the mini-subroutine can be called here instead of taking up several unique locations for the same exact duplicated bytes. Most efficient. When mov al, [mem+52h] is preceded by a xor eax, eax it should be safe to use the whole eax in this case > mov eax, [mem+52h], at least when eax gets overwritten later on.
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3659
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Extending the shiplist

Post by Flocke »

Tethys wrote: Sat Aug 16, 2025 4:24 pm Several locations in this last section above require relocation due to operand size. I think a mini-subroutine can be written for most of them, some will need to be single jump out and jump back in though. For example mov al, [mem+52h] is followed many times by a mov edx, esp; the mini-subroutine can be called here instead of taking up several unique locations for the same exact duplicated bytes. Most efficient.
That's a very good idea actually! I went through all 155 sub_46F9C0 + 15 sub_46F940 calls yesterday, and already included all of them but a few in upcoming IDA documentation update. It comes down to about 10 different cases to handle, all the rest mostly is repeated code. These include:
  1. +52h: Ship_t.shipType
  2. +2Eh: Starbase_t.shipType
  3. +66h: ShipClass_t.id
  4. +04h: ShipTechEntry_t.shipType
  5. +10h: ProdQueueItem_t.build.shipType
  6. +08h: ?? ProdQueueItem_t.build.structureType / EconomicOrder_t.structureType
  7. +2Ah: SceneNode_t.shipType <- new SceneNode_t looks very interesting to mpr btw :eek:
  8. xor eax, eax; mov al, 0FFh where 0FFh is a 32bit value
  9. mov edx, eax; xor eax, eax; mov al, dl; stuff, where edx is a 16bit sub-routine return value
  10. xor eax, eax; mov al, dl; where dx is a 16bit sub-routine call param
  11. mov al, [ebx] where ebx is a 32bit shipType value array
  12. shl eax, 2; lea ecx, [eax+ebp]; xor eax, eax; mov al, [ecx] which looks weird but computes and reads value from 32bit shipType value array offset
Plus all of them I have seen can be fixed in place, most if not all even without adding any helper routines. This investigation now makes me very optimistic that Tethys is right and we can fix all of these without too much effort, rather talking of seconds per repeated fixture.

Not saying that it isn't still alot of work. Already all the asm analysis was. But the work now rather is down to yet undetected code dependencies and exceptions, that I didn't figure yet, like:

Code: Select all

004D40D5         call    UI_TechScrn_TechScrn_4D1A70
004D40DA         mov     edx, ds:dword_5CBA40
004D40E0         mov     edx, [edx+38h]
004D40E3         mov     al, [eax+edx*4]
Eh, what? :shock:
Some scene related data structure reading the edx index of some 32bit shipType array? :eek:

Code: Select all

004F6066         mov     dx, [eax+ProdQueueItem_t.build.structureId]
004F606A         movsx   eax, dx
--
4F5134         mov     al, byte ptr [ebx+ProdQueueItem_t.build.structureId]
Why does it read the SystemBuildInfo_t.structureId here, not the SystemBuildInfo_t.shipType?
And is it actually reading the SystemBuildInfo_t.structureId or rather the EconomicOrder_t.structureId?
Tethys wrote: Sat Aug 16, 2025 4:24 pm When mov al, [mem+52h] is preceded by a xor eax, eax it should be safe to use the whole eax in this case > mov eax, [mem+52h], at least when eax gets overwritten later on.
Sure. Most importantly all the cases I have seen actually are stored as 16bit short values. It looks rather that the subroutines were declared to only accept 8bit values, which lead to all the 8bit conversion mess. :mad:

I didn't went through all your 268 case list yet, but noticed that you included a few false entries that actually read raceId like this one:
44907C systems_stations_list_recipient_al

I will try to finish updating IDA docs today. :mrgreen:

P.S. Given all that shipId and shipClass mess, I started to name those ShipClass_t ids "shipType". Not sure about it yet, but ship ids are reserved for actual savegame ship instances, while ship classes already are double-used by object database ship class descriptions. Furthermore the term "type" appears more generic to me for different sets of ship stats of same ship class model. Plus it would be fine to be used for buildings, too. :roll:
The only issue I have with it is that ShipType_t shipType = GetShipType(int shipType); does not read well when I rename ShipClass_t :lol:
Last edited by Flocke on Wed Aug 20, 2025 2:55 pm, edited 1 time in total.
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2556
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Extending the shiplist

Post by Tethys »

some of those may have a ? in the comment so they might be not well understood. If any are missing, it would be very few. Thanks for taking a look into this, and it makes me even more optimistic that you are also optimistic :D
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Tethys
Past Administrator
Past Administrator
Posts: 2556
Joined: Fri Jul 18, 2008 2:00 am
Location: Your mom's bed ;)
Contact:

Re: Extending the shiplist

Post by Tethys »

Flocke, I hadn't even noticed that there were other data (2E) which were also duplicated throughout. However some do use a different register and are sometimes combined with another register [eax+edx+num] in which case, unless there are 2 or more, would be more of a targeted change rather than some handler routine. So it looks like alot can be handled via call and retn, but care should be taken to make sure the right subroutines are being called in the right place ;)

There will be more than 10 mini-subroutines. If there are 2 or more instances it should have a shared handler. :grin:

I can begin working on code for a patch, but my time is a bit limited.
Not for the weak of heart...
Galaxies MOD v0.4.0 <--- GALM/Galaxies Mod latest version
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3659
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Extending the shiplist

Post by Flocke »

Tethys wrote: Thu Aug 21, 2025 3:10 am I can begin working on code for a patch, but my time is a bit limited.
Right now, mine sadly is a bit limited, too. The code will become much clearer to investigate with updated IDA DB.
Not sure if I find time for it today though. :roll:
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3659
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Extending the shiplist

Post by Flocke »

Tethys wrote: Wed Aug 13, 2025 1:59 pm

Code: Select all

00479836 81 C5 F4 01 00 00                   add     ebp, 1F4h       ; 500 unused combat.bin entry size
Is this truly unused? Can these bytes be utilized for extension?
By former research and testing it has been found unused. And I can't find any reference but the combat.bin load routine either. That load routine however loops all entries and at asm<47980D> validates that they are not -1. So you will want to nop the validation. ;)

However, keep care to not remove the hard-coded initialization at asm<479841>. That one is still in use for offensive strength calculation.

for reference:
Spocks-cuddly-tribble wrote: Thu Aug 06, 2009 6:26 pm 5B58A8: address of loaded combat.bin (seems unused)
DCER wrote: Sat Mar 28, 2009 1:45 pm I've removed combat.java from the code. It was meant for the combat.bin file, but as thunderchero pointed out it's not being used by botf at all. :(
thunderchero wrote: Sat Jun 26, 2010 1:23 pm I think Gowron tested these files at 1 point, But can not find the post about it. (may have been lost when forum was restored)

I think the files are needed or will cause crash (part of file check) But even if all values are set to (0) it has no effect on game play. The AI control was very messed up since it started as files but seemed to be changed to trek.exe so many files are not used like the combat.bin.
Spocks-cuddly-tribble wrote: Fri Dec 13, 2019 4:01 pm The unused combat.bin is a ship vs ship odds table (pointless unless 1vs1), so instead the AI reads stats like shipfunction, build cost and:

[shiplist.sst +9C] = AI indicator for offensive capability
[shiplist.sst +A0] = AI indicator for defensive capability
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3659
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Extending the shiplist

Post by Flocke »

Tethys wrote: Wed Aug 13, 2025 1:59 pm Off topic: why is starname bin with a 125 limit? Is this 125 scans of the map to place names or something?
004B78A4 83 F9 7D cmp ecx, 7Dh ; Compare Two Operands
This is a retry limit for generating the random star name index.

Each time it is looped, the random number is divided by the actual star name count to compute the edx remainder.
The remainder then is used to index the star names. And at 0x14 the entry is checked for the name length.
Only when that is zero, the random compution is repeated and the counter keeps incrementing.

While 0x14 by default is set to the name length, it might also be used to indicate what names are already in use. But I didn't do any debugging to check on it. :wink:

Taking a look at starname.bin. It looks to be split into 10 sections, with star names first sorted by length and then alphabetically:
000-001: 3 letters: Exo, Kea
002-016: 4 letters: Brax - Ruah
017-067: 5 letters: Alpha - Zayra
068-156: 6 letters: Altair - Yadera
157-208: 7 letters: Arkaria - Zytchin
209-246: 8 letters: Andevian - Xanthras
247-255: 9 letters: Adelphous - Tau Cygna
256-273: 10 letters: Berengaria - Zeta Alpha
274-284: 11 letters: Deinonychus - Theta Cygni
285-296: 12 letters: Alpha Cygnus - Ultima Thule

For each of the entries, at 0x14 the size of the string is stored as length + 0-termination character.
In AFC Vanilla install it has been defaulted to 0x15, which is wrong, because the max valid length is limited to 0x14. This however likely has no effect, because the string is still 0-terminated.

My original BOTF CD lists correct name length.
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 2175
Joined: Sun Apr 27, 2008 2:00 am

Re: Extending the shiplist

Post by Spocks-cuddly-tribble »

Yes, combat.bin is loaded but not used -> you can remove load code to prevent crash before removing file from stbof.res
Flocke wrote: Sat Sep 27, 2025 6:30 pmWhile 0x14 by default is set to the name length, it might also be used to indicate what names are already in use. But I didn't do any debugging to check on it. :wink:


For each of the entries, at 0x14 the size of the string is stored as length + 0-termination character.
In AFC Vanilla install it has been defaulted to 0x15, which is wrong, because the max valid length is limited to 0x14. This however likely has no effect, because the string is still 0-terminated.

My original BOTF CD lists correct name length.
Yes, starname.bin entry+14h mustn't be 0 for the used names marker, but any hex value other than 0 does the same. So vanilla or AFC value doesn't matter.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is....
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3659
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Extending the shiplist

Post by Flocke »

Flocke wrote: Wed Aug 20, 2025 1:52 am P.S. Given all that shipId and shipClass mess, I started to name those ShipClass_t ids "shipType". Not sure about it yet, but ship ids are reserved for actual savegame ship instances, while ship classes already are double-used by object database ship class descriptions. Furthermore the term "type" appears more generic to me for different sets of ship stats of same ship class model. Plus it would be fine to be used for buildings, too. :roll:
The only issue I have with it is that ShipType_t shipType = GetShipType(int shipType); does not read well when I rename ShipClass_t :lol:
I think this issue bugged me already from the start when I implemented UE savegame editing!
I remember I once already started some discussion on it and SCT you pointed me at trek.exe code, but I can't find any post on it. In trek.exe I however found the Order_t logging routine sub<455988>, which you probably reffered to.

By sub<455988> I'd have to rename "shipRole" + "shipFunction" back to "shipType", "shipType" to "shipClassId" and "buildingType" to "structureId".

Further ship role = ship type references:
asm<40CDCD> "shipCounts[item->shipType] >= 1"
asm<42B999> "Invalid ship type encountered"

Further ship type = ship class references:
asm<46FA75> "Need to open ship class database before ship description"
asm<46F9D7> "shipClassMem is NULL\n"


What I however dislike on it is:
  1. That each savegame building and energy structure should have a unique id to be identified.
    Even when BOTF doesn't assign any IDs, ids should be reserved for actual object instances.
    And maybe I'd add a mapping some time for identification.
  2. stbof.res building and ship entries should have a consistent naming convention.
  3. Ship class names are displayed ingame in the object database, still they have multiple entries for the upgrades, so the same ship class name identifier becomes ambiguous.
I however think it is better to remain trek.exe consistent and therefore I tend to rename all these entries accordingly.
If then I need another building ID, I'll find a different name like "instanceId".
And for the ship classes one still can argue they indeed are the same class but just different versions, therefore in trek.exe it is named "ClassId" and not simply "Class" *ha ha - ironic laugh*.

Any thoughts?

----
Edit: Moved OrderInfo findings to viewtopic.php?p=64927#p64927
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 2175
Joined: Sun Apr 27, 2008 2:00 am

Re: Extending the shiplist

Post by Spocks-cuddly-tribble »

Flocke wrote: Sat Oct 04, 2025 3:00 pmfor the ship classes one still can argue they indeed are the same class but just different versions, therefore in trek.exe it is named "ClassId" and not simply "Class" *ha ha - ironic laugh*.

Any thoughts?
function = type (e.g. colony, TT, outpost) -> makes sense -> so source label type is fine

'Class' is a mess e.g. Defiant, Galor or Galaxy might have different versions (shiplist.sst IDs)
-> shipClassDatabase (shiplist.sst) -> AI fleetbuildup and upgrade bugs since they failed to anticipate this concept bias

shipID (alone) -> also a mess -> GshipList OR shiplist.sst MUST be obvious
-> source code means savegame items (e.g. GshipList), but modder mean shiplist.sst items

Flocke wrote: Sat Oct 04, 2025 3:00 pmMoved OrderInfo findings to viewtopic.php?p=64927#p64927
0x38-0x3B action: again some diplomatic action type
possibly related to the target effects
ORDER_DIPLOMACY_typeId_5 +38h -> treatyType -> same as treaty+30h -> sub_45C6F8 used for new treaties
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is....
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3659
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Extending the shiplist

Post by Flocke »

Spocks-cuddly-tribble wrote: Sun Oct 05, 2025 4:18 pm shipID (alone) -> also a mess -> GshipList OR shiplist.sst MUST be obvious
-> source code means savegame items (e.g. GshipList), but modder mean shiplist.sst items
That was main reason I renamed shipId to shipType for UE savegame editing.
They could just not have same names when added to same Java ship class. :mad:

In trek.exe the shiplist.sst IDs are named "ClassId". But it is not surprising that when edifice.bst buildings have an entry ID, that ppl anticipate shiplist.sst items to have one, too! That is why (2.) I'd prefer a consistent naming convention.

But now, by trek.exe the buildings are actually structures and have both the edifice.bst structureId AND a structureType, which I guess stands for the food/industry/energy/intel/research type. And Ships have a shipType, which stands for the ship function or role.
So type would be consistent and looks fine to me as well.

And while I would be fine to rename buildingId and shipClassId to something else than a 'type', I would want to keep it consistent at least for all future documentation and I am not sure what name to use. :???:

To cause even more confusion, at asm<41874E> the shipType (function/role) is mapped to text values and printed with a "design" label... :mad:

Spocks-cuddly-tribble wrote: Sun Oct 05, 2025 4:18 pm ORDER_DIPLOMACY_typeId_5 +38h -> treatyType -> same as treaty+30h -> sub_45C6F8 used for new treaties
Thanks! I updated documentation on this. :grin:
Edit: By asm<45C7B1> the diplomatic orders actually turned out to fully contain the treaty data structure! :shock:
Edit2: By that subroutine, ResultHeader + 0x0C-0x0D is a notification mask of known empires for peace and alliance treaties, c.f. asm<45C7F5,45C80E,45C875,45C9CC>. I updated the result documentation accordingly.
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 2175
Joined: Sun Apr 27, 2008 2:00 am

Re: Extending the shiplist

Post by Spocks-cuddly-tribble »

Flocke wrote: Mon Oct 06, 2025 4:32 amBy that subroutine, ResultHeader + 0x0C-0x0D is a notification mask of known empires for peace and alliance treaties
There is still some GUI code glitches e.g. you can propose war pacts vs unknown empires and when proposing empire affiliation text changes to friendship.

Hotfixed the 'move stranded fleets due to ended treaty fix' by not ending old treaty when signing alliance in sub 45C6F8. Issue was when this code terminates affiliations, my code determines loss of shared range and wrongfully moves ships before alliance replaces the current affiliation treaty.
I hope this causes no issues if the pre-alliance treaty was time restricted and/or coupled with credits per turn.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is....
Post Reply

Return to “Extending the shiplist”