UE Map Editor

[thunderchero]

And on another note, testing the removal of task forces, I noticed that it indeed speeds ai turn processing time alot!
Adding and removing task forces or altering orders therefore could help further analysing this.

could you elaborate more on this?

my first thought would be to add code to force AI to not read old orders or clear old order before process begins.

[Spocks-cuddly-tribble]

the removal of task forces, I noticed that it indeed speeds ai turn processing time alot!
Adding and removing task forces or altering orders therefore could help further analysing this.

One main factor of AI turn time is [(number of AI ships) * (number of reachable sectors) ]^X

So yes, it's common knowledge that removing/destroying AI ships and reducing range/speed can help.

Also clearing AI orders might cause the illusion of an improvement due to leaving some ships idle/ignored for some time?


Here you go for the asm-application of your analysed AI files (remember the AI map data from my link is a parameter):

4037D6 call sub_405F70 load_main_AI_control (loops for all and executes the below) -> removing this skips any further NEW AI orders (not only for ships)


Empires: sub_407250 -> for ships 1.) sub_4070D0 2.) sub_407020 3.) sub_4101D0 new_AI_Ship_Tasks_Empires (here is your main problem)

Minors: sub_407FE0

Monster: sub_4336D0

force AI to not read old orders or clear old order before process begins.

Removing this might do what you want: 407132 jnz short loc_407110

[thunderchero]

Removing this might do what you want: 407132 jnz short loc_407110

this code change had no effect on turn processing, I misread Flocke post I thought he had already tested changing orders of a taskforce.

I did some manually editing of GWTForce/GTForceList and it also had no effect.

[Flocke]

sry for confusion, I didn''t start any further investigation, but just wanted to reaffirm common knowledge and state that you could utilize UE now to e.g. specifically remove or add task forces or clear orders and see what effect it has

I merged my changes to the master branch now, but there is still one issue to solve:

-------------------------------------
ST:BOF Mon May 24 13:59:00 2021

Version Under Test: 72
File: ..\..\source\game\solarapi.c, Line: 2792, System is not dest or source of trade route
Initialize State: 31
Player Empire: 3 Starting Seed: 1329645910
Galaxy shape: 0 Galaxy Size: 2
Turn State: 20
Turn Number: 206

The trade route crash happened a few turns after I moved the Largo system and is raised by traderoute_effects_credits (sub_442570).
All the other sav game corruption issues I think I have fixed by now.

Checking on the code it might depend on the system order, so I'd have to reassign all the system ids and all the references to it.
But doing some testing with UE I found another explanation. When in the SavGame class I disabled the systInfo parsing and then compared results with exported generic file descryption, I found UE accidentally zeroed some identifier values. Further there might be some missing variables to update in the systInfo file. So I'm currently analysing this.

[thunderchero]

did you ever test build queue project if it will effect map editing?

since I expect you are only changing some of the offsets in the sysinfo file (before build queue info) it should have no effect?

[Flocke]

no, testing the many mods and patches I rather leave to you
any change that changes static sav data structures most likely break UE

when you maintain the data structures but increase a counter or offset and size value UE uses like indexing ship models from stbof.res, and accordingly add new entries to such dynamic lists, UE should handle this just like the game does

dynamic lists are always reffered by some counter, structure size, or address offsets
some value that tells how to read those data and determine the next following data address

files like systInfo is a dynamic sized structure that contains both static data and dynamic lists
the dynamic lists I know is:
the planet entry list: here the planet number tells how many of those static sub data structures are contained
the traderoute list: indexed by the number of incoming trade routes + number of system trade routes
all the rest is static, see sav game documentation

the cases where you could increase static data structure sizes to add new stuff is rare, and in any such case it still would break data value offsets following the change, so the data would be mis-interpreted by UE

to fix such issues I'd need a clear reference from the sav files how to interpret those files that both keep valid for modification and vanilla as well as any other mod, to either convert it to a dynamic list or to implement some case switch for the different static structures

if there is no count or such available yet, you need to find some empty space that can be re-used
but the most will either break the game or are overridden during load and save

alternatively you could add some optional file that can be read by UE to adapt to those changes, be it some file directly witten to the sav or some separate one read from install path
the file must tell what sav file is affected and what change to apply
the change to apply might be a plain identifier for that the change is hard-coded to UE, or it must include more detailed information like the data offset and what size to skip

to simplify reading such configuration files, keep the file format consistent and clear. best use some known and already supported one like the ini file format

[thunderchero]

the cases where you could increase static data structure sizes to add new stuff is rare, and in any such case it still would break data value offsets following the change, so the data would be mis-interpreted by UE

this is where I think we got lucky with build queue project. even though it is a static build queue area, the increase was the last entry of each system, so no prior offsets was changed. only the size of area and offsets used by build queue within that area

I have not download your latest changes yet, I last downloaded and built master on 5-18. UE still reads all system data that I can see. but since I expect there still was issues I did not test moving systems yet.

[Flocke]

this is where I think we got lucky with build queue project. even though it is a static build queue area, the increase was the last entry of each system, so no prior offsets was changed. only the size of area and offsets used by build queue within that area

wrong, there comes the next system

I have not download your latest changes yet, I last downloaded and built master on 5-18. UE still reads all system data that I can see. but since I expect there still was issues I did not test moving systems yet.

reason why UE manages to list the systems is that current UE implementation is broken and does some try and error checks on the expected system id
this however needs to be fixed cause when you move systems, the system ids are out of order the next time the file is read
that case from what I've seen on the code, the whole moved system would be skipped.
But I didn't check on it yet, just can tell that it works by error not by purpose.

[Spocks-cuddly-tribble]

If UE changes system IDs when moving star systems all IDs for outgoing and incoming trade routes have to be adapted to prevent the saved game crash.

0x00-0x03 orderId:  similar to the resultId from result.lst this looks to be incremented starting from a race offset [raceId*max_uint/6]
0x04-0x07 turn:     likely the turn this order got issued, where 0 ist first turn (C1=>194, current=195) (already known)
0x08-0x0B race:     the race id this event is related to (already known)
0x10-0x13 type:     the order type
                      0 for system orders (already known)
                      1 ?? none in save7, but refer result.lst
                      2 for ship or task force orders (already known)
                      3 ?? none in save7, but refer result.lst
                      4 ?? by guess for intel results, refer result.lst
                      5 for diplomatic orders
                      7 for cancel build orders

ordInfo 0x10-0x13 order type: 7 = add/remove or new trade route (so cancel build should by type 7 OF system order type 0?)


[trdeInfo+10h] = credit output (AI gift attitude bonus) of incoming trade routes (0.05 * pop source system)



In case the problem is an AI order:

There is some interesting stuff hidden in sub_40DDC0:

sub_40D260 AI_set_Trade_Routes -> sub_463800AI_set_trade_route_order_7

sub_40D170 AI_New_TF_shipfunction_Sorting_War -> AI ship handling stuff



AITask (lenght 0xEC)

0x14-0x4D: 14 code pointer max 9 used (e.g. from 58BF88 Explore, 58C03C Colonize, 58F9B8 Raid)
0x4C-0x4E: Task modifier via agenda

0xE0-0xE3: Turn Number when set



ship header 0x20-0x27 = "some random seed overridden each app launch" -> Possible purpose for such a feature?



[GTForceList] -> Original BotF source code labels: (TaskForceSummary sub_447CF0)

+2Ch tMission (order/command)
+30h tMissionData (e.g. for TT system attack) & tMove Dest (if not -1) via sector.lst
+40h tAbilityMask (not Flags) only 16 bit read



GTForceList vs GWTForce (next turn vs current) coordinates [+38h/+3Ch] vs ordInfo type 2 -> A mess:

Due to a bug or feature the AI sometimes reads enemy GTForceList coordinates (i.e. next turn data) when setting tasks (e.g. sub_42CB00).

So by ID processing order the rom AI might know next turn data of players and all other AI's, but the card AI just player data?

Altered TF movements when reloading a saved game however are not anticipated by AI. So there must be some sort of out of sync data mess for one turn?




@ thunderchero

systInfo
Offset 0x0004 to 0x0007 uninhabited = 00 00 00 00, AI = 00 00 00 01, player = 00 00 00 02
[if it's a players system: Offset afterwards = data for production Queue, length = 0x006C]

So I guess UE checks IF [+4]=2 THEN add/skip 6C bytes. Now it would need you to store new length in an unused systInfo location....

[Flocke]

UE does not alter the system ids, that why they automatically adapt to new system location.
It might however be that the subroutine of the error requires the ids to be in order to look up the correct systInfo. Right now however I rather tend to think that either the overridden systeInfo values corrupt the trade routes or that there are some other id references to update. Possibly some missed sector index in systInfo. I havn't checked on it yet.

An alternative to find an unused systInfo location btw is to add in some marker value. e.g. we know that systInfo always starts with the system id of first system. It would be very unlikely to ever read "FF FF FF FF", so when you can add in some value like this, we can completely redefine how the file is getting read. Actually, why not just increase the version file's version number and feature that for some new file format where we put in all the missing size infos for possible changes to apply? Afterall that's what the version number was meant for.

[Spocks-cuddly-tribble]

requires the ids to be in order to look up the correct systInfo.

Yes, systInfo entries are loaded via star system ID used as index value.

I assumed UE changes only the system coordinates in entries, but not the system entry order i.e. system ID values in entries.


Writting an extra single byte value into systInfo entries with build queue data is very simple. But feel free to go ahead with your other suggestions.

[Flocke]

requires the ids to be in order to look up the correct systInfo.

Yes, systInfo entries are loaded via star system ID used as index value.

I assumed UE changes only the system coordinates in entries, but not the system entry order i.e. system ID values in entries.

Ah no, sry, what I meant is that the trade route code might expect the system id of the sectors to be in order of the sectors, counted left to right, top to bottom. This could be in case it iterates the sectors and looks up systInfo by the iteration counter instead of the system ID.
Like said, I however rather expect some missed sector index value that needs to be updated.

[thunderchero]

Hi Flocke,

I don't know if you found the issue for "System is not dest or source of trade route"

this issue may not have anything to do with moving systems, the issue in this post can be duplicated in a different way

viewtopic.php?p=32923#p32923

example; if you edit credits (credit can be edited on any empire) the romulan empire name and emperor name will go blank next turn on auto save sometimes. but it is always the romulans this happens to.

name.jpg (109.25 KiB) Viewed 3564 times

Edit; I just downloaded and built latest files and still got same issue.

[Flocke]

interesting, I've seen the name vanished one time as well along with the whole ai agent entry and already wondered how that might have happened
I still have that save and just checked for it, on next turn the romulan empire name indeed is cleared out

think I will have to check other files for possible corruption as well

I edited the credits from 5354 to 85354 cause I noticed that they scrap added task forces when they don't have enough money left
even when they are currently loosing a war and should be glad for the reinforcements...

[Flocke]

I don't know if you found the issue for "System is not dest or source of trade route"

this issue may not have anything to do with moving systems, the issue in this post can be duplicated in a different way

I continued to investigate this. This seems to occure whenever you touch the empsInfo file. Already when I extract it and add it back without having changed anything, the next autosave looses the romulan empire info.

I further completed analysis of the systInfo file, fixed one missed index variable, and added all the info to UE (not merged to the master branch yet), but to no effect. I then analysed the empsInfo file and patched the missing remaining credits value, but that is not of relevance and had no effect either.
The trdeInfo file I analysed and added to UE as well, but again to no avail. And I found no other missed reference on the trade route ids or sectors either.

Bypassing the complete file parsing but defaulting to the generic file support to compare files for other missed or corrupted values, I found no further issues either. Therefore I now feel confident that the file parsing is all correct now. Something that had to be checked anyway, same like analysing and patching above files.

Continuing investigations on the trade route bug, I then checked on the actually encrypted file data.
By coincidence both the corrupted and original empsInfo file have same encrypted size, which usually is not the case cause the trek.exe algorithm differs some little on the dictionary match making. But given this coincidence I could easily exchange both enrypted file data and even with the original empsInfo data, which still loads fine, the auto.sav keeps getting corrupted. Copying it the other way from patched to original sav, the auto.sav remains intact.

Next thing I'll try, is to implement some fixed sorting on the sav content files. While by trek.exe they are stored in a distinct sequence, UE moves touched files to the bottom of the sav file list. For file comparison I've never been happy on it anyhow.