Turn Processing again.....I need help
Moderator: thunderchero
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Turn Processing again.....I need help
Hi Everyone,
I was going though some of my old notes and it started some testing on turn processing again.
this sub routine looks to be the main culprit.
proc AI_AIAgent_405F70 near
but this sub calls "HashTable_Search" at 405F82, if this code is noop turn processing is instantaneous again
at 0x5382 change E8 09 45 11 00 -> 90 90 90 90 90
now this will disable AI ship movements/orders but rest of AI and game will continue and does not cause CTD
I am guessing this code repeats for each ship and each possible order to determine the best order to use.
anybody have any ideas?
is there an error in that code that can be fixed and keep all AI ship movements/orders?
thunderchero
I was going though some of my old notes and it started some testing on turn processing again.
this sub routine looks to be the main culprit.
proc AI_AIAgent_405F70 near
but this sub calls "HashTable_Search" at 405F82, if this code is noop turn processing is instantaneous again
at 0x5382 change E8 09 45 11 00 -> 90 90 90 90 90
now this will disable AI ship movements/orders but rest of AI and game will continue and does not cause CTD
I am guessing this code repeats for each ship and each possible order to determine the best order to use.
anybody have any ideas?
is there an error in that code that can be fixed and keep all AI ship movements/orders?
thunderchero
- Flocke
- BORG Trouble Maker
- Posts: 3246
- Joined: Sun Apr 27, 2008 2:00 am
- Location: Hamburg, Germany
- Contact:
Re: Turn Processing again.....I need help
Cool, we suspected it's related to the ship movement calculations for long!
Given the AI ship movement is rather stupid anyway I guess the best go is to replace it and implement something new.
How to place ships is known from the savegames, however more analysis is needed to figure all the side dependencies.
E.g. there likely are calls when a ship encounters some minor race during ship movement and such.
Identifying the routine is just the start of the work and HashTable_Search might just be part of it aborting some outer calculations.
Hard to tell without of a deeper analysis on the asm code.
But thumbs up for the find
Given the AI ship movement is rather stupid anyway I guess the best go is to replace it and implement something new.
How to place ships is known from the savegames, however more analysis is needed to figure all the side dependencies.
E.g. there likely are calls when a ship encounters some minor race during ship movement and such.
Identifying the routine is just the start of the work and HashTable_Search might just be part of it aborting some outer calculations.
Hard to tell without of a deeper analysis on the asm code.
But thumbs up for the find
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Re: Turn Processing again.....I need help
I would be happy with the same old stupid way, if it was a simple fix to speed it up.Flocke wrote:Given the AI ship movement is rather stupid anyway
BTW it is a very small sub-section with only 1 other sub-section call and that sub-section looks simple also.
but this is me looking at it and I have no idea really what I am looking at.
- cleverwise
- Captain
- Posts: 608
- Joined: Sun Jul 28, 2013 3:09 pm
- Location: 127.0.0.1
Re: Turn Processing again.....I need help
I wish I knew this game construct. I would like to contribute but my programming skills aren't in the language used by BotF.
I personally don't mind the delays, although little to none would be better of course. My biggest dislike is the constant crashing as the game goes on.
I personally don't mind the delays, although little to none would be better of course. My biggest dislike is the constant crashing as the game goes on.
Hailing frequencies close,
Jeremy
Whether you rise or fall depends on... you!
Jeremy
Whether you rise or fall depends on... you!
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Re: Turn Processing again.....I need help
I have not experienced an ingame crash in so long since I started using DxWndcleverwise wrote:My biggest dislike is the constant crashing as the game goes on.
- cleverwise
- Captain
- Posts: 608
- Joined: Sun Jul 28, 2013 3:09 pm
- Location: 127.0.0.1
Re: Turn Processing again.....I need help
Hmmm... Thanks for the feedback.thunderchero wrote:I have not experienced an ingame crash in so long since I started using DxWndcleverwise wrote:My biggest dislike is the constant crashing as the game goes on.
Hailing frequencies close,
Jeremy
Whether you rise or fall depends on... you!
Jeremy
Whether you rise or fall depends on... you!
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Re: Turn Processing again.....I need help
Interesting,
Since we suspect it is AI ship movements causing slow turn processing, I thought I would do a test to confirm.
so I edited a large map mod so all ships had a speed of 99 and range of 99 and on turn 2, turn time was 38 sec with only a few ships on the map.
so mods with high speed/long range ships this will cause turn lag sooner.
note; 99 speed with normal range and 99 range with normal speed, both had normal turn processing early in game.
Since we suspect it is AI ship movements causing slow turn processing, I thought I would do a test to confirm.
so I edited a large map mod so all ships had a speed of 99 and range of 99 and on turn 2, turn time was 38 sec with only a few ships on the map.
so mods with high speed/long range ships this will cause turn lag sooner.
note; 99 speed with normal range and 99 range with normal speed, both had normal turn processing early in game.
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Re: Turn Processing again.....I need help
Hi Everyone,
this is an old topic but feel it had best info on turn processing.
I did some testing the past few days, here are my results.
tests were done on 1366 x 768 UDML T5, irregular, large, 62 x 31, many minors, impossible, random and borg off
all tests use same random seed so majors and minor placement was the same.
images provided to show how AI reacted to changes.
first test unedited turn 76 (54.3 sec) base time no minors turn 76 (35.6) -18.7 sec all minor ships range set to short turn 76 (43.7) -10.6 sec without 18 fleet task force turn 76 (48.7) -5.6 sec vanilla range turn 76 (26.7) -27.6 I would not expect a 50% decrease all the time, but it just goes to show how just making a few changes will increase turn processing.
In my Debugging tests at 49331F
the code will always jump around this code statement.
So I expect since our systems now have enough memory for game, game no longer use Smartheap?
this give me some hope that if someone had the skills turn processing still might be able to be at least improved.
here are some other locations I found of interest
at 493310
System_Memory_InitializeMemory
at 49333C
total memeory pool? if so that would be 10.48576 mb
in same sub it breaks it down into sub pools
at 49338D
if I read it correctly each pool is same size 0.03277 mb?
example AIPool
my main interest is,
AIPool
TacticalPool
NetworkPool
these 3 area really need improvement/increase?
next area I found of interest I spoke of earlier in this topic at 405F70
the HashTable_Search create only has a size of 20? at 51A2E0
the other thing was the gdlList_Create with a size of 10? at 51B470
I have attempted to increase the size of all the above, but saw no improvement
but that did not surprise me much since I expect data added was never increased in those pools/lists. I fear that is beyond my ability.
So here is the info I collected, I hope someone, someday maybe able to use it.
thunderchero
this is an old topic but feel it had best info on turn processing.
I did some testing the past few days, here are my results.
tests were done on 1366 x 768 UDML T5, irregular, large, 62 x 31, many minors, impossible, random and borg off
all tests use same random seed so majors and minor placement was the same.
images provided to show how AI reacted to changes.
first test unedited turn 76 (54.3 sec) base time no minors turn 76 (35.6) -18.7 sec all minor ships range set to short turn 76 (43.7) -10.6 sec without 18 fleet task force turn 76 (48.7) -5.6 sec vanilla range turn 76 (26.7) -27.6 I would not expect a 50% decrease all the time, but it just goes to show how just making a few changes will increase turn processing.
In my Debugging tests at 49331F
Code: Select all
0049331F B9 54 CC 57 00 mov ecx, offset aSmartheapIniti ; "SmartHeap Initializing Memory"
So I expect since our systems now have enough memory for game, game no longer use Smartheap?
this give me some hope that if someone had the skills turn processing still might be able to be at least improved.
here are some other locations I found of interest
at 493310
System_Memory_InitializeMemory
at 49333C
Code: Select all
0049333C B8 00 00 A0 00 mov eax, 0A00000h ; size
in same sub it breaks it down into sub pools
at 49338D
Code: Select all
0049338D E8 9E FD FF FF call System_Memory_InitStrategicPool
00493392 E8 B9 FC FF FF call System_Memory_InitDBPool
00493397 E8 24 FD FF FF call System_Memory_InitUIPool
0049339C E8 3F FC FF FF call System_Memory_InitAIPool
004933A1 E8 FA FD FF FF call System_Memory_InitSoundPool
004933A6 E8 D5 FE FF FF call System_Memory_InitTacticalPool
004933AB E8 60 FE FF FF call System_Memory_InitNetworkPool
example AIPool
Code: Select all
00492FE3 68 02 80 00 00 push 8002h
AIPool
TacticalPool
NetworkPool
these 3 area really need improvement/increase?
next area I found of interest I spoke of earlier in this topic at 405F70
Code: Select all
00405F82 E8 09 45 11 00 call HashTable_Search
Code: Select all
0051A306 83 C0 20 add eax, 20h ; size
Code: Select all
0051B4B0 B8 10 00 00 00 mov eax, 10h ; size
but that did not surprise me much since I expect data added was never increased in those pools/lists. I fear that is beyond my ability.
So here is the info I collected, I hope someone, someday maybe able to use it.
thunderchero
- Spocks-cuddly-tribble
- Code Master
- Posts: 1926
- Joined: Sun Apr 27, 2008 2:00 am
Re: Turn Processing again.....I need help
SmartHeap itself is not cause of the turn-lag, but an unmet workaround (it increases the speed dynamic memory is allocated / deallocated and might prevent RAM fragmentation?). I.e. disabling / removing SmartHeap could slow down the game even more, but not by much. EDIT: Flocke mentioned temp files for tactical combat, SmartHeap might also affect these for larger battles?thunderchero wrote: ↑Wed May 05, 2021 10:00 pmSo I expect since our systems now have enough memory for game, game no longer use Smartheap?
this give me some hope that if someone had the skills turn processing still might be able to be at least improved.
(...) I have attempted to increase the size of all the above, but saw no improvement
but that did not surprise me much since I expect data added was never increased in those pools/lists. I fear that is beyond my ability.
Pakled translation: They crippled the source code itself at all sub-levels to limit ram usage. I.e. the same (or very similar) data gets created and deleted all over again in recursive (almost infinity) loops. This does not only slow down the game, but also renders the AI very stupid via comparing values of almost no significance. So increasing / changing the memory allocations won't change anything, at least whithout rewriting the ENTIRE AI code in order to use the memory in a smart way.
EDIT: Also you might want to look at my latest IDA database. Your 'HashTable_Search' = get AI files and 'gdlList_Create' just prepares some ingame lists. So some of your references do not even refer to memory allocations. Strange that the AI still worked with suchlike messups.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Re: Turn Processing again.....I need help
My theory was if those table/lists could be greatly increased the AI would not have to create/destroy them over and over and this would improve turn processing time.
- Spocks-cuddly-tribble
- Code Master
- Posts: 1926
- Joined: Sun Apr 27, 2008 2:00 am
Re: Turn Processing again.....I need help
And you shall receive. We found something. A chance, maybe....
Subroutine 426D90 happens to be our first hit deep into BotF's AI redeploying task forces calculations aka turn lag.
The global loop loops 128 times over all AI ships on the map:
Code: Select all
00426EC7 cmp eax, 80h // 0x262C8
00426ECC jl loc_426DA5h // 0x262CC 0F 8C D3 FE FF FF
The following ship sorting deviation fix might also help a bit (part of above named main loop): (cf: viewtopic.php?p=50855#p50855)
Code: Select all
NAME: AI ship sorting by function - deviation fix
DESC: Corrects a cruiser (02) vs artillery sorting deviation in ds:58AEF4 vs sub_4797C0.
AUTHOR: Spocks-cuddly-tribble
URL: https://www.armadafleetcommand.com/onscreen/botf/viewtopic.php?p=54732#p54732
>> 0x188CFA 01 00 02
<< 0x188CFA 02 00 01
# 0x188CF6 04 00 03 00 01 00 02 00 00 00 06 00 FF FF FF FF FF FF 05 00 -> lower more powerful
# cf479841 03 00 04 00 05 00 0A 00 0F 00 01 00 08 00 14 00 1E 00 02 00 -> higher more powerful (combat.bin bug)
I hope this helps.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Re: Turn Processing again.....I need help
That is very interesting,
I did a quick test on a saved game from Galaxy mod on turn 169 with a turn processing time of 8:20
I Decreased the loop count at 0x262C8 80 -> 01 turn processing on same saved turn was 2:30
Edit; I Decreased the loop count at 0x262C8 80 -> 00 turn processing on same saved turn was 1:58
I have all saved games after turn processing to compare, but just don't have the time right now. (just like I need to test/add other changes for my mods).
Also need to test ship sorting deviation fix.
Great find....
I did a quick test on a saved game from Galaxy mod on turn 169 with a turn processing time of 8:20
I Decreased the loop count at 0x262C8 80 -> 01 turn processing on same saved turn was 2:30
Edit; I Decreased the loop count at 0x262C8 80 -> 00 turn processing on same saved turn was 1:58
I have all saved games after turn processing to compare, but just don't have the time right now. (just like I need to test/add other changes for my mods).
Also need to test ship sorting deviation fix.
Great find....
- Spocks-cuddly-tribble
- Code Master
- Posts: 1926
- Joined: Sun Apr 27, 2008 2:00 am
Re: Turn Processing again.....I need help
Glad it works. Albeit everything above 20 seconds seems kind of unplayable. I think about 40x30 sectors is map limit for acceptable AI flaws and turn processing time.
Btw, while testing an AI one-per-empire buildings fix, I found the vanilla klingon T8 AI able to upgrade to type 9 industry. Due to the two byte value limit of build cost / industry invested this shouldn't be possible in unfixed vanilla. So I have to revisit your suspicions about further unknown AI building/buying cheats. If at all, it's not a cheat but a well hidden data corruption of overwritten system tasks.
Btw, while testing an AI one-per-empire buildings fix, I found the vanilla klingon T8 AI able to upgrade to type 9 industry. Due to the two byte value limit of build cost / industry invested this shouldn't be possible in unfixed vanilla. So I have to revisit your suspicions about further unknown AI building/buying cheats. If at all, it's not a cheat but a well hidden data corruption of overwritten system tasks.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
- thunderchero
- Site Administrator aka Fleet Admiral
- Posts: 7934
- Joined: Fri Apr 25, 2008 2:00 am
- Location: On a three month training mission, in command of the USS Valiant.
Re: Turn Processing again.....I need help
I did a test with value set to 01,
test was on vanilla large irregular t1 impossible many minors random and borg off (100 turns)
Note; this had no human interactions during test.
The AI preformed ok? (it is so hard to tell)
AI had basically the same amount of systems
AI expanded about the same
AI basically had the same amount of ships (except Klingons, it looked like they just had a major battle with Ferengi and lost fleet and 1 system with patch)
AI had the same relationships with other AI races.
So first test did look promising. but still AI is still very weak (so how much worse could it really be?).
test was on vanilla large irregular t1 impossible many minors random and borg off (100 turns)
Note; this had no human interactions during test.
The AI preformed ok? (it is so hard to tell)
AI had basically the same amount of systems
AI expanded about the same
AI basically had the same amount of ships (except Klingons, it looked like they just had a major battle with Ferengi and lost fleet and 1 system with patch)
AI had the same relationships with other AI races.
So first test did look promising. but still AI is still very weak (so how much worse could it really be?).
- Spocks-cuddly-tribble
- Code Master
- Posts: 1926
- Joined: Sun Apr 27, 2008 2:00 am
Re: Turn Processing again.....I need help
A valid point. I don't expect any AI improvements, it's just about worsenings.thunderchero wrote: ↑Thu Sep 09, 2021 7:30 pmAI is still very weak (so how much worse could it really be?).
Ship sorting deviation fix might help with AI task forces composition wrt range (e.g. merge strike & command more often -> vanilla short range), but I wouldn't bet on it.
I'd test with zero iterations (default of 128 looks like a remainder of an unused feature like e.g. combat.bin ship vs. ship odds table).
Pay attention to always start/load another game and after that reload turn 1 saved game for test one(to control deviations of random seeds and saved game dynamic data bugs). I messed up many tests in early days before I realized this cause for deviations.
EDIT; I ran some quick tests by myself and the iterations definitely change AI behavior. As a rule of a thumb I'd say more iterations improve global offensives, but there is still bugged/broken codes in the main loop. We are getting closer to read/fix those codes, but we are not there yet.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.