Turn Processing again.....I need help

You can talk about anything. (please read forum rules before posting)

Moderator: thunderchero

User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Turn Processing again.....I need help

Post by thunderchero »

Hi Everyone,

I was going though some of my old notes and it started some testing on turn processing again.

this sub routine looks to be the main culprit.

proc AI_AIAgent_405F70 near

but this sub calls "HashTable_Search" at 405F82, if this code is noop turn processing is instantaneous again

at 0x5382 change E8 09 45 11 00 -> 90 90 90 90 90

now this will disable AI ship movements/orders but rest of AI and game will continue and does not cause CTD :wink:
I am guessing this code repeats for each ship and each possible order to determine the best order to use.

anybody have any ideas?

is there an error in that code that can be fixed and keep all AI ship movements/orders?

thunderchero
User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 3178
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Turn Processing again.....I need help

Post by Flocke »

Cool, we suspected it's related to the ship movement calculations for long!

Given the AI ship movement is rather stupid anyway I guess the best go is to replace it and implement something new.
How to place ships is known from the savegames, however more analysis is needed to figure all the side dependencies.
E.g. there likely are calls when a ship encounters some minor race during ship movement and such.
Identifying the routine is just the start of the work and HashTable_Search might just be part of it aborting some outer calculations.
Hard to tell without of a deeper analysis on the asm code.

But thumbs up for the find :up:
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Re: Turn Processing again.....I need help

Post by thunderchero »

Flocke wrote:Given the AI ship movement is rather stupid anyway
I would be happy with the same old stupid way, if it was a simple fix to speed it up. :wink:

BTW it is a very small sub-section with only 1 other sub-section call and that sub-section looks simple also.

but this is me looking at it and I have no idea really what I am looking at.
User avatar
cleverwise
Captain
Captain
Posts: 608
Joined: Sun Jul 28, 2013 3:09 pm
Location: 127.0.0.1

Re: Turn Processing again.....I need help

Post by cleverwise »

I wish I knew this game construct. I would like to contribute but my programming skills aren't in the language used by BotF.

I personally don't mind the delays, although little to none would be better of course. My biggest dislike is the constant crashing as the game goes on.
Hailing frequencies close,

Jeremy
Whether you rise or fall depends on... you!
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Re: Turn Processing again.....I need help

Post by thunderchero »

cleverwise wrote:My biggest dislike is the constant crashing as the game goes on.
I have not experienced an ingame crash in so long since I started using DxWnd
User avatar
cleverwise
Captain
Captain
Posts: 608
Joined: Sun Jul 28, 2013 3:09 pm
Location: 127.0.0.1

Re: Turn Processing again.....I need help

Post by cleverwise »

thunderchero wrote:
cleverwise wrote:My biggest dislike is the constant crashing as the game goes on.
I have not experienced an ingame crash in so long since I started using DxWnd
Hmmm... Thanks for the feedback.
Hailing frequencies close,

Jeremy
Whether you rise or fall depends on... you!
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Re: Turn Processing again.....I need help

Post by thunderchero »

Interesting,

Since we suspect it is AI ship movements causing slow turn processing, I thought I would do a test to confirm.

so I edited a large map mod so all ships had a speed of 99 and range of 99 and on turn 2, turn time was 38 sec with only a few ships on the map. :shock:

so mods with high speed/long range ships this will cause turn lag sooner.

note; 99 speed with normal range and 99 range with normal speed, both had normal turn processing early in game.
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Re: Turn Processing again.....I need help

Post by thunderchero »

Hi Everyone,

this is an old topic but feel it had best info on turn processing.

I did some testing the past few days, here are my results.

tests were done on 1366 x 768 UDML T5, irregular, large, 62 x 31, many minors, impossible, random and borg off

all tests use same random seed so majors and minor placement was the same.
images provided to show how AI reacted to changes.

first test unedited turn 76 (54.3 sec) base time
unedited.jpg
unedited.jpg (509.78 KiB) Viewed 4184 times
no minors turn 76 (35.6) -18.7 sec
no_minors.jpg
no_minors.jpg (504.55 KiB) Viewed 4184 times
all minor ships range set to short turn 76 (43.7) -10.6 sec
minor_short.jpg
minor_short.jpg (511.69 KiB) Viewed 4184 times
without 18 fleet task force turn 76 (48.7) -5.6 sec
no_18.jpg
no_18.jpg (531.14 KiB) Viewed 4184 times
vanilla range turn 76 (26.7) -27.6
vailla_range.jpg
vailla_range.jpg (508.36 KiB) Viewed 4184 times
I would not expect a 50% decrease all the time, but it just goes to show how just making a few changes will increase turn processing.

In my Debugging tests at 49331F

Code: Select all

0049331F B9 54 CC 57 00          mov     ecx, offset aSmartheapIniti ; "SmartHeap Initializing Memory"
the code will always jump around this code statement. :shock:
So I expect since our systems now have enough memory for game, game no longer use Smartheap?

this give me some hope that if someone had the skills turn processing still might be able to be at least improved.

here are some other locations I found of interest
at 493310
System_Memory_InitializeMemory

at 49333C

Code: Select all

0049333C B8 00 00 A0 00          mov     eax, 0A00000h   ; size
total memeory pool? if so that would be 10.48576 mb

in same sub it breaks it down into sub pools
at 49338D

Code: Select all

0049338D E8 9E FD FF FF          call    System_Memory_InitStrategicPool
00493392 E8 B9 FC FF FF          call    System_Memory_InitDBPool
00493397 E8 24 FD FF FF          call    System_Memory_InitUIPool
0049339C E8 3F FC FF FF          call    System_Memory_InitAIPool
004933A1 E8 FA FD FF FF          call    System_Memory_InitSoundPool
004933A6 E8 D5 FE FF FF          call    System_Memory_InitTacticalPool
004933AB E8 60 FE FF FF          call    System_Memory_InitNetworkPool
if I read it correctly each pool is same size 0.03277 mb?

example AIPool

Code: Select all

00492FE3 68 02 80 00 00          push    8002h
my main interest is,
AIPool
TacticalPool
NetworkPool

these 3 area really need improvement/increase?

next area I found of interest I spoke of earlier in this topic at 405F70

Code: Select all

00405F82 E8 09 45 11 00          call    HashTable_Search
the HashTable_Search create only has a size of 20? at 51A2E0

Code: Select all

0051A306 83 C0 20                add     eax, 20h        ; size
the other thing was the gdlList_Create with a size of 10? at 51B470

Code: Select all

0051B4B0 B8 10 00 00 00          mov     eax, 10h       ; size
I have attempted to increase the size of all the above, but saw no improvement

but that did not surprise me much since I expect data added was never increased in those pools/lists. I fear that is beyond my ability.

So here is the info I collected, I hope someone, someday maybe able to use it.
thunderchero
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 1870
Joined: Sun Apr 27, 2008 2:00 am

Re: Turn Processing again.....I need help

Post by Spocks-cuddly-tribble »

thunderchero wrote: Wed May 05, 2021 10:00 pmSo I expect since our systems now have enough memory for game, game no longer use Smartheap?

this give me some hope that if someone had the skills turn processing still might be able to be at least improved.

(...) I have attempted to increase the size of all the above, but saw no improvement

but that did not surprise me much since I expect data added was never increased in those pools/lists. I fear that is beyond my ability.
SmartHeap itself is not cause of the turn-lag, but an unmet workaround (it increases the speed dynamic memory is allocated / deallocated and might prevent RAM fragmentation?). I.e. disabling / removing SmartHeap could slow down the game even more, but not by much. EDIT: Flocke mentioned temp files for tactical combat, SmartHeap might also affect these for larger battles?

Pakled translation: They crippled the source code itself at all sub-levels to limit ram usage. I.e. the same (or very similar) data gets created and deleted all over again in recursive (almost infinity) loops. This does not only slow down the game, but also renders the AI very stupid via comparing values of almost no significance. So increasing / changing the memory allocations won't change anything, at least whithout rewriting the ENTIRE AI code in order to use the memory in a smart way. :wink:

EDIT: Also you might want to look at my latest IDA database. Your 'HashTable_Search' = get AI files and 'gdlList_Create' just prepares some ingame lists. So some of your references do not even refer to memory allocations. Strange that the AI still worked with suchlike messups. :shock:
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Re: Turn Processing again.....I need help

Post by thunderchero »

My theory was if those table/lists could be greatly increased the AI would not have to create/destroy them over and over and this would improve turn processing time.
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 1870
Joined: Sun Apr 27, 2008 2:00 am

Re: Turn Processing again.....I need help

Post by Spocks-cuddly-tribble »

thunderchero wrote: Tue Feb 14, 2017 7:16 pmTurn Processing again.....I need help
And you shall receive. We found something. A chance, maybe....


Subroutine 426D90 happens to be our first hit deep into BotF's AI redeploying task forces calculations aka turn lag.

The global loop loops 128 times over all AI ships on the map:

Code: Select all

00426EC7                 cmp     eax, 80h    // 0x262C8
00426ECC                 jl      loc_426DA5h // 0x262CC 0F 8C D3 FE FF FF
Decreasing the loop count at 0x262C8 needs to be tested wrt turn time and unwanted changes in AI behavior.



The following ship sorting deviation fix might also help a bit (part of above named main loop): (cf: viewtopic.php?p=50855#p50855)

Code: Select all

NAME: AI ship sorting by function - deviation fix
DESC: Corrects a cruiser (02) vs artillery sorting deviation in ds:58AEF4 vs sub_4797C0.
AUTHOR: Spocks-cuddly-tribble
URL: https://www.armadafleetcommand.com/onscreen/botf/viewtopic.php?p=54732#p54732

>> 0x188CFA 01 00 02
<< 0x188CFA 02 00 01

# 0x188CF6 04 00 03 00 01 00 02 00 00 00 06 00 FF FF FF FF FF FF 05 00 -> lower more powerful
# cf479841 03 00 04 00 05 00 0A 00 0F 00 01 00 08 00 14 00 1E 00 02 00 -> higher more powerful (combat.bin bug)

I hope this helps. :smile:
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Re: Turn Processing again.....I need help

Post by thunderchero »

That is very interesting,

I did a quick test on a saved game from Galaxy mod on turn 169 with a turn processing time of 8:20

I Decreased the loop count at 0x262C8 80 -> 01 turn processing on same saved turn was 2:30

Edit; I Decreased the loop count at 0x262C8 80 -> 00 turn processing on same saved turn was 1:58

I have all saved games after turn processing to compare, but just don't have the time right now. (just like I need to test/add other changes for my mods).

Also need to test ship sorting deviation fix.

Great find.... :grin:
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 1870
Joined: Sun Apr 27, 2008 2:00 am

Re: Turn Processing again.....I need help

Post by Spocks-cuddly-tribble »

Glad it works. :smile: Albeit everything above 20 seconds seems kind of unplayable. I think about 40x30 sectors is map limit for acceptable AI flaws and turn processing time.

Btw, while testing an AI one-per-empire buildings fix, I found the vanilla klingon T8 AI able to upgrade to type 9 industry. Due to the two byte value limit of build cost / industry invested this shouldn't be possible in unfixed vanilla. So I have to revisit your suspicions about further unknown AI building/buying cheats. If at all, it's not a cheat but a well hidden data corruption of overwritten system tasks. :???:
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 7824
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.

Re: Turn Processing again.....I need help

Post by thunderchero »

I did a test with value set to 01,

test was on vanilla large irregular t1 impossible many minors random and borg off (100 turns)

Note; this had no human interactions during test.

The AI preformed ok? (it is so hard to tell)

AI had basically the same amount of systems
AI expanded about the same
AI basically had the same amount of ships (except Klingons, it looked like they just had a major battle with Ferengi and lost fleet and 1 system with patch)
AI had the same relationships with other AI races.

So first test did look promising. but still AI is still very weak (so how much worse could it really be?).
User avatar
Spocks-cuddly-tribble
Code Master
Code Master
Posts: 1870
Joined: Sun Apr 27, 2008 2:00 am

Re: Turn Processing again.....I need help

Post by Spocks-cuddly-tribble »

thunderchero wrote: Thu Sep 09, 2021 7:30 pmAI is still very weak (so how much worse could it really be?).
A valid point. I don't expect any AI improvements, it's just about worsenings.

Ship sorting deviation fix might help with AI task forces composition wrt range (e.g. merge strike & command more often -> vanilla short range), but I wouldn't bet on it.

I'd test with zero iterations (default of 128 looks like a remainder of an unused feature like e.g. combat.bin ship vs. ship odds table).

Pay attention to always start/load another game and after that reload turn 1 saved game for test one(to control deviations of random seeds and saved game dynamic data bugs). I messed up many tests in early days before I realized this cause for deviations. :mad:

EDIT; I ran some quick tests by myself and the iterations definitely change AI behavior. As a rule of a thumb I'd say more iterations improve global offensives, but there is still bugged/broken codes in the main loop. We are getting closer to read/fix those codes, but we are not there yet.
I don't know how many bugs is too many but that point is reached somewhere before however many in BotF is.
Post Reply

Return to “General Chat”