Increasing the trek.exe Palette Limit project

[thunderchero]

Not all are moved, only the used ones read from texture.lst (there is unused palettes by default).

Limit is total number of different palettes used by all textures. his is my question

I.e. palette.lst can be extended as long as max X different palettes are used by the shipt textures (this is our dynamic lists).

the ones I am adding to get palette over 127 are not used by any models they are only on palette index and textures.lst

note; vanilla palette.lst index have values for 80 (128) and 81 (129)

Note:
screensaver and tech screen ship uses 502B88
combat screen uses 495B2F
deleted call for color.lst at

Code: Select all

00502B7E BA 4C 2E 58 00          mov     edx, offset aColor_lst_0 ; "color.lst"

crash in tech and screen but not in combat.

[Flocke]

I went crazy....

I no-op the sub 00539C00 and all 3 3d screens loaded without issues (combat, screensaver, tech screen ships)

Wow, I wouldn't even have tempted to do so!
It however might also be used as a cache and crash the next combat when a) another ship is met or b) the same model and therefore textures is rendered or c) once your graphics card buffer is full (if software rendered you should see your system memory usage grow).

I know it has duplicates but where do the duplicates get removed?

Not relevant is the number of palette.lst entries. What is relevant is to what palettes they map.

Each Texture and each palette entry must be loaded by MPR when it is used. This is a technical need for when rendering with a graphics card. If not software rendered, the handles reference data stored in the graphics memory. And usually the graphics memory is backed by some duplicate in system memory to swap and reload rendered graphics when they are used.

It does not make sense to fill all the graphics buffer with data that is never getting rendered. Therefore the lists of texture and palette handles are only increased when new models are getting rendered. By mapping palette indexes, it will automatically reuse already loaded palettes and thereby eliminate duplicates.

When the buffers hold actual MPR texture handles, the palette index likely won't be used to determine the texture palettes any longer. Once it is loaded to graphics memory, it is likely that each of the loaded texture is backed by a palette of same index.
The textures on the other handl are linked by the loaded hob models. The buffer does not care from where it is loaded abd what id mappings were defined. These are only used for loading needed files to system memory, and not for mapping them in the graphics buffer.

are we looking at dead/not used code?

When the function is nopped, it might happen that some other routine, that is checking for entries of the buffer, will always find the zero initialized buffer and reload textures. That would mean a huge memory leak, but not cause any immediate crash.

Best fire up IDA, set a break point and debug trek.exe to see whether the code is getting called. If it is called, set another break point to the new allocated data buffer, to see whether it is getting read from.

this brings up my second question what is color.lst for?

Vanilla color.lst starts with the entry count of 18Ah = 394 values, followed by 4 bytes set to zero for whatever purpose, and then end with the 394 4 byte color values. Given that the fourth byte of most of the entries is set to FF indicates, that it is little endian ARGB values, with A set to full opacity.

It might be some predefined default palette or something the like.

Another thing that makes me wondering is the rendered galaxy class image. Why doesn't it simply crash as usual? All the pallettes must be loaded or either it crashed or there was a buffer overflow which most of the time would lead to some crash as well.
Is it possible that instead by some data calculation the indices are getting wrong? Like some signed byte calculation where 0x80 is mapped to -1 instead of 128?

As for the 128 entries limited buffer load from sub_539C00 and sub_539C50, I think SCT is right and it actually is a preallocated data array, that is getting loaded and possibly re-allocated over time. If re-allocated, it would only be the 128 initial entries, where from my guess the last entry must remain zero so that the routine can detect the memory end.

[Spocks-cuddly-tribble]

the ones I am adding to get palette over 127 are not used by any models they are only on palette index and textures.lst

note; vanilla palette.lst index have values for 80 (128) and 81 (129)

Yes, they don't have to be used by ships, but just to be listed in texture.lst. I.e. it tries to map all palettes listed in texture.lst.

I also missinterpeted the palette limit once (res file, not total number of different in texture.lst):

I just extended the palette limit via hex editor (no trek.exe code change needed)

hh3_a.gif using new palette index ID 129 81h (i.e. new palette 130, 1024 bytes added to end of palette.lst).


Each palette consists of 256 four byte RGB slots ending with FF each so 1024 bytes per palette.

Palette.lst header is an index list telling which palette to use for each texture in texture.lst.

Both vanilla files (palette/texture.lst) start with the value 299 (number of 36 bytes entries for gif files in texture.lst)

299 * 4 + 8 = 1204 i.e. 0x4B4 = start of first palette data in palette.lst (129 palette entries a 1024 bytes)

E.g. scar1.gif uses palette index ID 80h (i.e. palette 81h)

@ 0x4B0 (299*4+4) max palette index ID +1 aka number of palettes (default 81h) -> updating this value will fix the display glitch when extending palette.lst

[thunderchero]

Best fire up IDA, set a break point and debug trek.exe to see whether the code is getting called. If it is called, set another break point to the new allocated data buffer, to see whether it is getting read from.

big problem there no one has figured out why ida will crash as soon as 3d rendering is entered.
the best I have been able to do is start game normally go to 3d screen then with ida already open with trek.exe loaded (but not running in ida de-bugger) attach process I might have a few seconds before it crashes, that is how I saw GlobalPalette loaded. before attach all 00's after attach palette present.

[Flocke]

big problem there no one has figured out why ida will crash as soon as 3d rendering is entered.

That is an old issue long solved. It is a bit annoying, yes, but just ignore first exception but let the next one pass to the game whenever video is rendered. And for combat I think you needed to ignore one exception per ship load, so better don't have too many ships in combat.

I never debug with attach process, and don't even disable video, even though I'd recommend to disable the video to not be annoyed by the initial load exception. It just would feel incomplete without that video being started.

I am sure I reported on it when I did all my mpr++ analysis

Note that exceptions don't necessarily mean that an application crashed. I've seen programs that misuse these exceptions for the most stupid tasks like checking for whether some file exist or not. It bypasses the call stack but jumps way back to last try/catch block that handles the exception. Only if there is no handler, it will crash the application.

This is a real debugging pain. Even many of the Java library calls use such try/catch handling, which is so much annoying for debugging actual issues!

But given that the most exeptions in BotF must not be passed back to the game, and given that it happens by video and 3D rendering, I guess it is some render driver issue that tries to aquire focus or something but fails due to the attached debugger. Whatever it is, just keep ignoring it...

[Spocks-cuddly-tribble]

I no-op the sub 00539C00 and all 3 3d screens loaded without issues (combat, screensaver, tech screen ships)

'Scrap the Crap' is best approach when patching trek.exe

So works this with no issues with:

- default palettes
- 128+ palettes in texture.lst
- 128+ palettes used by ships
- 128+ palettes used in same battle



If not:

I'd scrap dynamic palette index list (your nop code) + palette dublicates.

In dynamic texture/palette index list (=other still used) just load palette Index ID * 1024 (from edited texture.lst):

Simplify texture.lst -> [texture entry+20] = palette Index ID * 1024

Simplify palette.lst -> scrap entire header = just palette data in 1024 blocks

Search and scrap mountains of unneeded code and switch to palette pointer for MPR+ to:

simplified loaded [palette.lst + palette Index ID * 1024] with (palette Index ID * 1024) read from new dynamic texture list/palette buffer entry

But analysis for this might take ages...

[thunderchero]

I am sure I reported on it when I did all my mpr++ analysis

I might have missed it

but during de-bug over palette I am confused.

when debug hits crash it goes to error log

Code: Select all

trek.exe: The instruction at 0x5CD521F referenced memory at 0x1E0028C. The memory could not be written (0x05CD521F -> 01E0028C)

but on screen it goes to

Code: Select all

mpr565.dll:05CD521F 89 50 0C                                        mov     [eax+0Ch], edx

instruction 5CD521F matches but I can't find matching code in mpr565.dll

ida.jpg (500.72 KiB) Viewed 1166 times

[Spocks-cuddly-tribble]

palette ... I know it has duplicates but where do the duplicates get removed?

Sorry, I missed this question. I think you mean moved to, not removed.

The copies of the palettes themselfs (1024 byte entries) are stored @ dynamic [603EA4] this is your:

.bss:00603EA4 dword_603EA4 dd 0C4D0020h ; DATA XREF: sub_538450+29w
.bss:00603EA4 ; sub_538450:loc_538701w ...

QD had it labeled as "GlobalPalette"

Here the desired palette gets passed to MPR via [603EA4] + dynamic palette index ID * 1024 to edx:

Code: Select all

00538C03           mov   edi, ds:GlobalPalette_loaded_total_size
00538C09           shl   eax, 0Ah                          ; *1024
00538C0C           add   edi, eax
00538C0E           xor   ebx, ebx
00538C10           mov   edx, edi
00538C12           mov   eax, ecx
00538C14           call  mpr_import__TextureEAX_or_PaletteEBX

[612B30] slould be the loaded palette.lst from stbof.res

Your:

I no-op the sub 00539C00 and all 3 3d screens loaded without issues (combat, screensaver, tech screen ships)

might force BotF to copy always all palettes from [612B30] to [603EA4]

This should be no issue these days and will not cause a bad memory leak.

Did you no-op calls for sub 00539C00 or the sub itself?

[thunderchero]

Did you no-op calls for sub 00539C00 or the sub itself?

I no-op the sub itself
but it still crashed on palettes over 127

'Scrap the Crap' is best approach when patching trek.exe

just for the heck of it, I converted all the gif's to bmp's changed texture.lst gif -> bmp
added bmp's and texture.lst and saved

Game did not crash

bmp.jpg (228.23 KiB) Viewed 1142 times

Another thing that makes me wondering is the rendered galaxy class image. Why doesn't it simply crash as usual?

could that have been 512 texture using hades or 1024 on software? (depending on palette would have changed colors)

1024.jpg (254.42 KiB) Viewed 1142 times

the only interesting thing I saw in mpr565.dll was;

Code: Select all

.rdata:10297194 word_10297194   dw 7Fh                  ; DATA XREF: engMessage:loc_10015ED6r

Code: Select all

10015ED6 D9 2D 94 71 29 10                               fldcw   ds:word_10297194

when lowered to 70 would ctd if light cruiser, strike, dread selected in research with feds
when lowered to 65 no fed ship would crash in research
if lowered to 0 crash on start up
if increased to 80 or 90 crash on start up (with or without palette with above or below 127)
if increased to FF no crash 127 or below, crash when entering 3d above 127
I don't think this is limit but thought I would post info

[Flocke]

instruction 5CD521F matches but I can't find matching code in mpr565.dll

lol, at first I missed the middle 5 and wondered why that routine is written to .bss section, but no, the address is 7 letters.

Other than trek.exe, dll files like mpr565 are getting relocated on load. With it by the relocation table all code references are getting mapped to new location. If doing a hex value search, only search for the code instructions, not for the data references or jump addresses.

5CD521F matches loc_1001521F this case.

When paused, you can find the loaded dlls in segments view and in debugger modules view.
When you right click a dll in the moduls view, you can instruct IDA to analyse the loaded module code. Alternatively type 'c' on some hex code that you identify as code.

When you debug trek.exe with unmodified working palettes, you can pause it before combat, load the mpr565.dll module, search the code location of interest, set a breakpoint, and then continue to see when it is getting called and follow instructions to figure the call stack.
There is also a debugger option for tracing the call stack, but that never worked out for me, specially when analysing errors.

E.g. set a breapoint to the return instruction of said routine, then step by instruction or 'run until execution returns from the current function' to see what it has been called by.
For convenience, I usually enable the "Debugger commands" toolbar you can find in View->Toolbars tab.

That way you will find, that in normal execution it is called by:
mpr565_10015610, which is called by
mpr565_1000999C, ...
mpr565_10002E43
mpr565_100156E5
mpr565_10001522
mpr565_1001A271
mpr565_10015EAD (engMessage)
mpr97_1000290F (MPRMessage case 2)
trek_53AE8B (MPRMessage_context_buffer)
trek_53AE0E (call_MPRMessage_context_buffer)
trek_511758 (cc_MPRMessage_context_buffer)
trek_402484 (c_call_Time_render_object)
-> which finally is a method that tells about what it's actally doing here that fails.
From the read it looks like it is aquiring some MPR buffer context to swap the render buffer.
trek_4740CD (Tactical_Combat_Commands_AI_Automatic)

then it starts to loop and render, at least 1-2 seconds
followed by same call sequence one more time

from there you can proceed to investigate code execution by setting some additional breakpoints
or you can search Falcon code, whether you find something of interest regarding the MPRMessage context buffer

[thunderchero]

I don't believe it.... Edit; it was too good to be true... once I add a new ship via ue crash came back.

I was able to reload working file,

here is working files (Current beta vanilla 1366)
vanilla184.zip
files in zip are stbof.res, trek.exe and mpr565.dll

184 palettes in game with just 1 byte changed

184.jpg (142.97 KiB) Viewed 1099 times

mpr565.dll
0x0017a0e 80 -> FF

how I found this value (tested with 184 palettes)
I put a break point on 00538C14 call mpr_import__TextureEAX_or_PaletteEBX
it looped though this breakpoint 127 time without issue but on 128th time crashed at mpr565.dll:05CD521F 89 50 0C mov [eax+0Ch], edx
while running though first loop slowly I noticed the comp 80 so I copied sub to notepad (talk about lucky) found it later in mpr565.dll and edited and tested.

note 4 byte value so no idea how high we could go.

Edit2; after crash I started over with fresh install and just the edited mpr565.dll.
added multiple ships via ue got palette to 136 and tested
ships load in research screen, but when switching ships it crashes on light cruiser..... (see post above )
but I call this progress.

EDIT3; the way it is acting right now I think it has some kind of memory overlap. example after loading a few ships and switch to main map screen or main menu now it crashes at 53115C via windows event viewer.

[Spocks-cuddly-tribble]

I started over with fresh install and just the edited mpr565.dll.
added multiple ships via ue got palette to 136 and tested

I'd also suggest to start this project from scratch (remove all my trek.exe codes). Looks as if I/we misinterpreted 59FE30 and 59FE30.

I converted all the gif's to bmp's changed texture.lst gif -> bmp
added bmp's and texture.lst and Game did not crash

loc_538D4E list GIF, BMP, APL and TGA as valid texture types to load from stbof.res.

This is part of sub_538CA0 texture_256_bytes_field__load_texture.

59FE30 256 x0 for the textures -> not an index list and not part of this project? Same for esp in sub_538CA0.

Mpr565 crashes started after my wrong? code suggestions.

Original issue is just this:

Another thing that makes me wondering is the rendered galaxy class image. Why doesn't it simply crash as usual? All the pallettes must be loaded or either it crashed or there was a buffer overflow which most of the time would lead to some crash as well.
Is it possible that instead by some data calculation the indices are getting wrong? Like some signed byte calculation where 0x80 is mapped to -1 instead of 128?

There is also some test 80 checks with switches, but doesn't look a like negative byte check (I'd have to find them again).

I think it has some kind of memory overlap. example after loading a few ships and switch to main map screen or main menu now it crashes

@ Flocke can you check whether these values are buffer size values or just other stuff like flags.

One is used for MPR+ context buffer:

Beginning with dword_58AB20 you see twelve times the value 20100h = 131,328 dec for array_store and delete_free memory.

This is suspicious close to the max palette limit 131,072 and used for our codes in question.

[thunderchero]

I started over with fresh install and just the edited mpr565.dll.
added multiple ships via ue got palette to 136 and tested

I'd also suggest to start this project from scratch (remove all my trek.exe codes). Looks as if I/we misinterpreted 59FE30 and 59FE30.

I did start from scratch, but not test your 59FE30 changes with mpr565 changes

Mpr565 crashes started after my wrong? code suggestions.

mpr565.dll crash happens without any changes when palette is above 127

[Spocks-cuddly-tribble]

I did start from scratch, but not test your 59FE30 changes with mpr565 changes

Might be just for zero terminated text strings?

[thunderchero]

Might be just for zero terminated text strings?

running out of time today but did some more debugging and here where I am at now.

when I look at GlobalPaletteHandle some text is not displaying as ascII
if you look at highlighted line 0AF61928 the ascII should be vr8_
so when it trys to load defiant crash
why it is doing this I have no idea

vr8.jpg (577.99 KiB) Viewed 1276 times

here is notes on GlobalPalette and I think GlobalPaletteHandle
Note area will change location and size of area will increase as needed.

GlobalPalette

0D3D0020 E9 F1 F3 FF
0D3F2FFC 00 00 00 00
debug256:0D3F2FFF end of 00 00 00 00
debug268:0D400000 next line


DC 2F 02 00 hex size before next line
143324 size before next line

GlobalPaletteHandle

0AF60020 74 30 31 5F
0AF65FFC 00 00 00 00
debug224:0AF65FFF end of 00 00 00 00
debug225:0AF7B000 next line

E0 5F 00 00 hex size before next line
24544  size before next line

crash when defiant is selected
AUTO:00530554 53                                              push    ebx
AUTO:00530555 8B 5C 24 08                                     mov     ebx, [esp+4+arg_0]
AUTO:00530559 66 31 DB                                        xor     bx, bx
AUTO:0053055C 66 8B 43 0E                                     mov     ax, [ebx+0Eh]
AUTO:00530560 24 F8                                           and     al, 0F8h
AUTO:00530562 66 3D D0 CA                                     cmp     ax, 0CAD0h
AUTO:00530566 74 12                                           jz      short loc_53057A
AUTO:00530568 BA 0E 00 00 00                                  mov     edx, 0Eh
AUTO:0053056D 31 C0                                           xor     eax, eax
AUTO:0053056F E8 88 DB FF FF                                  call    sub_52E0FC
AUTO:00530574 31 C0                                           xor     eax, eax
AUTO:00530576 5B                                              pop     ebx
AUTO:00530577 C2 04 00                                        retn    4