Increasing the trek.exe Palette Limit project

[Flocke]

with continued analysis (previous post updated) there seem to be multiple render modes that depend on the ship model being rendered

in all cases, the rendering gets initialized with genMessages case 9 and 10
the rendering then is performed with genMessages case 7 (MPR_MSG_PACKET, which is checked prior to the case switch)

the switch for what render routine to execute happens in asm_10002E43:

Code: Select all

.text:10002E43                 call    dword ptr [edx+38h]

which is kinda ugly, since it calls a pointer that's being set by

Code: Select all

.text:100180AD                 mov     [edi+eax], edx

this one is iterated 22 times and sets sub routine pointers in the range of [edi = ecx+4Ch] to [edi+58h]
it actually merges two pointer arrays, for which only the 15th entry changes, the one of interest

it is called by

Code: Select all

.text:10018114                 call    sub_10018080
.text:1001919B                 call    sub_100180D0
.text:10015EAD                 call    dword ptr [eax+edx*4] (engMessage case 7)

the render routine 15 of 22 is set by sub_10003CE0 to one of the following subroutines:

Code: Select all

.text:10003D53                 mov     dword ptr [eax+38h], offset sub_1000EC60
.text:10003D60                 mov     dword ptr [ecx+38h], offset sub_10005B70  <- eax flags = 11002002h (mode 3, 4)
.text:10003D74                 mov     dword ptr [edx+38h], offset sub_1000D6E0
.text:10003D81                 mov     dword ptr [eax+38h], offset sub_10004390
.text:10003D9C                 mov     dword ptr [ecx+38h], offset sub_1000F990
.text:10003DA9                 mov     dword ptr [edx+38h], offset sub_1000DE40
.text:10003DBA                 mov     dword ptr [eax+38h], offset sub_10004F00
.text:10003DC7                 mov     dword ptr [ecx+38h], offset sub_10004F00
.text:10003DE2                 mov     dword ptr [edx+38h], offset sub_1000D340
.text:10003DEF                 mov     dword ptr [eax+38h], offset sub_1000D340
.text:10003E00                 mov     dword ptr [ecx+38h], offset sub_10006700
.text:10003E0D                 mov     dword ptr [edx+38h], offset sub_10003140
.text:10003E21                 mov     dword ptr [eax+38h], offset sub_10010C70
.text:10003E2E                 mov     dword ptr [ecx+38h], offset sub_10007E80
.text:10003E52                 mov     dword ptr [eax+38h], offset sub_100165D0
.text:10003E5F                 mov     dword ptr [ecx+38h], offset sub_10016F30
.text:10003E73                 mov     dword ptr [edx+38h], offset sub_10016B80
.text:10003E80                 mov     dword ptr [eax+38h], offset sub_10017750
.text:10003EB6                 mov     dword ptr [ecx+38h], offset sub_100072A0
.text:10003ECA                 mov     [eax+38h], off_10298110[edx*4]
for eax flags = 11002007h: sub_100098C0 (mode 7, 8)
for eax flags = 11006002h: sub_10016F30 (crystal entity mode 6)
.text:10003EE3                 mov     dword ptr [edx+38h], offset sub_1000CE40
.text:10003EF0                 mov     dword ptr [eax+38h], offset sub_1000C790  <- eax flags = 01200001h, 01200401h (mode 2, 12)
.text:10003EFD                 mov     dword ptr [ecx+38h], offset sub_1000CAD0
.text:10003F0F                 mov     dword ptr [edx+38h], offset sub_10018380  <- eax flags = 01200000h (mode 1)
.text:10003F1E                 mov     dword ptr [eax+38h], offset sub_100040D0

the render routine selection evaluates multiple render flags and is called from

Code: Select all

.text:100180E8                 call    sub_10003CE0
.text:1001919B                 call    sub_100180D0
.text:10015EAD                 call    dword ptr [eax+edx*4] (engMessage case 7)

engMessage case 7 updates the render flags for every packet of type 2 (MPR_PKT_RESTORESTATEID, first 4 bytes)
the third dword of the message is the render mode, which triggers this all
yes, another mode switch for switching the flags for selecting the render routine...

Code: Select all

.text:10015EA6                 mov     edx, [esi] <-- where packet type = 2
.text:10015EAD                 call    dword ptr [eax+edx*4]
->
.text:10019160                 mov     eax, [esp+arg_4]
.text:10019164                 mov     eax, [eax+8] <-- render mode

the render mode is converted to flags by

Code: Select all

.text:10019195                 mov     [ecx+4], edx

for vanilla BOTF the render mode is mapped like this:

Code: Select all

mode 1:  .bss:1003EE24 dd offset unk_1200000 (a few ships & outposts + crystal entity)
mode 2:  .bss:1003EE28 dd offset unk_1200001(some ships, outposts & starbases)
mode 3:  .bss:1003EE2C dd 11202002h (romulan starbase) => writes optional palette data
mode 4:  .bss:1003EE30 dd 11002002h (many ships & outposts + crystal entity) => writes optional palette data
mode 5:  .bss:1003EE34 dd 11206002h (unused)
mode 6:  .bss:1003EE38 dd 11006002h (crystal entity)
mode 7:  .bss:1003EE3C dd 11202007h (sheliak, starbases, federation heavy escort)
mode 8:  .bss:1003EE40 dd 11002007h (most ships & outposts)
mode 9:  .bss:1003EE44 dd 11206007h (unused)
mode 10: .bss:10031EE48 dd 11006007h (unused)
mode 11: .bss:1003EE4C dd offset unk_1200400 (unused)
mode 12: .bss:1003EE50 dd offset unk_1200401 (starbases)
mode 13: .bss:1003EE54 dd 11202406h (unused)
mode 14: .bss:1003EE58 dd 11002406h (unused)
mode 15: .bss:1003EE5C dd 11206406h (unused)
mode 16: .bss:1003EE60 dd 11006406h (unused)
mode 17: .bss:1003EE64 dd 11202407h (unused)
mode 18: .bss:1003EE68 dd 11002407h (unused)
mode 19: .bss:1003EE6C dd 11206407h (unused)
mode 20: .bss:1003EE70 dd 11006407h (unused)

this array is pre-initialized and luckily keeps same

used ship model render modes are:

Code: Select all

crystal entity: 1, 4, 6 => 6 for translucency?
borg cube: 8?
yridian scout: 4, 8 (4 altering and only when the back becomes visible)
vulcan ship: 8
acamarian raider: 8
tamarian defender: 8
talarian warship: 4, 8 (4 altering and only when the back becomes visible)
pakled transport: 8
sheliak raider: 7
ktarian raider: 4, 8 (altering for back and front)
bajoran fighter: 8
trill warship: 8
cardassian scout: 2, 8 (2 altering and only when the back becomes visible)
cardassian destroyer: 8
cardassian cruiser: 8
cardassian heavey cruiser: 8
cardassian attack cruiser: 4, 8 (altering for back and front)
cardassian battleship: 2, 8 (2 altering and only when the back becomes visible)
cardassian colony ship: 2, 4, 8 (altering for back and front)
cardassian outpost: 8
cardassian starbase: 2, 7, C (altering)
cardassian troop transport: 8
federation scout: 2, 8 (2 altering and only when the front becomes visible) => no optional palette data (federation tested only)
federation destroyer: 8 => no optional palette data
federation heavy destroyer: 4, 8 (altering for back and front) => writes optional palette data
federation heavy escort: 1, 7 (altering for back and front) => no optional palette data
federation heavy cruiser: 4, 8 (4 altering and only when the front becomes visible) => writes optional palette data
federation light cruiser: 4, 8 (4 altering and only when the front becomes visible) => writes optional palette data
federation strike cruiser: 8 => no optional palette data
federation command cruiser: 4, 8 (4 altering and only when the front becomes visible) => writes optional palette data
federation dreadnought: 8 => no optional palette data
federation colony ship: 4, 8 (4 altering and only when the back becomes visible) => writes optional palette data
federation outpost: 1, 2, 8 (altering, 1 when top becomes visible) => no optional palette data
federation starbase: 7, C (altering) => no optional palette data
federation troop transport: 4, 8 (altering for back and front) => writes optional palette data
ferengi explorer: 2, 4, 8 (altering for back and front, 4 rarely seen)
ferengi light raider: 4, 8 (4 altering and only when the bottom becomes visible)
ferengi war cruiser: 4, 8 (4 altering and only when the bottom becomes visible)
ferengi strike cruiser: 4, 8 (4 altering and only when the bottom becomes visible)
ferengi marauder: 8
ferengi colony ship: 2, 4, 8 (altering for back and front)
ferengi outpost: 8
ferengi starbase: 7, C (altering)
ferengi troop transport: 8
klingon scout: 2, 4, 8 (altering for back and front)
klingon destroyer: 8
klingon battle cruiser: 8
klingon heavy cruiser: 4, 8 (4 altering and only when the back becomes visible)
klingon strike cruiser: 4, 8 (altering for back and front)
klingon attack cruiser: 8
klingon colony ship: 8
klingon outpost: 8
klingon starbase: 7, C (altering)
klingon assault transport: 8
romulan scout: 4, 8 (altering for back and front)
romulan destroyer: 4, 8 (altering for back and front)
romulan battle cruiser: 4, 8 (altering for back and front)
romulan cruiser: 4, 8 (altering for back and front)
romulan strike cruiser: 4, 8 (altering for back and front)
romulan warbird: 1, 8 (altering for back and front)
romulan colony ship: 1, 8 (altering for back and front)
romulan outpost: 4, 8 (altering for back and front)
romulan starbase: 2, 3, 7, C (altering, C rarely seen)
romulan troop transport: 8

the render mode initialization I tracked down to

Code: Select all

.text:10015E44                 call    sub_10018060

the returned eax value is the cached render mode structure that is passed along
the active render mode integer is read from and stored to:

Code: Select all

.text:10015E24                 mov     eax, ds:dword_1001B110
.text:10015E6A                 mov     ds:dword_1001B110, edi

stored with it is a pointer to the current rendered ship model and palette entry
the ship model is updated by

Code: Select all

.text:1001881E                 mov     [edi+24h], ebx
.text:10018E43                 call    sub_10018800 (engMessage case 9, first message per model rendering)

unlike expected, the optional palette data however is not immediately updated by the changed render mode
instead it is updated by engMessage case 7 type C MPR_PKT_TRIFAN! (Ch 4h 6h)

Code: Select all

.text:10005BF2 call    sub_5CF8500
.text:10002E43 call    dword ptr [edx+38h]
.text:100156E5 call    sub_5CE1C50
.text:10001522 call    sub_5CF5630
.text:1001A271 call    [esp+2Ch+var_8]
.text:10015EAD call    dword ptr [eax+edx*4] (engMessage case 7 of type C)

for it however the render routine must be set to either of:

Code: Select all

sub_10004390 (unused)
sub_10005B70 (mode 3, 4)
sub_1000D6E0 (unused)
sub_1000EC60 (unused)
sub_100165D0 (unused)
sub_10016B80 (unused)
sub_10016F30 (unused)
sub_10017750 (unused)

my initial thought was to skip the optional palette data, and then by shrinking the palette entries make place for additional entries
I'm not sure whether that's a feasible approach, maybe it'll be easier to just move the whole palette entries
knowing the routines of interest however might help reduce work and also will help when testing

e.g. you might try to nop seemingly unused routines and see if it breaks anything
in addition, like already proposed, you might skip relocation of successive data after the palettes to reduce work

Nonetheless, I rather switch over to UE now and leave further tests to both of you.
I'm sure it can be done, but it'll be quite some work whatever approach you choose and too much work for me to proceed with.

[Flocke]

that is a lot of new info you added

yep, and I continued to update the post

Graphics\BSPlib\TexBank.cpp doesn't look any interesting to me, it just pre-loads a pool of images
but it shows that they are transferred to the MPR render engine

Edit I can't believe some of the comments in source
"palID = 0; // Punt! Might break somewhere else later (runtime?) but for now..."

Actually that one is harmless. When it fails to load some image, it adds an empty entry to the texture pool.
If lateron that missing image is used, it most likely raises a null pointer exception and crashes the game.
Ofc one could have properly handled the missing image and logged an error, but we know crashes on missing images...

So the sole purpose of MPR565.dll is converting the palette data to 16bit for the MPR context dialog?

Of course not. It is just some data pre-processing for onfollowing render calls of render mode 3 and 4.
I wonder why some models use this render mode and others not.

The default render mode seems to be number 8, but it seems like that for more complex models, they are seperated into multiple parts that each are rendered in the different render modes 1, 2, 3, 4, 6, 7, 8 and 12.
One might try to switch these modes or see what model changes are causing them.

Edit: Wait, I found the different packet types and render modes.
Packet types are defined by MPRPacketID of mpr.h:

Code: Select all

0: MPR_PKT_END
1: MPR_PKT_SETSTATE
2: MPR_PKT_RESTORESTATEID -> changes render mode
3: MPR_PKT_STARTFRAME
4: MPR_PKT_FINISHFRAME
5: MPR_PKT_CLEAR_BUFFERS
6: MPR_PKT_SWAP_BUFFERS
7: MPR_PKT_POINTS
8: MPR_PKT_LINES
9: MPR_PKT_POLYLINE
A: MPR_PKT_TRIANGLES
B: MPR_PKT_TRISTRIP
C: MPR_PKT_TRIFAN -> causes optional palette buffer load

Render modes can be found in Context.h:

Code: Select all

00: STATE_SOLID <- unused by unmodded BOTF
01: STATE_GOURAUD <- mode 1
02: STATE_TEXTURE <- mode 2
03: STATE_TEXTURE_GOURAUD <- mode 3
04: STATE_TEXTURE_TRANSPARENCY <- mode 4
05: STATE_TEXTURE_GOURAUD_TRANSPARENCY <- unused by unmodded BOTF
06: STATE_TEXTURE_PERSPECTIVE <- mode 6
07: STATE_TEXTURE_GOURAUD_PERSPECTIVE <- mode 7
08: STATE_TEXTURE_TRANSPARENCY_PERSPECTIVE <- mode 8
09: STATE_TEXTURE_GOURAUD_TRANSPARENCY_PERSPECTIVE <- unused by unmodded BOTF
0a: STATE_ALPHA_SOLID <- unused by unmodded BOTF
0b: STATE_ALPHA_GOURAUD <- unused by unmodded BOTF
0c: STATE_ALPHA_TEXTURE <- mode 12
0d: STATE_ALPHA_TEXTURE_GOURAUD <- unused by unmodded BOTF
0e: STATE_ALPHA_TEXTURE_TRANSPARENCY <- unused by unmodded BOTF
0f: STATE_ALPHA_TEXTURE_GOURAUD_TRANSPARENCY <- unused by unmodded BOTF
10: STATE_ALPHA_TEXTURE_PERSPECTIVE <- unused by unmodded BOTF
11: STATE_ALPHA_TEXTURE_GOURAUD_PERSPECTIVE <- unused by unmodded BOTF
12: STATE_ALPHA_TEXTURE_TRANSPARENCY_PERSPECTIVE <- unused by unmodded BOTF
13: STATE_ALPHA_TEXTURE_GOURAUD_TRANSPARENCY_PERSPECTIVE <- unused by unmodded BOTF

there are more render modes defined, but mpr565 only seems to support the first 20 render modes

The used render flags are:

Code: Select all

00000001h: MPR_SE_SHADING -> interpolate ARGB color values
00000002h: MPR_SE_TEXTURING -> interpolate UVW texture coordinates
00000004h: MPR_SE_MODULATION -> modulate texture with color values
00000400h: MPR_SE_BLENDING -> per pixel blending of the rendered faces
00002000h: MPR_SE_FILTERING -> texture filtering
00004000h: MPR_SE_TRANSPARENCY -> use transparency (used by crystal entity)
00200000h: MPR_SE_NON_PERSPECTIVE_CORRECTION_MODE -> used for gouraud shading
01000000h: MPR_SE_PALETTIZED_TEXTURES -> well, disable it if we can :lol:
10000000h: MPR_SE_DITHERING -> to emulate higher color depth by altering the pixel colors

btw, there also exist an MPR_SE_ANTIALIASING flag 08000000h which might give nice results


Given that the actual render pipelines are very lengthy and hard to decrypt, I don't think we can easily replace the optional palette entry for render mode 3 and 4. But I didn't continue to investigate on this.

I think I would rather attempt to relocate the whole palette entry array but leave onfollowing data in place.

[thunderchero]

Thanks for all your efforts, but I agree at this time move on from this project.

00000002h: MPR_SE_TEXTURING -> interpolate UVW texture coordinates
00000004h: MPR_SE_MODULATION -> modulate texture with color values
00000400h: MPR_SE_BLENDING -> per pixel blending of the rendered faces

this sound very interesting,

we have 3 sets of MPR files
Vanilla botf mpr files (allows 512 x 512 gif)
original falcon 4 mpr files (limited to 256 x 256 gif)
updated falcon 4 1.0.8 mpr files (limited to 256 x 256 gif)

if only the mpr565 is replaced with either falcon 4 file limit is lowered to 256 x 256

let's raise texture limit to 1024 x 1024 lol
but to be honest any larger textures are pointless with palette limits
(why make an image perfect just to remove detail with low palette limit)

Note; I will probably split this project off to general modding section.

[Flocke]

Note; I will probably split this project off to general modding section.

Yeah, would have been great when there was no need for reducing palettes any longer, but I guess this will remain for a while longer.

[Spocks-cuddly-tribble]

Judging from this random google quote about palettes in games:

Most modern games don't really use a color palette, per se, unless you're going with an 8-bit or 16-bit vibe.

If possible, wouldn't it be the best solution to just disable this limiting feature? I.e. allow any color to be rendered from any texture.

[thunderchero]

Judging from this random google quote about palettes in games:

Most modern games don't really use a color palette, per se, unless you're going with an 8-bit or 16-bit vibe.

If possible, wouldn't it be the best solution to just disable this limiting feature? I.e. allow any color to be rendered from any texture.

I would not care if it was limited to gif -> bmp
and converting the few files would be no issue I would do all mods with for that update
but while you were at it you might as well get rid of 16 bit tga's also.
btw I have always thought 16 bit tga is why we have mouse lag without dxwnd

[Flocke]

If possible, wouldn't it be the best solution to just disable this limiting feature? I.e. allow any color to be rendered from any texture.

Sure, but I doubt that'll be an easy switch.

From the interface and code I've seen, it is likely that mpr565 by itself can also render in 32bit mode. I think there also was a test utility that iterates 32bit modes. By the MPR API it is just a matter of selecting the right render device. They can be checked with MPRGetDeviceResolution and MPRGetSurfaceDescription.

But we'd have to also convert the video and stuff. I fear that'll be a big problem because you can't mix bit depths.
Therefore I'd rather replace the whole engine in one go at some point...

[thunderchero]

Did either of you have an idb file with notes/labels on mpr565.dll?

[Flocke]

Did either of you have an idb file with notes/labels on mpr565.dll?

Nope, but for any further analysis I agree it would be great to have one!

[Spocks-cuddly-tribble]

But it's only one the fly guessing crap as usual:

mpr565.zip

(231.54 KiB) Downloaded 17 times