Trek.exe: in-memory patching with C/C++

General Modding Information/Questions; support/discussion/questions

Moderator: thunderchero

Forum rules
:idea: Please search before starting new topic. :idea:
There is a good chance it has already been asked.
User avatar
QuasarDonkey
Code Analyst
Code Analyst
Posts: 433
Joined: Tue Jul 26, 2011 8:29 pm
Location: Ireland

Trek.exe: in-memory patching with C/C++

Post by QuasarDonkey » Sat Aug 18, 2012 11:55 pm

(Edited) For people that have been following this: it turns out Trek doesn't need to be a DLL at all. It works just as well with an EXE, with this new approach.

For programmers, this C program should explain it well enough:

Code: Select all

#include <windows.h>

// Function pointer set to trek.exe's main() function.
static void (* start_botf)(void) = 0x54AC6A;

void main()
{
        HMODULE DllBase = LoadLibrary( "trek.exe" );
        assert( DllBase == 0x400000 );

        // BotF will exit the whole process when done, so no code after this would be called.
        start_botf();
}
(Note I've left out normal error-checking here for clarity.)

:!: The only requirement: when compiling and linking the loader program, you MUST specify a base offset that doesn't conflict with the memory addresses used in BotF. Trek.exe loads at address 0x400000, so if you choose, say, 0x800000 as the base address, this program will run BotF. If you don't do this, the relocation table will be used to move Trek.exe to a different location in memory, making most existing patches useless.

If using the OpenWatcom compiler, here's the commands needed to compile and link:

Code: Select all

wcc386 loader.c
wcl386 -l=WIN95 -fe=loader.exe -\"OPTION OFFSET=0x800000\" loader.o
The main reason to use Watcom is because it was used to compile BotF, and Watcom uses special function calling conventions.

So you can access variables and functions in Trek.exe through pointers. Just set them equal to the asm offsets in IDA Pro.

Patching Trek.exe in memory

If you tried to patch code in a running program, it would crash out. This is because the memory is protected from being overwritten. But we can change the properties in Trek.exe to make the code writable, so we can patch it while it's running. Basically, all we need to do is patch a single byte in Trek.exe. You could use special tools, like CFF Explorer, but I'll just tell you which byte to set in trek.exe:

Code: Select all

At offset 0x019f, change C0 to E0
Patching is as simple as overwriting data via pointers. But "monkey patching" the code is much nicer. Basically we write a far jump x86 instruction to redirect execution to new code we've written. Here's the function to do this:

Code: Select all

// Allows us to replace functions at the given address by far jumping
// to our target address. It requires 10 bytes to do.
void monkeyPatch (unsigned int offset, void * targetAddr)
{
        unsigned char * trampoline = offset;
        trampoline[0] = 0xff;
        trampoline[1] = 0x25;
        *(void**)(trampoline+2) = trampoline + 6;  // address of the address to jump to
        *(void**)(trampoline+6) = targetAddr;  // address of the target code
}
Below, you'll see some new code for generation of "sparse galaxies":

Code: Select all

// This function takes in a 2D array of zeroes, and we write 1 where we want a star.
// Note that the grid is 5*5 times larger than the actual map.
void Universe_MkGal_GalaxyGrid (unsigned char **grid, int shape)
{
        // Get MapLongEdge*5 and MapShortEdge*5
        const size_t galW = *(unsigned *)(0x5CB304 + 0x40);
        const size_t galH = *(unsigned *)(0x5CB304 + 0x38);

        size_t X, Y;

        // Top-left
        for (Y = 0; Y < 10; Y++)
        for (X = 0; X < 10; X++) {
                grid[Y][X] = 1;
        }

        // Top-right
        for (Y = 0; Y < 10; Y++)
        for (X = galW - 10; X < galW; X++) {
                grid[Y][X] = 1;
        }

        // Bottom-right
        for (Y = galH - 10; Y < galH; Y++)
        for (X = galW - 10; X < galW; X++) {
                grid[Y][X] = 1;
        }

        // Bottom-left
        for (Y = galH - 10; Y < galH; Y++)
        for (X = 0; X < 10; X++) {
                grid[Y][X] = 1;
        }

        // Center
        for (Y = galH/2 - 5; Y < galH/2 + 5; Y++)
        for (X = galW/2 - 5; X < galW/2 + 5; X++) {
                grid[Y][X] = 1;
        }
}
Here's how you apply the patch (in the main function):

Code: Select all

monkeyPatch( 0x4AF9A0, Universe_MkGal_GalaxyGrid );
Here's the new code in action:

It places empires in the corners and center, and leaves no stars in-between the empires. Works on large galaxies too :)

This should allow for some interesting patches, with no need to modify Trek.exe.
Last edited by QuasarDonkey on Sun Aug 19, 2012 10:13 pm, edited 6 times in total.

User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6402
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Trek.DLL

Post by thunderchero » Sun Aug 19, 2012 12:17 am

Flocke will be very happy to see this, But I was lost after "After some interesting discussions with Flocke"

So since you say you have it working, Does it run any better as dll? will this help with memory problems and crashing or will new code need to be written to correct those problems?

thunderchero

User avatar
QuasarDonkey
Code Analyst
Code Analyst
Posts: 433
Joined: Tue Jul 26, 2011 8:29 pm
Location: Ireland

Re: Trek.DLL

Post by QuasarDonkey » Sun Aug 19, 2012 12:24 am

BotF is not any faster or better (yet).

But let me put it this way: that "ZIP bypass patch" I gave as an example, I started writing that in asm shortly after I joined here. I could never get it right, even after hours of modding at a time. It took about 5 minutes to get it working today, typing it out of my head.

Using trek.dll will allow for extremely sophisticated mods. One of the reasons is that the new code is written in C/C++, which is far easier to write than assembly code. The other reason is that since we're overwriting the Trek.exe code with jumps to our new EXE code, we have unlimited space to write the new code in (since it's in a new EXE).

User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6402
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Trek.DLL

Post by thunderchero » Sun Aug 19, 2012 12:39 am

Ok, I was hoping, but I expected that answer. :wink:

So I guess my next question would be when and what to expect to see first?

thunderchero

User avatar
QuasarDonkey
Code Analyst
Code Analyst
Posts: 433
Joined: Tue Jul 26, 2011 8:29 pm
Location: Ireland

Re: Trek.DLL

Post by QuasarDonkey » Sun Aug 19, 2012 12:52 am

I'm really not sure yet. I'm thinking of maybe blending the new BotF clone code with existing Trek.exe so we can keep the old game mechanics, AI, etc., but with a new GUI & Networking. It depends on how much work is required. It might just be faster to rewrite the whole lot.

I mentioned above that there might be problems using existing patches with the DLL. One possible solution for porting all the existing patches to work with the DLL (provided they have assembly code provided), would be to use asmjit - the Just-In-Time Assembler. This would let us assemble code in memory (so we can accommodate for the dynamic DLL base address), and then overwrite Trek.dll in memory. So this means we can use Assembly, C, or C++ for writing new patches.

User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 2581
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Trek.DLL

Post by Flocke » Sun Aug 19, 2012 7:33 am

thunderchero wrote:Flocke will be very happy to see this
Yes I am, great work and great article! :D
QuasarDonkey wrote::arrow: You must use the OpenWatcom compiler for this, since BotF was compiled with Watcom, and Watcom uses special function calling conventions.
cancel the 'must', you know I already used gcc with botf in mpr++, it's just a bit more tricky to follow non-supported watcom calling conventions using inline-assembler. ;)
QuasarDonkey wrote::arrow: Once the DLL is loaded, it will NOT be relocated again, so it's safe to patch the code in memory without worrying about the relocation table.
That's a big relief! Somewhere I thought I had read that on demand, it *might* get moved by the operation system again, but seems you're right and that was non-sense. Once we get a handle to the dll it should remain valid :!:

So the question remains, what to do with it?

As pointed out before, most current patches will corrupt the relocation table sadly driving this approach useless - for of now.
I see three options to get around:
  1. the relocation table has to be fixed for each patch applied to the dll file
  2. all patches must include information on code-references they use to fix with actual address when patching on the fly
  3. the dll has to be assured to load to original exe base-offset and new exe has to be compiled with different base-offset
First one is for sure the cleanest as there's no need for on-the-fly patching and making sections writable.
While first and second both require information on code-references used, first approach also needs an algorithm to determine offsets in relocation table to remove and the ability to lengthen the relocation table which I'm not sure of whether that's possible without affecting other code right of now.
Then there's the dirty solution of just changing the base-offsets, but that's a cumbersome solution for coding and rarely might fail in case of conflicting dlls loading to random offsets. :roll:

Is that work worth it?
For modding botf probably not, as it's easier/cleaner to let trek.exe call a dll immediately after being launched like I do already, implementing a plugin-system for loading further dlls eventually, and we can achieve all the same. I've been kinda euphoric on converting the exe to dll before, but fixing all the relocation offsets, who's going to do all that work?
No, we'd get it partially done and then trouble with people mixing fixes again like it's always been. :(
Non the less it would be cool to include relocation information for the patch-tool files - just in case.

For writing an own application like the clone-project this might be a nice approach to get access to some specific code though. But like QD already noted will have to be seen where it's possible to isolate code with reasonable work.

In conclusion, not sure on where to use either but find it a very cool research and great discovery for that it's possible! :)

User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6402
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Trek.DLL

Post by thunderchero » Sun Aug 19, 2012 8:09 am

So I guess the big question is how many patches currently changed the relocation table? and how would this be figured out?

thunderchero

User avatar
anjel
Past Administrator
Past Administrator
Posts: 666
Joined: Thu May 08, 2008 2:00 am
Location: Bs As - Argentina

Re: Trek.DLL

Post by anjel » Sun Aug 19, 2012 3:05 pm

well, all i have to say is that with every post this is getting very exciting, to start thinking how would be, because we have a tool to start doing it, and not just "is it possible... o wait... we need to make a tool...".
I think i have to be up to date withh c++ lessons... :oops:

User avatar
QuasarDonkey
Code Analyst
Code Analyst
Posts: 433
Joined: Tue Jul 26, 2011 8:29 pm
Location: Ireland

Re: Trek.DLL

Post by QuasarDonkey » Sun Aug 19, 2012 3:08 pm

Another Update: You CAN force trek.dll to load at its preferred base address (0x4000000). You do this by (a) stripping the relocation information from Trek.dll, and (b) when linking the loader program, set it to have a different base address (0x3000000 works fine).

A similar procedure is actually used in MS Windows for the kernel DLLs, etc., where they are given fixed addresses, and programs voluntarily use a different base address.

In layman's terms, this means that all existing patches CAN be used with the DLL, I'll update the main post with the new information shortly.

User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6402
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Trek.DLL

Post by thunderchero » Sun Aug 19, 2012 4:42 pm

QuasarDonkey wrote:In layman's terms,
We need more of that lol
QuasarDonkey wrote:this means that all existing patches CAN be used with the DLL,
this sounds great, and since many older mods are no longer supported by creator maybe some of the more major problems can be addressed without changing mod concepts.

thunderchero

User avatar
anjel
Past Administrator
Past Administrator
Posts: 666
Joined: Thu May 08, 2008 2:00 am
Location: Bs As - Argentina

Re: Trek.DLL

Post by anjel » Sun Aug 19, 2012 4:49 pm

thunderchero wrote:
QuasarDonkey wrote:In layman's terms,
We need more of that lol

lol, agreed :mrgreen:

User avatar
thunderchero
Site Administrator aka Fleet Admiral
Site  Administrator aka Fleet Admiral
Posts: 6402
Joined: Fri Apr 25, 2008 2:00 am
Location: On a three month training mission, in command of the USS Valiant.
Contact:

Re: Trek.DLL

Post by thunderchero » Sun Aug 19, 2012 9:05 pm

QuasarDonkey wrote: Note: the main multi-mod version of Trek.exe has corrupted section names. So I used the official 1.0.2 EXE to make the DLL, and then patched in the no-CD fix.
Flocke just pointed this out to me earlier this year. That goes way back to 1999 after 1.0.2 release "TBC team" released a edited version of the trek.exe. Flocke even sent me the original readme with the edited trek.exe.

thunderchero

User avatar
QuasarDonkey
Code Analyst
Code Analyst
Posts: 433
Joined: Tue Jul 26, 2011 8:29 pm
Location: Ireland

Re: Trek.exe: in-memory patching with C/C++

Post by QuasarDonkey » Sun Aug 19, 2012 9:47 pm

Lol, I was wondering why all the section names have been changed to [Tbc]. It doesn't actually matter now. I've updated the main post to show the state-of-the-art.

User avatar
Flocke
BORG Trouble Maker
BORG Trouble Maker
Posts: 2581
Joined: Sun Apr 27, 2008 2:00 am
Location: Hamburg, Germany
Contact:

Re: Trek.DLL

Post by Flocke » Mon Aug 20, 2012 1:11 am

thunderchero wrote:Flocke even sent me...
I did sent you WHAT? 8O
I just cleaned my disk, that's all I did... *lalalalala* :lol:
QuasarDonkey wrote:Another Update: You CAN force trek.dll to load at its preferred base address (0x4000000). You do this by (a) stripping the relocation information from Trek.dll, and (b) when linking the loader program, set it to have a different base address (0x3000000 works fine).
Exactly. But I have a bad feeling with potentially many libraries loaded in turn before of loading trek.dll/trek.exe (btw nice you can simply load the exe, didn't ever thought of that :o). When there are just a few dlls loading to a random OR to a conflicting address forcing to load one to a random one instead, then loading up the game from time to time might simply crash as dynamicly loading trek.dll/trek.exe will be performed last.
To assure this won't happen, trek.dll needs to be loaded first after your launcher executable.
For that it needs to be loaded statically and I dunno in what order those static dll loads are performed, maybe it can be simply controlled by adjusting the name of trek.dll. :roll:
Edit: not sure but could be that for static dll loads the os keeps care on the addresses moving other dlls on need :idea:

User avatar
QuasarDonkey
Code Analyst
Code Analyst
Posts: 433
Joined: Tue Jul 26, 2011 8:29 pm
Location: Ireland

Re: Trek.exe: in-memory patching with C/C++

Post by QuasarDonkey » Mon Aug 20, 2012 1:27 am

I forgot about that. I guess the simplest answer is to keep the loader program simple, and use LoadLibrary() to load plugin DLLs that do any heavy modding.

Post Reply

Return to “General Modding Information/Questions”

Who is online

Users browsing this forum: No registered users