Trespasser File Formats

Disclaimer: The information contained in this document is provided exclusively for Your Own Use in order to enjoy Your Legal Copy of the game Trespasser. You must not use this information in any way that would break the conditions of the Trespasser License Agreement.

This document contains the latest known information about the file formats used by Trespasser, and is written by Andres. It is based on the original formats.txt document created by PenguiN42, TSOrd, and Andres. Last updated 24 July 2003.

Unless otherwise specified, all numeric values are stored as a 4 byte integer, henceforth called a long (possibly unsigned, which is mentioned explicitly if the distinction is important).

Contents

Info about Trespasser, and conventions used in the levels:
  Directory structure
  File locations and naming
  Model and Instance names
Format of the data files:
  GROFF Block structured file format used by GRFs, SCNs and Savegames
  GRF - Main level file Models, instances, scripting
  SCN/Savegame - Scene file (level startup) Savegame state, including spatial partition tree
  PID - Texture index Texture information
  SWP/SPZ - Texture data Raw and compressed textures
  TPA - Audio file format PCM/ACPCM samples
  WTD - Terrain Wavelet compressed terrain
  SMK - Video
  DDF - Menu

Directory structure

File locations and naming

Model and Instance Names

Some conventions are required by Trespasser - the level will not load if these are not followed:

• The instance names must each be unique, as they identify blocks in the scn and savegame.
• The names of all instances of a model must be identical to the model name, except for the optional postfix number (eg. "-00","-01","-999"). The postfix numbers don't have to be consecutive, just unique.

Some conventions affect the way the level works:

• Objects with names beginning with '$' are called $ objects, or physics objects. $ causes an object to be invisible, so it need not specify Visible=false in the valuetable (which is important for type1 geometry objects, because they will crash Trespasser if they are Visible objects).

The following conventions are found in the Trespasser data files, however they are not required by the game engine. (I assume they were used to assist level designers.)

• Models are named Blah-00, and instances using that model are Blah-00, Blah-01, ..., Blah-99, Blah-100, ... . However, some models that are used only once omit the '-00' and and instance that uses it also omits that suffix.
• Each model has a single valuetable entry, which has the same name. Thus all Trespasser instances of a particular model always have the same valuetable entry. (The actual value names seem to be ignored, an index is used to identify them.)
• The model names seem to be ignored by Trespasser [Is that true? I thought they had to match the instance names or trespasser crashed (see above).]

Modifying files

TODO: Changing the position of an object can cause Trespasser to crash, or objects to disappear.

If an instance has a block in the SCN/Savegame, that instance's info in the region and value tables are ignored.

GROFF - Main structure of GRFs, SCNs and Savegames

Groff is the name given to a certain block structured file format, and is named because the string "GROFF" is found in the executable. Groff files consist of a HEADER, a DATA DIRECTORY, DATA BLOCKS, and a NAME TABLE, and are the format for the GRF, SCN and Savegame files.

HEADER

DATA DIRECTORY ENTRY

The DATA DIRECTORY describes the DATA BLOCKS

NAME TABLE

The NAME TABLE stores strings which are referenced by the symbol handle. I believe the handles can be any set of unique values (as long as they match in all the places they're used). In the GRF the strings are used as the names of DATA BLOCKS, names of instances in the REGION TABLE, and names of textures in the .mapping block. In the SCN and Savegames, the strings are used as names of the DATA BLOCKS, and also are used within many of the data blocks (of which the exact format is not yet known).

Trespasser files have the name table sorted alphabetically by the string, but that is not required by the game engine.

GRF - Main level file

GRF Blocks:

- Header
- Region
- Valuetable
- Main Object Block<
- - Geometry>
- - Mapping - - Material>

General Information

GRFs hold all the object geometry, positioning, and scripting for a level. They use the Groff format, and were made with a custom 3DS MAX plugin.

GRFs can be either compressed (as in the game) or uncompressed (as in the demo). Compressed files use Lempel-Ziv (LZ) compression, and can be decompressed using WinZip. Alternatively the Microsoft MS-DOS utilities Compress.exe and Decompress.exe can be used. From within MS Visual C++, LZOpen() and releated functions can be used to decompress them.

GRFs contain three main types of data. 'Models' are actual geometry (eg. textured triangles) and are stored in seperate named data blocks. 'Instances' are instantiations of a model in the level (eg. a model reference + position,orientation,scale) and are listed in the regiontable (".region"). The valuetable contains extra data associated with instances (render parameters, scripting, ...) (".valuetable").

Notes on naming. Blocks types can be identified by either their string name or data directory 'type'.

Models consist of several blocks, a Main block with the name of the model, then optional ".geometry", ".material", and ".mapping". Main blocks contain handles to the geometry and mapping blocks, and the mapping block contains a handle to the material block. (They always seem to be written in consecutive blocks, but this should not be relied on.) There are two different types of geometry format, which I call Type1 and Type2.

Mapping strings to IDs

Textures and objects in the script are referred to by a string, which must be mapped to an integer ID which is used by the PID and TPA. This mapping is done using a hash function tresHash(), described (TODO).

Textures have three strings, for texture, opacity, and bump maps. If they are all empty strings, there is no texture. Otherwise the texture string will exist, and the opacity and bump strings seem to match the info in the PID entry ie. bumpmapped textures always have a bump string, and textures with transparency always have an opacity string (so all three may exist). To find the textureID, the texture and bump names are combined with the Diffuse value for the mesh (default 1.0).

The GRF blocks

".header"

Exactly one ".header" per GRF.

0x00	BE BA CE 0A (groff magic number)
0x04	unknown (always 0x8000000F?)

".region"

The regiontable lists all the objects (instances) in a level, there is exactly one ".region" per GRF. Note that when changing the position of a 'primary' instance (described TODO), many of the models listed in the script also have to be moved. They have absolute positions, yet the relative position compared to the primary instance is used to instantiate those models on other instances of a mesh. (eg. Model00s, Detail0, DetailShadow, A00s. See document on the valuetable for more information).

HEADER (4 bytes)

0x00	Number of entries
0x04	Beginning of REGION ENTRYs

REGION ENTRY (44 bytes)

0x00	Main Handle of model
0x04	symbolHandle. Name of this block, each entry must have a unique name.
0x04	Position in level (3float)
0x14	Orientation, as Euler angles. Rotation order is x then y then z going object->world. (3float)
0x20	Scale (float)
0x24	Valuetable entry number, or 0 if 0x28 is 0
0x28	0 - No valuetable entry 1 - Has valuetable entry

POST-REGIONTABLE (4 bytes)

0x00	Main Handle of valuetable (long)

".valuetable"

The valuetable stores extra initialization information for instances in the REGIONTABLE, and there is exactly one ".valuetable" per GRF. A region instance will always reference a Group value entry which lists the values for that instance. The entry will always have the same name as the instance's model, so it's more correct that the valuetable stores info about the models, not the instances.

HEADER (20 bytes)

0x00	00 00 00 00 00 00
0x06	Number of strings in STRINGTABLE
0x0A	00 00 00 00 00 00
0x10	Number of strings in STRINGTABLE (again)
0x14	Start of STRINGTABLE
0xxx	ENTRY HEADER
0xxx	VALUETABLE ENTRYs

STRINGTABLE ENTRY

0x00	Null-terminated string
0xxx	Number of this string (numbered consecutively from 0) (long)
0xxx	Number of times this string occurs in ASSEMBLY. Seems to be ignored by Trespasser, can just be zero. (long)

ENTRY HEADER (18 bytes)

0x00	00 00 00 00 00
0x05	Number of entries
0x09	00 00 00 00 00
0x0E	Number of entries (again)
0x13	Beginning of VALUETABLE ENTRYs

VALUETABLE ENTRY

0x00	Entry type (unsigned short) 0x101 - Boolean 0x103 - Integer 0x104 - Float 0x105 - String 0x106 - Group (.exe calls this CObjectValue)
0x02	00
0x03	Entry number (consecutively numbered from 0)
0x07	01 00 00 00
0x0B	Entry name, index into STRINGTABLE. If this entry is directly used by an instance, this name will be the name of the instance's mesh. However, Trespasser seems to ignore it (eg. can use the string "unused" for all names).
0x0F	Number of times entry name has been used so far. Seems to be ignored by Trespasser, can just be zero.
0x13	00 00 00 00
0x17	00 00 00 00
0x1B	Entry data, depends on entry type: Boolean - Unsigned char, 0=false, 1=true 1byte Integer - (signed?) long 4 bytes Float - float 4 bytes String - zero terminated ASCII L bytes Group - Group of entries, N (long) = size, followed by 2*N longs. The first of each pair is an entry number, the second is always 0x01. (Perhaps the second is the number of times the entry is used?) 4+8*N bytes
0xxx	Entry number again (long)
0xxx	01 00 00 00

Main object block

This represents an object model or 'mesh', and is referred to by the .regiontable. A model must have a .geometry block, and has optional .mapping (with corresponding .material).

0x00	Symbol Handle (stringtable) Name of the mesh. Trespasser seems to ignore it (eg. can use string "unused" for all mesh names).
0x04	Geometry Handle (data block)
0x08	Mapping Handle (data block)
0x0C	Physics Handle (always zero)
0x10	AI Handle (always zero)
0x14	Sound Handle (always zero)
0x18	Special Handle (always zero)

".geometry"

Geometry blocks contain the actual object geometry, and contain one of two data formats. I currently use a heuristic to tell which format it contains: If there is a .mapping block, check the size of that block. If the size is 4 bytes, it is type 2, otherwise it is type 1. If there is no mapping block, check the 2nd int of the block (pivotOffset.y in type 1, mapping flag in type 2). If this is 1, the block is type 2, otherwise it is type 1. This works because the floating point interpretation of 0x01 is 1.4013e-045, which is quite unlikely, and in fact never occurs in pivotOffset.y in the trespasser data files)

Type 1:

Type 1 geometry blocks mostly contain untextured cubes (which are physics objects or triggers) and diamonds and spheres (which are triggers), although sometimes they have dummy textures (blank or with "Dummy" written on them). Sometimes they contain textured visible geometry, particularly dinosaurs. If the object is textured, the texture coordinates are stored in the .mapping block.

0x00	float[3]	Pivot Offset (how used?)
0x0C	int	(A) Vertex Count
0x10	int	(B) Face Count. Matches face count in .mapping if .mapping exists.
0x14	int	(C=3*B) Vertex Normal Count
0x18	int	(D) Wrap Vertex Count
0x1C	int[3]	Default Color (.BMP style BGR order)
0x28	float[3][A]	Vertices. Also used to find the physics bounding box. Note that the bounding box is centered about the origin, so even if all vertices are in the range [0.1,0.7], the bounding box will be [-0.7,0.7]
0x--	int[3][B]	Faces. Index into the Vertices.
0x--	float[3][B]	Face Normals. One normal per triangle.
0x--	float[3][C]	Vertex Normals. Normal for vertex j of triangle k is in position 3*k+j.
0x--	float[3][D]	Wrap Vertices. It is not known what these are used for. Trespasser uses the standard vertices (above) to calculate the bounding box. In the Trespaser data files these are all vertices from the vertex list, and so are the convex hull of the shape. However I haven't noticed any side-effects of just finding the min & max x,y,z values of the vertices and writing out the 8 vertices of this bounding cube.

Type 2:

Type 2 geometry contains texture coordinate info, so the corresponding .mapping block is only 4 bytes long.

0x00	uint	Material Handle. 0x00 if no .material block or if .material contains no textures.
0x04	uint	No Material flag, 0 if has .material, 1 if has no .material.
0x08	uint	Default Color 0x00RRGGBB. Used only if there's no .material.
0x0C	float[3]?	Pivot Offset. See description in Type 1 geometry.
0x18	int	(A) Vertex Count
0x1C	int	(B) Textured Normal Count
0x20	int	(C) Face Vertex Index Count
0x24	int	(D) Wrap Vertex Count
0x28	int	(E) Face Count
0x2C	float[3][A]	Vertices.
0x--	TexNorm[B]	Textured Normals, see below (32 bytes each)
0x--	int[C]	Face Vertex Indices (used by Faces)
0x--	FaceInfo[E]	Faces, see below (40bytes each)
0x--	float[3][D]	Wrap Vertices. See description in Type 1 geometry.

TexNorm (32 bytes)

0x00	uint	Vertex index (index into Vertices)
0x04	float[3]	Vertex normal
0x10	float[2]	Texture coordinates
0x18	uint	0x00000001
0x2C	uint	0x00000000

FaceInfo (40 bytes)

0x00	uint	Number of vertices
0x04	uint	Index of first vertex in Face Vertex List (others follow consecutively)
0x08	float[3]	Normal of polygon
0x14	float	-(p dot n), where n = face normal, p = a point on the face, dot = dot product. This is the closest distance from the origin (0,0,0) to the plane containing the face, positive if the origin is in front of the face (in the direction of the normal), negative otherwise.
0x18	uint	Face material (index into .mapping). Must be zero if there are no materials.
0x1C	uint	Type of normals: 0 - Flat shaded, use polygon normal. (Do vertex normals always equal face normal in this case?) 4 - Smooth shaded, use vertex normals. (vertex normals may not equal face normal)
0x20	uint	0x00000000
0x24	uint?	(?) Same for whole object. One of about 20 values in a level. Different levels have different sets of values. eg. 8 1-2 digit values, couple inbetween going to 9 6 digit values, then 6 8 digit values. (Do they match that unknown field in the PID entries?) Seem to be able to write zero for this.

".mapping"

If mapping.NumVerts = 0, NumFaces may be greater than 0, but all entries will be 0. So must take faces from the geometry block. NumVerts=0 seems to happen when a material has all texturenames = "", so really has no texture at all, so NumTexVerts = 0.

(TODO compare to TresView source, found out new things while writing it) Type1 geometry contains texturing info in this block, Type2 has just the material handle (as the texturing info is in the .geometry block). If this block exists, the .material does too.

struct object {
  uint materialHandle;
  uint nverts;
  uint nfaces;
  float2 tvert[nverts];
  uint face[nfaces][3] ;
  uint faceMaterial[nfaces] ;
}

0x00	uint	Material Handle
0x04	uint	Texture Vertex Count (A). Can be zero, which occurs if object is not textured, but .material is still used (will contain one material, with empty strings for textures). Note that this does not necessarily equal the vertex count in the geometry block. To find the info for vertex j of triangle i: tex coord is map.tvert[map.face[i][j]], position is geom.vert[geom.face[i][j]]
0x08	uint	Texture Face Count (B), matches the face count in the .geometry. Can be non-zero even if (A) is zero.
0x0C	float[A][2]	Texture coordinates
0x--	uint[B][3]	Face vertex indices. These index into the Texture coordinates array. eg. tex coord for vert j of face i is tvert[face[i][j]]. If (A) is zero, these are all zero.
0x--	uint[B]	Face material. These index the .material texture lists.

".material"

Contains info on textures, transparency and bump maps used by model. Also some lighting parameters (which are possibly ignored, or overwritten by data elsewhere, the valuetable perhaps). Blocks are all multiples of 24 bytes, minimum 48 bytes. This block exists if and only if there is a .mapping block too. If .mapping.nverts = 0, the map handles all point to the empty string, otherwise tmap points to a non-empty string, and omap and bmap are optional.

Type	Description
uint	Material count = N > 0
uint[N]	Texture map handle list (symbolHandles, see below)
uint[N]	Opacity map handle list (ditto)
uint[N]	Bump map handle list (ditto)
uint[3][N]	Material color list. R,G,B values, 1 long each. Usually all 0x00000080.
uint	Ambient lighting, 0-100 (usually 50 = 0x32)
uint	Diffuse lighting, 0-100 (usually 50 = 0x32)
uint	Specular lighting, 0-100 (usually 50 = 0x32)
uint	Opacity value, 0-100 (usually 100 = 0x64)
uint	Bumpiness value, (usually 0)

The symbol handles are all to strings of the form "Map\LevelName\ImageName.bmp", or an empty string if there is no texture of the type. Textures usually post-fixed with"t2", bumpmaps with "b8", and opacity maps with "o8", though that is not required. eg. Blaht2.bmp, Blahb8.bmp for a bumpmapped texture.

The strings are mapped to the TextureIDs in the PID using a hash function, see (TODO).

 

UTILITIES: GRFHack 0.10 -- extracts all the blocks from GRFs, SCNs, and SAVEGAMES.
	   GeomExt 0.12 -- extracts .geometry as 3ds files. Does most textures.
	   MasterHack 0.1 -- can sorta change savegame positions...

SCN/Savegame - Scene file (level startup)

The SCN and Savegame files use the Groff format, and consist largely of data update blocks which replace information from the GRF. The SCN is basically an 'initial savegame', though the Savegame files contain a few extra blocks specific to savegames. A savegame can be renamed as a SCN, and will then show up on the level load cheat screen (if put in the data directory along with all the other SCNs).

I don't believe the SCN contains any level information that is not derivable from the GRF - the SCN would have been generated from the GRF using a utility, after the GRFs were exported from MAX.

SCN and Savegame blocks

SCNs contain a Header, Partitions, Hierarchy, TerrainHierarchy, and TriggerHierarchy, along with many data blocks (named with a hexadecimal string). That hex number is a hash of a string. Most are of instance names from the GRF, some are strings from the .exe ("DefaultCon", "Render DBase", "AI System", "Joe", "CBloodSplats", "Audio Daemon", "Terrain").

Important note: It appears to be possible to trick Trespasser into creating most of the SCN for us, which is a good thing because we don't need to figure out the format of the other blocks. All blocks except the Header can be deleted and the level will still load. Immediately save the level, then rename the savegame as the required SCN. Trespasser will have recreated all the blocks that are in a normal SCN. (If the blocks are not recreated in this way the level will not restart correctly as there would be no InstanceUpdate blocks to reset the level state.) However, Trespasser will not create a proper Partitions block, so levels may run very slow.

About the GROFF data blocks: The data type must always be 0x80000000. I believe the block handles are ignored. The blocks with hexadecimal names are the ones that we believe update information originally in the GRF. The hexadecimal values are hashes of name from the GRF. For example, the 36byte blocks update CInstances in the GRF, and hence the hex name will be a hash of the instance name in the GRF.

SCN block: "Header" (1348 bytes)

The Hierarchy blocks

These are used along with that Partitions block to define a spatial tree of axis-aligned bounding boxes. If nodeID is the hash of a (case sensitive) instance name, the node's bounding box is the bounding box of that instance. Otherwise nodeID is the hash of a PartitionEntry (see description of hash above). nodeID is always non-zero. There seems to be no limit to the number of children a node can have. The partitions are fixed, so as objects move around a level they are placed into different hierarchy nodes.

In the original Trespasser levels: Each hierarchy block has its own set of partitions. In each hierarchy block there are often more than one node with no parent. These often enclose disjoint parts of the level (eg. one root node for main level objects, one for some outlying objects a long way away).

// Stored in SCN blocks named "Hierarchy", "TriggerHierarchy", and "TerrainHierarchy"

struct HierarchyBlock {
  HierarchyNode nodes[]; // Calculate array size from size of data block??? eg. size/8
}

struct HierarchyNode {
  uint nodeID;   // Hash of node name or of a PartitionEntry.

  uint parentID; // Hash of parent node's name, or zero if no parent.
} // sizeof(HierarchyNode) = 8 bytes

SCN Block: "Hierarchy"
Contains all classes not in other hierarchy blocks? Certainly all of the following: CAnimal, CEntityWater, CGun, CInstance,CLightDirectional, Player, Teleport, ""(no class), CWaterDisturbance. Physics subobjects (all those with instance name beginning with '$') are not included in the hierarchy, except for the body parts of CAnimal and Player. However, levels seem to work fine even if the body parts aren't included in the hierarchy block.

Most nodeIDs use the hash of their instance name, however there are some exceptions. class "Player" uses the string "Player" instead of the instance name (normally "Anne"). Dino body parts (CAnimal values "Body", "Head", and "Tail") and player body parts ("Body", "Foot", and "Hand") are named eg. "$RaptorBody+RaptorB" where "$RaptorBody" is the body part of instance "RaptorB". Note that although there is only one "$RaptorBody" there are entries for the body parts of every instance of dinos eg. there might be a "$RaptorBody+RaptorB300-00". Also, there are entries named eg. "LFoot+31519BCE" and "RFoot+31519BCE" where the number is the hash of "RaptorB" (not sure if should include leading zeros). I don't know which instances these correspond to or how their position is determined (probably two of the joints), but as mentioned above it seems to be okay to not have hierarchy entries for any of these body parts.

SCN Block: "TriggerHierarchy"
Contains all triggers (classes named "*Trigger"). Only CLocationTrigger are stored as a partition tree structure, the other triggers have no 'location', and are stored with parentID = 0.

SCN Block: "TerrainHierarchy"
Contains all CTerrainObj classes. There seem to be some (as yet unknown) restrictions on the terrain hierarchy - with the tree structures I build, some levels to crash on pentium Trespasser builds (but not the K6 build). As a work around I create a single partition containing all the terrain objects as children.

SCN block: "Partitions"

This a list of bounding boxes used by the Hierarchy blocks. The partitions are static (ie. they are not modified, created or deleted by Trespasser, although the ordering (and hence names) may change). In the hierarchy blocks, a partition is identified by a uint, which is the sum of the hashes of a partition's position and size. ie. nodeID = tresHash(&partition.position, 3*sizeof(float)) + tresHash(&partition.size, 3*sizeof(float)). Size of partition node is size of bounding box of child nodes + <0.01,0.01,0.01>

struct PartitionsBlock { // Stored in SCN block named "Partitions"

  char unknown;          // Always 0x10? (Or is it 0x01?)
  PartitionEntry partitions[]; // Calculate array size from size of data block??? eg. (size-1)/28
}

struct PartitionEntry {

  uint flags;      // Flags. When saving a level I set this to 0x10, then the level seems
                   // to work fine. Trespasser seems to recreate the proper flags as they
                   // appear in the savegame if the new level is saved.
  float3 position; // Center of bounding box
  float3 size;     // Half length of bounding box sides

} // sizeof(PartitionEntry) = 28 bytes, except in TestScnNght where size = 24! (no flags)

The Object Update blocks

These hold dynamic state information for objects in the level. The string name of the block is the hash (uppercase hex, without leading zeros) of the instance name (case sensitive) eg."79815A3" for "PDomino-07". The format for most of the object update blocks is unknown. 

struct InstanceUpdateBlock {  // sizeof(block) = 36 or 44 bytes
  float3 position;
  float4 rotation; // Quaternion in order [w,x,y,z]
  float  scale;
  uint   flags;    // (unknown, but similar to PartitionEntry flags)
  // 44 byte blocks have two more fields:

  uint   unknown1; // (usually a small number)
  uint   unknown2; // (always zero?)
}

// Format of other object update blocks (triggers, dinos, teleports etc) is unknown.

Savegame blocks

In addition to the SCN blocks, Savegames contain three extra blocks:

Savegame block: "SaveGameName"

Zero terminated string, name of savegame, as seen in the Trespasser load level dialog box. (Possibly 14 extra unknown bytes after the string.)

Savegame block: "SaveBMP"

Image in standard BMP format. This image is displayed in the Trespasser load level dialog box.

Savegame block: "SaveSCNName"

Zero terminated string, name of the SCN this savegame is based on. Seems to be ignored by Trespasser. (Possibly 6 extra unknown bytes after the string.)

*** From Andres's hacking diary ***)

Found some parsing related stings near CAnimationScript in exe. Nearby strings
were ".asa" and ".asb". Keywords are possibly: scale, rot, pos, frame, end_object,
object, forced_rate, version, Camera. Couldn't figure out how to use them.

*** Other stuff ***

"We used dummy meshes to represent gameplay objects like triggers and
defined game behavior of all objects by typing code into their
object properties buffers"

Utilities

TresEd, unless you're interested in hex data. Then there's GRFHack from the original THS to extract the raw blocks, and there's the unreleased MasterHack by TSOrd.

PID - Texture index

Picture Index files, or PIDs, describe the data in the corresponding SWP files. They consist of a HEADER followed by PALETTE BLOCK(s) then PID ENTRY(s). See the section on the SWP below for information on how the textures must be packed into the SWP.

HEADER

PALETTE BLOCK

PID ENTRY

Utilities

The classic SWPAdd and SWPExt can be used to add/extract textures, but they are hard to use.

SWP/SPZ - Texture data

The SWP/SPZs are swap files holding raw bitmap data for all the texture maps, bump maps and pre-computed mip maps used in the game. An SPZ file is a compressed SWP file, and the data in the SWP is described in the PID file.

If the SWP doesn't exist for a level, or if the 'modified' time/date stamp on the SWP doesn't match the modified time/date stamp on the SPZ, then Trespasser uncompresses the SPZ on to your hard drive to run the level on load-time (this is reflected in the "Level File Copy" progress bar). Thus Trespasser will overwrite a tampered-with SWP if its 'modified' time/date is different

Also, if Trespasser determines it doesn't need the data in the SPZ (ie. SWP already exists and has the correct 'modified' date/time stamp), then it doesn't even look at the data within the SPZ - any non-zero length "dummy SPZ" will satisfy the engine. (Andres - Actually the SPZ must contain the correct swp length in the first 4 bytes, but the rest of the file is ignored if length + date/time match the SWP.)

SPZ

An SPZ is a compressed SWP file. The compression is a variant of run-length encoding. The Demo does not use SPZs.

The SPZ data is divided into blocks consisting of a single byte Block Code, then some Block Data. The decompressed data is appended to the SWP file as the decompression progresses.

Block code: 8 bits specifying how the block data is to be interpreted. A '1' bit specifies a raw data byte, and a '0' bit specifies a two byte command code. Lowest bit corresponds to the first data byte.

Command code: 2 bytes, copy 'count' bytes from swp file 'offset'. Note that the source and destination may overlap.
Offset = command byte 1 + 256 * command byte 2 hi nibble
Count = 3 + command byte 2 low nibble

I'm not certain of the interpretation of the offset, but I believe:
- Must add 0x12 to get offset from startt of the output stream (offset is not relative to the current position)
- Offset is signed (so must sign extend))
- If the offset is before the start of tthe output stream, output zeros. For encoding a string of zeros, Trespasser initially uses an offset -0x12 - NumZeros
- Not sure what happens when required offfset is larger than 3 nibbles. Taken from start of file? From start of current 0x100000 block?

Example:

Data in spz: 9D 77 EE F3 1A 40 74 F6 F0 FA F0 02
Data in swp: 77 77 77 77 77 77 77 1A 40 74 40 40 40 40 02

SWP

The SWP file holds the raw data for all the texture maps, bump maps and pre-computed mip maps used in the game. The bitmaps are described in the PID file, one bitmap per PID ENTRY. The bitmaps are arranged into blocks that are usually 512 or 1024 bytes wide. I assume bitmaps were organised into blocks in order to simplify the memory management needed for the texture caching used by Trespasser. (I'm not sure how high the blocks are, but almost certainly a multiple of the largest texture height, which is 256.) The bitmaps are all palette based, either standard 8-bit bitmap data, or a custom 16-bit format which includes bumpmap information. For more details of the bitmap format see the "bit depth" field of the PID ENTRY description.

The SWP is divided into two sections. The first is 'non-swapable', it is loaded during level startup and remains in memory. This occurs about 3/4 of the way through level loading. Changing the size of this section (in the PID header) noticeably creates a pause during level loading, as more data is being loaded. This section MUST contain at least the smallest mipmap of every texture, or the level won't load (or will crash randomly). If a texture has no mipmaps, the whole texture must be in this section. (except for the large dino textures at the end of some pids?) The second section contains the rest of the textures, which are loaded as they are needed. The area between the first and second sections is (always?) zero, and can be used when adding new textures to the swp.

It appears that textures can be written sequentially one after the other, instead of packed into pages. (This is how all the small dino textures with non-power of two sizes are stored). However, doing this breaks texture wrapping and makes the mipmapping screwy. If the textures are packed into 512 byte wide pages, texture wrapping works fine and happens automatically. (Haven't verified that mipmaps work correctly).

Most textures require all mipmaps to exist until one of the dimensions is less than 16. eg. 16x32 would get divided again, but 15x32 or 8x32 are okay as the smallest mipmap. There are some odd textures with no mipmaps (don't know how they work, haven't been able to write one myself).

Utilities

Again, the classic SWPAdd and SWPExt can be used to add/extract textures, but they are hard to use.

TPA - Audio file format

Updates to the original formats.txt notes (which are below):

Extra data at the end of Effects.tpa identified as a 'Foley table', listing
pairs of SoundMaterial hashes, the sound(s) to play when objects with those
materials collide, and a bunch of extra data (mostly floats).

Changes to the HEADER:
0x00000C  Number of FTEs (zero except for Effects.tpa)
0x000010  Offset to Foley Table (= file size if number of FTEs is zero) (UL)
                                (= wte.Offset+wte.Length where wte is the last wte)

New structure:
FOLEY TABLE ENTRY (FTE) (0x98 bytes)
0x00 SoundMaterial1 hash
0x04 SoundMaterial2 hash
0x08 NumEffectSounds 
     low nibble: = number of Effect hashes (always 1 or 2?). See 0x14.
     next nibble: 0, or 1 if has OtherEffect hash (ie. 1=0x80 because this is the high nibble of the byte).
0x0A ? (float?)
0x10 ?
0x14 dword[2] Effect hashes
0x1A OtherEffect hash. Usually zero, otherwise one of just a few values.
...  ? unidentified data, mostly floats

Other notes:
-ADPCM uses the 'new' DVI form ie. decodde low nibble then high nibble.
-PCM data can be 8bit or 16bit, mono or  stereo.
-For compressed blocks, the UnCompSize ffield is as if each ADPCM block
 writes the predicted value to the stream, plus one extra sample. But
 CoolEdit seems not to write the predicted value. Who is right?

(remainder copied directly from formats.txt)

----------------- TPA ------------------
Updated: 01/26/2000

tpa -- Trespasser Audio Files

Found in the \Trespasser Directory, \Demo\data directory, and the D:\data directory. 
Arbitrarily named (Effects, Ambient, Menu, Stream).

They basically hold all the sound data (PCM or DVI/IMA ADPCM [not Microsoft ADPCM])
for the game, and caption data. TSORD - I've tested substituting the Microsoft
ADPCM (which is about 75% smaller) but it doesn't work because Trespasser doesn't
know how to play the data.

Because of the repeating string "WBOR" throughout the file, I've named most of 
the elements after this "WBOR". Here are my names:
First, there's a HEADER,
Then, a WBOR Table (WBTable), followed by a number of WBOR Table Enteries (WTE's)
Then the rest of the file is composed of WBOR Blocks (WBBlocks), which consist
of a WBOR Block Header (WBH) and the WBOR Block Data (WBD).

Most data is aligned to unsigned longs / intel byte order.

Here are the details so far:
HEADER
0x000000        Version number (Tres will only accept 0x150)
0x000004        Number of WBBlocks (UL)*
0x000008        18 00 00 00 (version number?)
0x00000C        00 00 00 00
0x000010        File size in bytes (UL)
0x000014        Number of WTE's (equal to or greater than #WBBlocks) (UL)
0x000018        BEGINNING OF WBTable

*Notes from TSOrd: 
This (# of WBBlocks) should equal the number of WBOR table entries, 
sometimes it doesn't. The reason for this discrepancy is because the same 
sound data sometimes has two (or more?) SoundID's! There are always more 
WBOR Table Entries than WBOR Blocks when this happens. (SoundID's are stored 
in the WBOR Table Entry and not in the sound data.) My program compensates 
for this by writing multiple WBOR blocks - later I may work on fixing this 
problem properly. [To fix this problem requires naming the sound files,
which will be necessary to use them properly elsewhere. I'm thinking more
on this.]

BEGINNING OF WBTable (Offset = 0x18, each entry = 0x38 length)
There are x number of WTE's (specified in header)... 
o0x00        Sound ID (UL) (first one is always D9 7B A0 F0 - )*
o0x04        Offset of WBBlock from beginning of file (UL)
o0x08        Length of WBlock in bytes (UL)
o0x0C        ?        Other Data (?) (usually 00 00 00 00, but sometimes not - no apparent 
reason, but the hex codes are interesting)
o0x10        ?        Other Data (?) (usually 00 00 00 00, but sometimes not - no apparent 
reason, but the hex codes are interesting)
o0x14        'WBOR'
o0x18        6E 00 00 00
o0x1C        length of WBH in bytes (UL)
o0x20        ADPCM Block Length
o0x24        Length of WBD in bytes (UL)
o0x28        Uncompressed WBOR data length (UL)
o0x2C        Sample Rate (UL)
o0x30        Bitrate (UC)
o0x31        01 -- MONO, 02 -- STEREO (UC)
o0x32        00 -- PCM, 01 -- DVI/IMA ADPCM (UC)
o0x33        00
o0x34        Caption Flag (0x00000000 if no captions, 0x00000024 if captions)

*Notes from TSOrd:
The first WBOR Table Entry points to a block of zeroes that is 16536 or 22120 bytes 
in length. This first entry is the default sound/caption block used whenever the TPA 
is missing the Sound called for. This is a special case but it is *NOT* the only 
case where the sound ID is duplicated between TPA files. It appears that there is 
one discrepancy in that normally a duplicate ID results in a duplicate sound. See 
the DITTO table for more information.

BEGINNING OF WBlocks (Offset = 0x38 * Number of WBOR Table Entries + 0x18)
Described by the WTE's
WBH (info better agree with WTE info)
o0x00        'WBOR'
o0x04        6E 00 00 00
o0x08        length of WBH in bytes (UL) (24 unless captions)
o0x0C        ADPCM Block Length
o0x10        Length of WBD in bytes (UL)
o0x14        Uncompressed WB data length (UL)
o0x18        Sample Rate (UL)
o0x1C        Bitrate (UC)
o0x1D        01 -- MONO, 02 -- STEREO (UC)
o0x1E        00 -- PCM, 01 -- DVI/IMA ADPCM (UC)
o0x1F        00
o0x20        Caption Flag (0x00000000 if no captions, 0x00000024 if captions)
Caption Data -- variable length (Option -- only if o0x20 == 0x00000024)
o0x24        Length of Caption Data
o0x28        Num of caption lines (can't be 0)
o0x2C Duration to display this caption in 250ms increments (I think)
o0x30 Caption text - \0 terminated string (Length of caption can be zero)
(last two fields repeated for each line)




WBD BEGINS

FORMAT OF THE DATA IN THE WBD'S DESCRIBED IN THE HEADERS

UTILITIES: TPAHack 0.2 -- extract all the data in the tpa's to wavs.
MasterHack 0.1 -- extract and replace wav data. 
TPAAdd 0.0 -- Add and replace wave data.

WTD - Terrain

The WTD (World Terrain Data) holds wavelet compressed terrain data. The wavelet coefficients are stored in a zerotree (which is like a quadtree storing children only if they are significant), which is how the terrain supports varying level of detail. Ask me if you want to know the details of how the wavelet encoding/decoding actually works, it's a bit involved and I don't want to go into it here.

WTD HEADER

WTD TREENODE

Created: 03/02/2003

SMK - Video

(copied directly from formats.txt)

Updated: 12/30/1998

smk -- Smacker Video Files

Found in the \Demo\data\menu directory and D:\data\menu directory. Arbitrarily named.

Contains the movies used at the beginning and end of trespasser. This is a standardized 
format rather than a DreamWorks Proprietary format -- Mechwarrior 2 also uses SMK files.

SMACKW32.DLL holds the routines to play them back, but we don't care about that :) ... 

I got a free SMK compressor from www.radgametools.com ... cool stuff.

UTILITIES: Smacker Utilities, available for free download.

DDF - Menu

(copied directly from formats.txt)

Updated: 1/21/2000

Figure these out at some point, they look easy.

Text files which script the appearance and actions of buttons and
windows and such. Basically the GUI layer of the game.

Found in the \Demo\data\menu, D:\data\menu and Trespasser\menu directories.