[RELEASE] wudecon - File/Model Converter for SM1/SM2

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
Hey guys,

This is just a simple converter based on the ShenmueDKSharp SDK. Currently, it supports PKF/PKS, SPR, AFS, GZ, IPAC, TAC/TAD extraction and most importantly, MT5 and MT7 model conversion.

Models are converted to OBJ and are supplied with MTLs.

Example usage:

Code:
wudecon.exe --allv C:\some\dir C:\output\dir

This command will process C:\some\dir (an extracted TAD/TAC folder) and convert all PKF/PKS, SPR, AFS, GZ, IPAC and MT5 and MT7 files into their own subfolders inside C:\output\dir.

Code:
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>
        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC needs to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '--allv' or 'v--mt5'

Latest Releases
GitHub

Most of this was only possible with PhilYeahz and derplayer's work on ShenmueDKSharp :)
 

Attachments

  • wudecon_v1.0.7061.27731_windebug.zip
    2 MB · Views: 83
Last edited:

Kodama

Now Loading...
Banned
Joined
Jul 28, 2018
Location
New York
PSN
Shrooms
XBL
Grampa Gohan
Steam
(BANGS) BASS wub wub
Dreamcast Now
Kodama
what does this mean to the layman?
 
OP
OP
LemonHaze

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
This means that you can now convert or unpack many file formats. Including models and maps:

unknown.pngunknown.png

This can be useful for lots of things like printing out models from the game. The converter currently writes out OBJ's which can then be converted into STL or something useful for printing. (note, the OBJ writer is pretty new and it's best to import them into Blender and then export to a file of your liking)
 
Last edited:
Joined
Aug 21, 2018
Location
North Wales, UK
Favourite title
Shenmue
Currently playing
with myself
Epic.. is it all 100% model rips , landscapes and characters?
 
Last edited:
Joined
Aug 21, 2018
Location
North Wales, UK
Favourite title
Shenmue
Currently playing
with myself
I've had a look now and Ive tested with Shenmue 2,

from what I can tell it can process single MT7 files (like MT5 with embedded pvr texture)
but It cant process MAPM files from the extracted PKS files (Textures in corresponding PKF files)
unless im just using the wrong extraction command ?
 
OP
OP
LemonHaze

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
It works with both SM1 and SM2 files, such as ".mt7", ".mt5", ".mapm", ".prop", ".chrm".

This can be done by extracting the entire TAD with a specific command, which will build up the entries for those embedded textures and then extract them and their models and assign them to the correct materials in OBJs.

Code:
wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt7>

For example,

Code:
.\wudecon.exe --tacfull "E:\Games\Shenmue\sm1\archives\dx11\data\disk_5be2c578.tad" "E:\output\sm1\extracted_archive\" "E:\output\converted_models\" mt5

For Shenmue 1, and:

Code:
.\wudecon.exe --tacfull "E:\Games\Shenmue\sm2\archives\dx11\data\disk_5be2c4e2.tad" "E:\output\sm2\extracted_archive\" "E:\output\converted_models\" mt7

For Shenmue 2.

You'll get two folders as the command and directory names imply, one with the extracted TAD archive contents and another with the archive, except the models and textures are converted respectively.

Please use the latest version on GitHub if you're running into any issues, too. Although, I'll update the OP, too.
 
Joined
Aug 21, 2018
Location
North Wales, UK
Favourite title
Shenmue
Currently playing
with myself
Ive always struggled with command prompt exe's , Ive extracted the wudcon folder to the d: directory (just to make running it easier than typing its location, I created an EXT folder and a MODELS folder I copyed the links as you describe in your code for shenmue 2 and nothing happend... GUI ? :p
Code:
D:\>wudecon.exe --tacfull "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archive
s\dx11\data\disk_5be2c4e2.tad" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\ar
chives\dx11\data\Ext\" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archives\d
x11\data\Models\" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

D:\>
 
OP
OP
LemonHaze

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
Ive always struggled with command prompt exe's , Ive extracted the wudcon folder to the d: directory (just to make running it easier than typing its location, I created an EXT folder and a MODELS folder I copyed the links as you describe in your code for shenmue 2 and nothing happend... GUI ? :p
Code:
D:\>wudecon.exe --tacfull "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archive
s\dx11\data\disk_5be2c4e2.tad" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\ar
chives\dx11\data\Ext\" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archives\d
x11\data\Models\" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

D:\>

I bet if you remove the last \'s from your EXT and MODELS paths, it will work. As far as a GUI is concerned, there's ShenmueHD ModTools which might do something similar, but unfortunately I have no plans to do anything like that as my time is quite limited.
 
Joined
Aug 21, 2018
Location
North Wales, UK
Favourite title
Shenmue
Currently playing
with myself
I coped the tad/tac to a closer root folder to simplify paths

with \
Code:
D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

without \ seems to start to work (but thinks SDK is in H: & doesnt like the EXT folder) ?

Code:
D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

Filename database initialized: 68679 entries.
Oops! D:\SM2DATA\disk_5be2c4e2.tad failed!
Exception: System.NullReferenceException: Object reference not set to an instanc
e of an object.
   at ShenmueDKSharp.Files.Containers.TAC.Unpack(Boolean verbose, Boolean raymon
f, String folder) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAC.cs:line 66
   at wudecon.Program.ExtractTAC(String tadFilepath, String folder) in H:\wudeco
n\wudecon\Program.cs:line 654
Processing GZ..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing GZ!
Processing AFS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing AFS!
Processing IPAC..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing IPAC!
Processing SPR..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing SPR!
Processing PKF..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKF!
Processing PKS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKS!
Processing MT7..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing MT7!
Completed 0/0 operations.

D:\>
 
OP
OP
LemonHaze

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
I coped the tad/tac to a closer root folder to simplify paths

with \
Code:
D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

without \ seems to start to work (but thinks SDK is in H: & doesnt like the EXT folder) ?

Code:
D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

Filename database initialized: 68679 entries.
Oops! D:\SM2DATA\disk_5be2c4e2.tad failed!
Exception: System.NullReferenceException: Object reference not set to an instanc
e of an object.
   at ShenmueDKSharp.Files.Containers.TAC.Unpack(Boolean verbose, Boolean raymon
f, String folder) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAC.cs:line 66
   at wudecon.Program.ExtractTAC(String tadFilepath, String folder) in H:\wudeco
n\wudecon\Program.cs:line 654
Processing GZ..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing GZ!
Processing AFS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing AFS!
Processing IPAC..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing IPAC!
Processing SPR..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing SPR!
Processing PKF..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKF!
Processing PKS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKS!
Processing MT7..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing MT7!
Completed 0/0 operations.

D:\>

Cool! Seems like some stuff was implemented in ShenmueDKSharp which breaks wudecon, I've fixed this now and tested it, please try this build :) Also, change your --tacfull to --tacfullv or v--tacfull to enable verbose logging.
 

Attachments

  • wudecon-1.0.7025.24479_windebug.zip
    1.9 MB · Views: 34
Last edited:
Joined
Aug 21, 2018
Location
North Wales, UK
Favourite title
Shenmue
Currently playing
with myself
Worked and Its Freaking Epic, Thank you

But now that we've found love what are we gonna do with it?

I'm sure porting to Unreal / Unity / ..Fallout 5.. etc will be a fun way to go, but what do you think are the chances of importing new mt5/mt7 models to replace old models? (yes cols are in mapinfo.bin but if a col exporter cant be made at the least keeping the same model structure would work)

Secondly
Dreamcast Mods would crash due to memory issues (swapping beta sakuragouka will crash because map is bigger/more complex) even so if new models were possible could the new Shenmue PC '..emulator..' take more expansion.

Personally I am going to practice my animation skills and see how I feel after playing Shenmue 3
 
Last edited:
OP
OP
LemonHaze

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
Glad you got it working :).

To be honest, there's still a lot to do, there are some normals issues which generate some weird looking stuff, for example on Ryo's jacket.. although, thankfully, these can be easily fixed or otherwise mediated, they along with other issues still prevent us from writing our own models. @philyeahz has had some luck in replacing models, but I'm unsure to what extent this is.

With regards to Dreamcast Mods, generally, they should all pretty much work out-of-the-box, for example, texture swapping, but when it comes to some of the AM2-specific formats, it gets a bit hairy, as d3t have made some changes and/or brought in features from the SM2 codebase which ultimately mean that there'll need to be some changes. For example, the entire SCN sections of MAPINFO files can be found in a new folder in the archives of the re-releases, SCENEARIO\act01_d000.scn (for Dobuita).

I do have plans myself, but I'm yet to disclose what they are properly, yet..so stay tuned for my answer on that front! :D
 
Joined
Aug 21, 2018
Location
North Wales, UK
Favourite title
Shenmue
Currently playing
with myself
Just extracted the GDIs of Whats Shenmue & The Passport disk by using the --all command on the extracted folder, so glad I didnt need to faff about selecting individual models/folders , Thanks again x
 
Joined
Aug 21, 2018
Location
North Wales, UK
Favourite title
Shenmue
Currently playing
with myself
Just started importing into 3DS max, Ive noticed a few issues so I will just list them for reference
For Shenmue 1
(havent looked at Shenmue 2 models yet)
The Extraction process for Images leaves .TEXN. It would be better if it left the PVR for editing with WinPVR

All Objs are combined as a single Object

All Materials have their specularity at 100%

and Ive noticed a few maps with missing Polys, but it does convert stuff Heliwigs didnt (my guess is animating objects broke the conversion process)

For things like street roads Ive talked before how there is an odd mirroring technique on certain polygons



This is how it was explained to me, but I dont really expect or want 3DS Max to render textures this way, Im sure you are aware of this but I certainly dont think its worth worrying about unless you want UE4/Unity to render in the same method
by TapamN » Sat Aug 01, 2015 1:03 pm


The Dreamcast's 3D processor has a mode that can mirror textures. It's not on an arbitrary location, but whenever the texture repeats. This can be done on one or both axis of the texture. It is exactly equivalent to the following in OpenGL:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_MIRRORED_REPEAT);

So if this is your entire texture:
/

Instead of repeating it and getting
////////
////////

It will mirror it each time it repeats and you can get
/\/\/\/\
/\/\/\/\
or
/\/\/\/\
\/\/\/\/

The ripper tools either aren't recording the flip flags, or the model format the ripper generates doesn't support texture mirroring. The original MT5/MT7 will have the flags stored somewhere in them. The "Ninja" model format, used in the DC Bios, Sonic Adventure, Skies of Arcadia, Rez, and more, has what is almost a raw hardware command stored in it to setup how polygons are drawn; Shenmue probably does something similar.

I think Sega used some version of Lightwave for modeling, so Lightwave at least probably supports texture mirroring. Even if a modeling program or format doesn't support mirroring, it would still be possible, with a little extra work, for a convertor to work around it by subdividing polygons with mirrored textures and baking in the UV mirroring manually.
 
Last edited:
OP
OP
LemonHaze

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
Just started importing into 3DS max, Ive noticed a few issues so I will just list them for reference
For Shenmue 1
(havent looked at Shenmue 2 models yet)
The Extraction process for Images leaves .TEXN. It would be better if it left the PVR for editing with WinPVR

All Objs are combined as a single Object

All Materials have their specularity at 100%

and Ive noticed a few maps with missing Polys, but it does convert stuff Heliwigs didnt (my guess is animating objects broke the conversion process)

For things like street roads Ive talked before how there is an odd mirroring technique on certain polygons



This is how it was explained to me, but I dont really expect or want 3DS Max to render textures this way, Im sure you are aware of this but I certainly dont think its worth worrying about unless you want UE4/Unity to render in the same method

Definitely aware of these sorts of issues - however I've mad mixed results depending on which application you import the resulting OBJ files out. Try importing them into Blender, you should see that they are separate there.
 
Joined
Apr 10, 2019
Hello everyone,

I'm having issues unpacking tac files.
My goal is to extract textures and resize them with something like NNEDI3.
I use the following command :


D:\wudecon>wudecon.exe --tc "D:\Games\Shenmue\sm1\archives\dx11\data\disk_5b7d8ad1.tac" "D:\EXT"
wudecon v1.0.7023.39908

Oops! D:\Games\Shenmue\sm1\archives\dx11\data\disk_5b7d8ad1.tac failed!
Exception: System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
at System.IO.BinaryReader.ReadUInt32()
at ShenmueDKSharp.Files.Containers.TADEntry.Read(BinaryReader reader) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAD.cs:line 226
at ShenmueDKSharp.Files.Containers.TAD._Read(BinaryReader reader) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAD.cs:line 98
at ShenmueDKSharp.Files.BaseFile.Read(BinaryReader reader) in H:\wudecon\ShenmueDKSharp\Files\BaseFile.cs:line 87
at ShenmueDKSharp.Files.BaseFile.Read(Stream stream) in H:\wudecon\ShenmueDKSharp\Files\BaseFile.cs:line 77
at ShenmueDKSharp.Files.BaseFile.Read(String filepath) in H:\wudecon\ShenmueDKSharp\Files\BaseFile.cs:line 66
at ShenmueDKSharp.Files.Containers.TAD..ctor(String filepath) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAD.cs:line 60
at wudecon.Program.ExtractTAC(String tadFilepath, String folder) in H:\wudecon\wudecon\Program.cs:line 651
Completed 0/0 operations.


I am running an up-to-date Windows 10.
Default .NET package shipped with Windows 10 (I didn't install Visual Studio)
 
Top