[RELEASE] wudecon - File/Model Converter for SM1/SM2

LemonHaze · Jan 4, 2019

Hey guys,

This is just a simple converter based on the ShenmueDKSharp SDK. Currently, it supports PKF/PKS, SPR, AFS, GZ, IPAC, TAC/TAD extraction and most importantly, MT5 (all types supported) and MT7 (non-DC models as of now) model conversion.

Models are converted to OBJ and are supplied with MTLs.

Example usage:

Code:

wudecon.exe --allv C:\some\dir C:\output\dir

This command will process C:\some\dir (an extracted TAD/TAC folder) and convert all PKF/PKS, SPR, AFS, GZ, IPAC and MT5 and MT7 files into their own subfolders inside C:\output\dir.

Code:

wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>
        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC needs to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '--allv' or 'v--mt5'

Latest Releases
GitHub

Most of this was only possible with PhilYeahz and derplayer's work on ShenmueDKSharp :)

Wudecon: https://wulinshu.com/

Kodama · Jan 4, 2019

what does this mean to the layman?

LemonHaze · Jan 4, 2019

This means that you can now convert or unpack many file formats. Including models and maps:

This can be useful for lots of things like printing out models from the game. The converter currently writes out OBJ's which can then be converted into STL or something useful for printing. (note, the OBJ writer is pretty new and it's best to import them into Blender and then export to a file of your liking)

Willom · Jan 26, 2019

By using this program and files ripped from my XBOX copy I found a low-poly Ren. Still trying to find the proper one.

HazDS · Jan 28, 2019

Amazing work, thanks for the release!

shenhaZ · Jan 29, 2019

good stuff

ReeceKun · Mar 8, 2019

Epic.. is it all 100% model rips , landscapes and characters?

ReeceKun · Mar 23, 2019

I've had a look now and Ive tested with Shenmue 2,

from what I can tell it can process single MT7 files (like MT5 with embedded pvr texture)
but It cant process MAPM files from the extracted PKS files (Textures in corresponding PKF files)
unless im just using the wrong extraction command ?

LemonHaze · Mar 25, 2019

It works with both SM1 and SM2 files, such as ".mt7", ".mt5", ".mapm", ".prop", ".chrm".

This can be done by extracting the entire TAD with a specific command, which will build up the entries for those embedded textures and then extract them and their models and assign them to the correct materials in OBJs.

Code:

wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt7>

For example,

Code:

.\wudecon.exe --tacfull "E:\Games\Shenmue\sm1\archives\dx11\data\disk_5be2c578.tad" "E:\output\sm1\extracted_archive\" "E:\output\converted_models\" mt5

For Shenmue 1, and:

Code:

.\wudecon.exe --tacfull "E:\Games\Shenmue\sm2\archives\dx11\data\disk_5be2c4e2.tad" "E:\output\sm2\extracted_archive\" "E:\output\converted_models\" mt7

For Shenmue 2.

You'll get two folders as the command and directory names imply, one with the extracted TAD archive contents and another with the archive, except the models and textures are converted respectively.

Please use the latest version on GitHub if you're running into any issues, too. Although, I'll update the OP, too.

ReeceKun · Mar 26, 2019

Ive always struggled with command prompt exe's , Ive extracted the wudcon folder to the d: directory (just to make running it easier than typing its location, I created an EXT folder and a MODELS folder I copyed the links as you describe in your code for shenmue 2 and nothing happend... GUI ?

Code:

D:\>wudecon.exe --tacfull "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archive
s\dx11\data\disk_5be2c4e2.tad" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\ar
chives\dx11\data\Ext\" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archives\d
x11\data\Models\" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

D:\>

LemonHaze · Mar 26, 2019

ReeceMix said:

Ive always struggled with command prompt exe's , Ive extracted the wudcon folder to the d: directory (just to make running it easier than typing its location, I created an EXT folder and a MODELS folder I copyed the links as you describe in your code for shenmue 2 and nothing happend... GUI ?

Code:

D:\>wudecon.exe --tacfull "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archive
s\dx11\data\disk_5be2c4e2.tad" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\ar
chives\dx11\data\Ext\" "D:\[SteamGames]\steamapps\common\SMLaunch\sm2\archives\d
x11\data\Models\" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

D:\>

I bet if you remove the last \'s from your EXT and MODELS paths, it will work. As far as a GUI is concerned, there's ShenmueHD ModTools which might do something similar, but unfortunately I have no plans to do anything like that as my time is quite limited.

ReeceKun · Mar 27, 2019

I coped the tad/tac to a closer root folder to simplify paths

with \

Code:

D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

without \ seems to start to work (but thinks SDK is in H: & doesnt like the EXT folder) ?

Code:

D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

Filename database initialized: 68679 entries.
Oops! D:\SM2DATA\disk_5be2c4e2.tad failed!
Exception: System.NullReferenceException: Object reference not set to an instanc
e of an object.
   at ShenmueDKSharp.Files.Containers.TAC.Unpack(Boolean verbose, Boolean raymon
f, String folder) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAC.cs:line 66
   at wudecon.Program.ExtractTAC(String tadFilepath, String folder) in H:\wudeco
n\wudecon\Program.cs:line 654
Processing GZ..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing GZ!
Processing AFS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing AFS!
Processing IPAC..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing IPAC!
Processing SPR..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing SPR!
Processing PKF..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKF!
Processing PKS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKS!
Processing MT7..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing MT7!
Completed 0/0 operations.

D:\>

shengoro86 · Mar 27, 2019

nice work!

LemonHaze · Mar 27, 2019

ReeceMix said:

I coped the tad/tac to a closer root folder to simplify paths

with \

Code:

D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

        wudecon <mode> <input> <output>
        wudecon --all <input dir> <output dir>
        wudecon --mt5 <mt5 file> <obj file>
        wudecon --mt5 <dir with mt5's> <output dir>
        wudecon --mt7 <mt7 file> <obj file>
        wudecon [--pkf|--pks|--spr|--ipac|--gz|--afs] <source file> <output dir>

        wudecon --tc <tad file> <output dir>
        wudecon --tacfile <file in tac to extract> <output dir>
        wudecon --tacfull <tad file> <tac output dir> <model output dir> <mt5/mt
7>

        Batch conversion possible by replacing file argument for path
        When using --tacfile (or -tfile) the path to search for within the TAC n
eeds to be in lowercase.
        For verbose logging, add 'v' to the beginning or end of the mode, e.g. '
--allv' or 'v--mt5'

without \ seems to start to work (but thinks SDK is in H: & doesnt like the EXT folder) ?

Code:

D:\>wudecon --tacfull "D:\SM2DATA\disk_5be2c4e2.tad" "D:\EXT" "D:\MODELS" mt7
wudecon v1.0.7023.39908

Filename database initialized: 68679 entries.
Oops! D:\SM2DATA\disk_5be2c4e2.tad failed!
Exception: System.NullReferenceException: Object reference not set to an instanc
e of an object.
   at ShenmueDKSharp.Files.Containers.TAC.Unpack(Boolean verbose, Boolean raymon
f, String folder) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAC.cs:line 66
   at wudecon.Program.ExtractTAC(String tadFilepath, String folder) in H:\wudeco
n\wudecon\Program.cs:line 654
Processing GZ..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing GZ!
Processing AFS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing AFS!
Processing IPAC..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing IPAC!
Processing SPR..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing SPR!
Processing PKF..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKF!
Processing PKS..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing PKS!
Processing MT7..
D:\EXT does not exist as a file, falling back to batch mode.
Finished processing MT7!
Completed 0/0 operations.

D:\>

Cool! Seems like some stuff was implemented in ShenmueDKSharp which breaks wudecon, I've fixed this now and tested it, please try this build :) Also, change your --tacfull to --tacfullv or v--tacfull to enable verbose logging.

Wudecon:

Team Wulinshu - Home

The team behind mods, tools, games & research for the Shenmue video game series.

wulinshu.com

ReeceKun · Mar 27, 2019

Worked and Its Freaking Epic, Thank you

But now that we've found love what are we gonna do with it?

I'm sure porting to Unreal / Unity / ..Fallout 5.. etc will be a fun way to go, but what do you think are the chances of importing new mt5/mt7 models to replace old models? (yes cols are in mapinfo.bin but if a col exporter cant be made at the least keeping the same model structure would work)

Secondly
Dreamcast Mods would crash due to memory issues (swapping beta sakuragouka will crash because map is bigger/more complex) even so if new models were possible could the new Shenmue PC '..emulator..' take more expansion.

Personally I am going to practice my animation skills and see how I feel after playing Shenmue 3

LemonHaze · Mar 27, 2019

Glad you got it working :).

To be honest, there's still a lot to do, there are some normals issues which generate some weird looking stuff, for example on Ryo's jacket.. although, thankfully, these can be easily fixed or otherwise mediated, they along with other issues still prevent us from writing our own models. @philyeahz has had some luck in replacing models, but I'm unsure to what extent this is.

With regards to Dreamcast Mods, generally, they should all pretty much work out-of-the-box, for example, texture swapping, but when it comes to some of the AM2-specific formats, it gets a bit hairy, as d3t have made some changes and/or brought in features from the SM2 codebase which ultimately mean that there'll need to be some changes. For example, the entire SCN sections of MAPINFO files can be found in a new folder in the archives of the re-releases, SCENEARIO\act01_d000.scn (for Dobuita).

I do have plans myself, but I'm yet to disclose what they are properly, yet..so stay tuned for my answer on that front!

ReeceKun · Mar 28, 2019

Just extracted the GDIs of Whats Shenmue & The Passport disk by using the --all command on the extracted folder, so glad I didnt need to faff about selecting individual models/folders , Thanks again x

ReeceKun · Apr 4, 2019

Just started importing into 3DS max, Ive noticed a few issues so I will just list them for reference
For Shenmue 1 (havent looked at Shenmue 2 models yet)
The Extraction process for Images leaves .TEXN. It would be better if it left the PVR for editing with WinPVR

All Objs are combined as a single Object

All Materials have their specularity at 100%

and Ive noticed a few maps with missing Polys, but it does convert stuff Heliwigs didnt (my guess is animating objects broke the conversion process)

For things like street roads Ive talked before how there is an odd mirroring technique on certain polygons

This is how it was explained to me, but I dont really expect or want 3DS Max to render textures this way, Im sure you are aware of this but I certainly dont think its worth worrying about unless you want UE4/Unity to render in the same method

by TapamN » Sat Aug 01, 2015 1:03 pm

The Dreamcast's 3D processor has a mode that can mirror textures. It's not on an arbitrary location, but whenever the texture repeats. This can be done on one or both axis of the texture. It is exactly equivalent to the following in OpenGL:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_MIRRORED_REPEAT);

So if this is your entire texture:
/

Instead of repeating it and getting
////////
////////

It will mirror it each time it repeats and you can get
/\/\/\/\
/\/\/\/\
or
/\/\/\/\
\/\/\/\/

The ripper tools either aren't recording the flip flags, or the model format the ripper generates doesn't support texture mirroring. The original MT5/MT7 will have the flags stored somewhere in them. The "Ninja" model format, used in the DC Bios, Sonic Adventure, Skies of Arcadia, Rez, and more, has what is almost a raw hardware command stored in it to setup how polygons are drawn; Shenmue probably does something similar.

I think Sega used some version of Lightwave for modeling, so Lightwave at least probably supports texture mirroring. Even if a modeling program or format doesn't support mirroring, it would still be possible, with a little extra work, for a convertor to work around it by subdividing polygons with mirrored textures and baking in the UV mirroring manually.

LemonHaze · Apr 7, 2019

ReeceMix said:
Just started importing into 3DS max, Ive noticed a few issues so I will just list them for reference
For Shenmue 1 (havent looked at Shenmue 2 models yet)
The Extraction process for Images leaves .TEXN. It would be better if it left the PVR for editing with WinPVR

All Objs are combined as a single Object

All Materials have their specularity at 100%

and Ive noticed a few maps with missing Polys, but it does convert stuff Heliwigs didnt (my guess is animating objects broke the conversion process)

For things like street roads Ive talked before how there is an odd mirroring technique on certain polygons

This is how it was explained to me, but I dont really expect or want 3DS Max to render textures this way, Im sure you are aware of this but I certainly dont think its worth worrying about unless you want UE4/Unity to render in the same method

Definitely aware of these sorts of issues - however I've mad mixed results depending on which application you import the resulting OBJ files out. Try importing them into Blender, you should see that they are separate there.

Boulotaur2025 · Apr 11, 2019

Hello everyone,

I'm having issues unpacking tac files.
My goal is to extract textures and resize them with something like NNEDI3.
I use the following command :


D:\wudecon>wudecon.exe --tc "D:\Games\Shenmue\sm1\archives\dx11\data\disk_5b7d8ad1.tac" "D:\EXT"
wudecon v1.0.7023.39908

Oops! D:\Games\Shenmue\sm1\archives\dx11\data\disk_5b7d8ad1.tac failed!
Exception: System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
   at System.IO.BinaryReader.ReadUInt32()
   at ShenmueDKSharp.Files.Containers.TADEntry.Read(BinaryReader reader) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAD.cs:line 226
   at ShenmueDKSharp.Files.Containers.TAD._Read(BinaryReader reader) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAD.cs:line 98
   at ShenmueDKSharp.Files.BaseFile.Read(BinaryReader reader) in H:\wudecon\ShenmueDKSharp\Files\BaseFile.cs:line 87
   at ShenmueDKSharp.Files.BaseFile.Read(Stream stream) in H:\wudecon\ShenmueDKSharp\Files\BaseFile.cs:line 77
   at ShenmueDKSharp.Files.BaseFile.Read(String filepath) in H:\wudecon\ShenmueDKSharp\Files\BaseFile.cs:line 66
   at ShenmueDKSharp.Files.Containers.TAD..ctor(String filepath) in H:\wudecon\ShenmueDKSharp\Files\Containers\TAD.cs:line 60
   at wudecon.Program.ExtractTAC(String tadFilepath, String folder) in H:\wudecon\wudecon\Program.cs:line 651
Completed 0/0 operations.

I am running an up-to-date Windows 10.
Default .NET package shipped with Windows 10 (I didn't install Visual Studio)

[RELEASE] wudecon - File/Model Converter for SM1/SM2

LemonHaze

Kodama

Now Loading...

LemonHaze

Willom

A Superstar Rock

Attachments

HazDS

shenhaZ

Mr. Nagai

ReeceKun

ReeceKun

LemonHaze

ReeceKun

LemonHaze

ReeceKun

shengoro86

Please Kill Me.

LemonHaze

Team Wulinshu - Home

ReeceKun

LemonHaze

ReeceKun

ReeceKun

LemonHaze

Boulotaur2025