This week, I’d like to go over my release toolchain that I’ve developed to take the guesswork out of releasing mods.
Since I work on fairly large and completely separate mod projects and switch between them frequently, finding ways to spend less time doing “grunt work” and more time focusing on making new and cool stuff became a serious priority. It’s extremely easy to get bogged down in the release phase because you’re afraid of missing something and so you start second- and third- and forth-guessing yourself. This tended to make me update-adverse. I wouldn’t want to patch my own mods until I had spent time making sure everything was included and packaged up correctly. This is stressful and fatiguing.
I’d rather just let a script handle all of that! So, I wrote some. Save time, save sanity.
I use a collection of scripts to solve the following problems:
- Ensure reproducibility. Every release package built from a particular set of files should always be the same, every time.
- Ensure that no files accidentally get left out
- Enable fast iteration and testing by reducing the time it takes me to create a release package
- Reduce stress by not having to worry about accidentally leaving something out. Try as I might, this almost always happened when I packaged my mods manually (or even using the CK!). This means no embarassing “1.0.1a update: oops I accidentally left out a texture file” mod releases down the road.
- Increase confidence in the things I release. If a user says “Blah Blah doesn’t work”, you can rule out an improperly packaged release archive and focus on the real problem.
- Automate as much of the process as possible. As much should be driven from a script or command line as possible. No GUIs or mousing around!
All of this assumes that you are using Mod Organizer. Using MO gives you the freedom to work in a single folder per project, and quickly switch between releases you build in order to test quickly.
Project Directory and Archive.exe Files
I make a copy of Archive.exe from the Skyrim base game directory and put it in my project directory for the mod I’m working on using Mod Organizer. This is so the Release Builder script can easily get to it to generate the BSA archive. I also create an Archive Builder and an Archive Manifest text file. These are both used to drive the command line version of Archive.exe.
The Archive Builder file is a text file that Archive.exe uses to know how to package a BSA archive from command line options. This feature is not very well documented (even the CK Wiki has gaps in documentation regarding this feature), so I hope this is useful. Here is Frostfall’s Archive Builder file. (Also found here on GitHub.)
Log: Campfire\FrostfallArchiveLog.txt New Archive Check: Meshes Check: Textures Check: Menus Check: Sounds Check: Voices Check: Shaders Check: Trees Check: Fonts Check: Misc Check: Retain Directory Names Check: Retain File Names Check: Compress Archive Set File Group Root: Data\ Add File Group: ./FrostfallArchiveManifest.txt Save Archive: ./Frostfall.bsa
The Archive Manifest file is a text file that is a list of every file that will be packaged in the mod, usually inside the BSA archive. I keep this file religiously up to date as I work. (However, that still leaves room for human error. But I have another script in the process that sanity checks this, below.)
Here is an exerpt of Frostfall’s Archive Manifest file. It’s too long to post here in its entirety, see the full version on GitHub.
meshes\frostfall\_Frost_VaporBlastAreaEffect.nif meshes\frostfall\_frost_WoodHarvestingAnimAxe.nif Scripts\FrostfallAPI.pex Scripts\FrostUtil.pex Scripts\qf__de_trackingquest_010177d7.pex Scripts\_FrostInternal.pex Scripts\_Frost_APDatastoreDefaultData.pex Scripts\_Frost_ArmorProtectionDatastoreHandler.pex Scripts\_Frost_ArmorSpellScript.pex Scripts\_Frost_ArmorSpellUpdateWarmth.pex Scripts\_Frost_BaseSystem.pex Scripts\_Frost_BoundCloakRemoveScript.pex Scripts\_Frost_BoundCloakScript.pex Scripts\_Frost_BranchAliasScript.pex Scripts\_Frost_BranchHarvestNodeController.pex
The Release Builder Script
I write a Python script for packaging up my BSA archives, moving files to the appropriate place, and zipping everything up into a ZIP file that I can directly upload to the Nexus. You just give it a version number, and it’s off to the races. The output of this script is a new folder in Mod Organizer’s mods directory ready to be immediately enabled in MO for testing, along with the ZIP file for uploading assuming that everything in testing checks out. This single part of the toolchain makes the process almost push-button.
You can find the release builder script for Frostfall on GitHub. This is a good example because it’s packaging up a plug-in file, a BSA archive, and several files that aren’t included in the BSA, like the StorageUtil dll and the readmes. It’s a pretty broad example. I usually copy and paste this script for each project, tweaking it slightly for each one to suit my needs. This code won’t win any Python awards, it was hacked together fairly quickly in order to solve a problem, but it gets the job done. It also has some ASCII art at the top. (Unnecessary, but it makes me happy when I see it. What can I say, I enjoy simple pleasures.) This script uses both the Archive Builder and Archive Manifest files to do its job.
Here’s what it looks like to run:
C:\ModOrganizer\mods\Campfire>python Frostfall_BuildRelease.py =============================== | Frostfall Release Builder | | \/ | | _\_\/\/_/_ | | _\_\/_/_ | | __/_/\_\__ | | / /\/\ \ | | /\ | =============================== Enter the release version: 3.0.3 Creating temp directories... Copying project files... Creating new build... Generating BSA archive... Removing temp files... Created ./Frostfall 3.0.3 Release/Frostfall_3_0_3_Release.zip Done!
And that’s it, Frostfall is now built and ready for testing and redistribution.
I hit Refresh in MO’s left pane and my build now shows up and I can jump in immediately.
Manifest Checker
I run another script before I run the Release Builder, called (unimaginatively) the Manifest Checker. This is another kind of crappily hacked together Python script, but it does three things:
- Makes sure that every file listed in the Manifest Archive actually exists on the file system (prevents trying to build an incomplete BSA archive).
- Checks to see if any files are in the project directory are not in the Manifest Archive (prevents accidentally leaving a file out).
- Allows you to give it file prefixes you want to ignore, in case you work on multiple projects out of a single directory.
The Manifest Checker for Frostfall is available on GitHub.
This python script takes n-number of arguments. The first argument is always the Archive Manifest text file, the second is always the name of the mod itself, and the third through n are file name prefixes you wish to ignore.
I almost always never run this python script by hand, I run it from a batch file, due to the number of file prefixes I want to usually ignore. Here is what my Frostfall Manifest Check batch file looks like. (Also on GitHub.)
@ECHO OFF python manifestcheck.py FrostfallArchiveManifest.txt Frostfall _Test _Camp Camp _Seed Seed SKI ski skyui _DE _de _HN ddUnequip BladesSparringScript C00TrainerScript C00VilkasScript CompanionsSingleCombatantScript DGIntimidateAliasScript DGIntimidatePlayerScript JsonUtil MS11CalixtoScript StorageUtil TentSystem
Here is what it looks like to run it.
C:\ModOrganizer\mods\Campfire>_Frost_ManifestCheck.bat
===============================================================
Checking Frostfall project files...
===============================================================
Parsing manifest...
OK - All files in manifest found.
Parsing project directories...
WARN - ./readmes: FrostDevKit_readme.txt found in project directory, but not in manifest file!
WARN - ./readmes: Frostfall_SkyUI_AddOn_readme.txt found in project directory, but not in manifest file!
OK - ./Interface/Frostfall
WARN - ./Interface/exported/widgets/Frostfall: status.swf found in project directory, but not in manifest file!
OK - ./Interface/Translations/
OK - ./meshes/Frostfall
OK - ./textures/Frostfall
OK - ./Scripts
OK - ./Scripts/Source
OK - Skipping ./SEQ
OK - ./sound/fx/Frostfall
As we see, it shows us that, yes, every file listed in the Archive Manifest can indeed be found in the file system. It also points out a few files to us that it found that weren’t in the Archive Manifest (and also didn’t match any of the exclusions listed in the batch file) and brings them to our attention. We see “OK” on directories that it either skipped (because there were no files, for instance, Frostfall doesn’t need an SEQ file) or had a complete match between the file system and what was in the manifest.
Thankfully, for Frostfall, all of these WARN (warnings) are ones I expect and I know what they are. No surprises. I’d feel very confident building and releasing this.
Workflow
The workflow for this toolchain is simple.
- I run the Manifest Checker batch file. If I forgot to add something to the Archive Manifest, or if the Archive Manifest has files in it that no longer exist, I make the required changes and run it again until everything is gravy.
- I run the Release Builder.
- I test the built release.
- If everything looks good, I upload the ZIP file that the Release Builder created.
Done! No stress, fast releases.
Closing Thoughts
Of course, every project is different. You’ll need to adapt these files to your own needs, should you choose to use them. Do what works best for you.
It might seem like a lot of upfront work, but I can personally testify to the amount of sanity and speed I’ve regained by doing things this way. This is especially true for my larger projects like Frostfall. I spend a lot more time doing fun modding stuff and a lot less time doing crappy file packaging every time I want to fix a quick bug.
Anyone that wants to use any code on this blog post is free to do so.
Have fun!
— Chesko