Unix file packing cheat-sheet

Always can’t remember exactly how to pack some files into that damn tar.gz or something? With this cheat sheet, you’ll never have to google it again!

Stick this on the wall to always have a cheat-sheet on how to pack/list/unpack gz, bz2, tar.gz, tar.bz2, tgz, tbz2 or zip file under Unix (Linux/MacOS/whatever).
Continue reading “Unix file packing cheat-sheet”

How to start with transcoding aka fairly simple long general transcoding guide

So, again an transcoding (encoding – similiar terms) guide, this time general and in English. This one should serve as an equivalent for the Czech one i wrote some time ago (although much newer and up-to-date).

Note that this guide is for Windows (XP and later), since you efectively can’t transcode on Linux and other systems (in other words – Avisynth is for Win).

Introduction


x264 encoder logo

So, again a transcoding (encoding – similiar terms) guide, this time general and in English. This one should serve as an equivalent for the Czech one i wrote some time ago (although much newer and up-to-date).

Note that this guide is for Windows (XP and later), since you efectively can’t transcode on Linux and other systems (in other words – Avisynth is for Win).
Also note that most of this guide focusses on video transcoding. Audio usually can be demuxed from source and muxed to output pretty easily.

I should stress out that I focus on transcoding of anime (Japanese animated movies/series) and so does this guide. There are slight differencies in settings required to meet satisfying quality with live-action stuff. However, the basic principles described here apply for any encoding.

Images used here are only ilustrational and should give general idea how the things should look like. For code examples refer to codes I have in the text.

Tools Required

x264.exe – encoder (choosing versions: choose 32bit version since Avisynth is 32bit; 8bit/10bit version according to how wide compatibility you need – these days most PCs will play 10bit encoding, but some devices might not)
Avisynth – frameserver
notepad or other simple txt editor (not MS Word or something like that!)
VirtualDub – for checking the script before actual transcoding

If you want to transcode DVDs (already decrypted), then additionally these:
DGIndex – DVD indexing tool

About the Tools

Many people come to me and ask what do I use for transcoding – MeGUI? ZarxGUI? Something else? I don’t use any GUI. All you need for transcoding is Avisynth, x264.exe encoder and batch file to supply parametrs to x264.exe. Encoding then runs in command line. Nothing with GUI is needed (well, except some simple text editor and Windows OS).


notepad.exe

Avisynth is so called frameserving SW. It takes frames from video, processes them and supplies them to encoder. It doesn’t have any GUI, so don’t expect any windows after instalation. It runs on background and you’ll never see it. You control it thru scripts, video processing is handled by plugins (filters) in form of .dll libraries.

x264.exe is encoder for h.264 codec, which is probably the most advanced codec you can get your hands on. It’s for free (basically; read licence if you want to make money with it or something). x264.exe has extensive posibilities of control via parameters but defaults are sane and good, so you don’t need to change anything. Discuss. (I’m not really good with x264.exe parametres.)

You will need text editor for creating Avisynth (AVS) scripts and batch files.

DGIndex creates index file of frames in DVD which allows Avisynth to read the vob (MPEG-2) files. You can use it for selecting certain parts of DVD (e.g. one episod of series), so you don’t have to encode it whole. It also outputs audio of selected part.

Encoding Process

Step 1: Creating AVS script

AVS script has to contain loading used filters, loading source video file and commands how to process the video. Basic filters don’t have to be manually loaded so some minimalistic script can look like this:

avisource("d:\encoding\source_video.avi")

Avisource is name of plugin which loads avi videos. For more universal use, directshowsource() and FFVideoSource() are available. FFVideoSource needs to be loaded manually:

loadplugin("d:\encoding\AVS_filters\ffms2-r430\ffms2.dll")
FFVideosource("d:\encoding\source_video.avi")


AVSPmod editor

If you put all your plugins to plugin folder of Avisynth, then you don’t have to load them manualy. I don’t do it for certain reasons, you can.

MP4/MKV

Now, let’s say you have full HD raw anime video in mp4 container (can be mkv too, doesn’t make difference) with h.264 codec that has 900 MB filesize, and you need to make it smaller and 720p (720p TV rip should be below 350 MB per 24 minutes) and you need to hardsub SubstationAlpha subtitles (ASS) to it (so they’re “burned” in images). There is also some black border you want to crop out from the video. The script would look like this:

loadplugin("d:\encoding\AVS_filters\vsfilter.dll")
directshowsource("d:\encoding\Anime_ep01[1080p].mp4", audio=false)
Crop(0, 4, -0, -4)
LanczosResize(1280,720)
textsub("d:\encoding\sub\Anime_ep01_hardsub.ass")

AVSPmod editor

This script loads VSfilter, which provides certain other filters (textsub). Then it loads video (without audio) using directshowsource. Then it crops 4 pixels from it on the right and left. Then it resizes the video using Laczsos resize method (for other method see this page). Then it hardsubs the subtitles into video. The processed frames then go to encoder.

You can create such script now and drag&drop it to VirtualDub to test it.

DVD

Now another example. Let’s you have a DVD (in other words bunch of .vob files) and you want to transcode them into mkv with h.264 codec.

First you create .d2v file (index) in DGIndex. If you don’t know how to do this, refer to this guide.

Now, be prepared that when you transcode DVDs you will need to do a lot of filtering. DVDs are usually the worst crap you can find considering the quality of image (except streams). Depending on region the DVD is from, it’ll have either 50 fieldes/second (25 full frames/second) or 60 fieldes/second (30 full frames/second). You will have to put these fieldes togeteher to create full frames. This brings need of deinterlacing and other stuff, it’s beyond frame of this guide to go into it more thoroughly. Your script basically needs to load d2v file, get rid of fieldes, deinterlace and clean image. Compare the cleaned (upper) and raw (lower) screens of DVD R1:


DVD image before and after processing

The script might look like this:

#in the 1st part of script we load plugins:
loadplugin("d:\encoding\AVS_filters\vsfilter.dll")
loadplugin("d:\encoding\AVS_filters\DGDecode.dll")
loadplugin("d:\encoding\AVS_filters\TDeint.dll")
loadplugin("d:\encoding\AVS_filters\ColorMatrix.dll")
loadplugin("d:\encoding\AVS_filters\TBilateral.dll")
loadplugin("d:\encoding\AVS_filters\TweakColor.dll")
loadplugin("d:\encoding\AVS_filters\MSharpen.dll")
loadplugin("d:\encoding\AVS_filters\TIVTC.dll")
loadplugin("d:\encoding\AVS_filters\RemoveGrain\RemoveGrain.dll")
loadplugin("d:\encoding\AVS_filters\WarpSharp.dll")
loadplugin("d:\encoding\AVS_filters\MaskTools-v1.5.8\MaskTools-v1.5.8\MaskTools.dll")
loadplugin("d:\encoding\AVS_filters\masktools-v2.0a48\mt_masktools-26.dll")
loadplugin("d:\encoding\AVS_filters\temporalcleaner_25_dll\TemporalCleaner.dll")
loadplugin("d:\encoding\AVS_filters\degrainmedian_20061008\degrainmedian.dll")
loadplugin("d:\encoding\AVS_filters\FluxSmooth-1.1a\FluxSmooth.dll")

#and in second part we process the image:
mpeg2source("d:\encoding\SomeDVD\SomeDVD_part01.d2v", info=3) tfm(d2v="d:\encoding\SomeDVD\SomeDVD_part01.d2v") ColorMatrix(hints=false, interlaced=true, threads=0) tdecimate(mode=1,hybrid=1) TDeint(order=1) DeGrainMedian(limitY=5,limitUV=5,mode=1) fluxsmoothST(4,4) MSharpen(strength=50) tweakcolor(sat=1.1, bright=-6, cont=1.4) TBilateral(diameterL=7, diameterC=3, sDevL=1.7, sDevC=1.7, iDevL=1.6, iDevC=5.4, csL=1.0, csC=1.3, d2=false, chroma=true, kernS=1, kernI=8, resType=0, gui=false) textsub("m:\Other\encoding\logoACX_v1.ass") #if you want to hardsub some logo in ASS subs or something

Well… it’s bloody long, isn’t it? If you want to understand it, study following:

Mpeg2Source
TFM, TDecimate
Colormatrix
DegrainMedian
fluxsmoothST – generally about smoothers
MSharpen
tweakcolor
TBilateral – yeah, awsome filter… you can spend days on settings like I did and you’ll never discover everything… of course there are maybe better smoothers, but I like this one.

Many filters and references for them can be found at Avisynth wiki.

Here you can download most of the filters I used in this script.

Bluray

Encoding BDs is covered in my other guide here.

So that’s it for creating AVS script. That was the hardest part (although it’s actually very simple if you don’t start by encoding DVDs). We can now move on to the actual encoding.

Step 2: Encoding with x264.exe

As I said, x264 has CLI (command line) interface, so any settings are passed to it by parameters. Syntax is like this:

x264.exe [options] --output outfile infile

To make it simplier, you can use batch file. To create one, open up notepad.exe and copy this:

x264.exe --crf 17 --preset slow --tune animation --output encoded_video.mp4 AVS_script.avs
pause

The parameters above mean folowing:

–crf – constant quality mode (variable bitrate), lower number means higher quality, sane range of values is 15-25. Higher the resolution is, higher values you can use. There is also –bitrate option sets constant bitrate (in kB/s) – do not use this unless you REALLY need predictable filesize!
–preset – controls speed/compression ratio – slower speed means better compression and smaller output file.
–tune – presetted tunning for film, animation, grain, stillimage, psnr, ssim, fastdecode, zerolatency. If you don’t really fit into any of them, don’t use it at all.
encoded_video.mp4 is output file, AVS_script.avs is input file.
pause – that is command for windows CMD to wait for a keystroke.

Now you have this line in your text editor, save the file as encode.bat to the same folder where you have the x264.exe and AVS script. Then run it (as executable) and see what happens.

If everything went how it should, you will see this:


x264.exe output after successfull process

If not, you might encounter screens like these.


x264.exe output after encoding failure

Now, what might have gone wrong:

error in AVS script – you can be pretty sure that’s it. Open the VirtualDub, load the AVS file and see what it tells you. If you don’t understand it, google the error, it’ll probably help you. Most common errors are that plugin cannot be found (bad path), filter does not exist (plugin not loaded) and bad syntax.
input files are not where they’re supposed to be – before you start freaking out check the paths
Avisynth is given something it can’t read – aren’t you loading some .vob file or something which can’t be recognized by Avisynth?
movax – for everything what goes wrong blame movax.

When you fix all the errors in your script, run the encode and wait long enough, it will finish and print out some statistics. When you see this, it means the encode finished:


x264.exe output on encode finish
Step 3: Audio

If you transcode whole files (you don’t cut off some parts), it’s very easy with audio. Demux the audio from the original file and mux it with the transcoded video. If you didn’t mess up with frame rates, everything should be OK. In case you were encoding DVD and you decided that you want film framerate (24000/1001 fps), then you might encounter problems with audio/video sync. However, this is again above the frame of this even now awfully long guide.

Step 4: Subtitles

If you have DVD and want to get text subtitles, then use SubRip for ripping subtitles from DVD and converting them with OCR to text subs. As for how to work with SubRip, see this guide (scroll down to the SubRip guide).

Then you can style the subtitles in Aegisub and mux them with video and audio.

Some advices

It might not look like it, but there is much to mess when encoding.

Constant bitrate

Do not use constant bitrate! It’s total bullshittery. There is no sane reason to require predictable file size and thus constant bitrate. (No, nobody is burning videos to CDs anymore and nobody cares if the size is 233 MB or 258.4 MB.) With constant quality you can achieve better quality and smaller files.


1: source image, 2: overcompressed image (view in full resolution)
Overcompressed video

Don’t be faggot and check if you have enough high –crf value set for your video. Each video requires something slightly different, but generally you won’t mess up using –crf 17 for 480p and smaller video resolution, then raise it by one for each cca 200 px of height of the video (very general rule and might not apply in some cases).

Undercompressed video

As I wrote in my BD guide, don’t be like THORA and CoalGirls and such fansub groups, which have 2 hours 1080p movie in 12 GB. For 2 hours anime movie 5 GB is far enough even with grain scenes in half of it. (Live action movies require slightly lower –crf values and thus bigger files, however 8 GB should be far enough too.) As for such movie in 720p, 3 GB should be enough. Episode of 24 mins should exceed 600 MB and 400 MB respectively. (Yes, I am being slightly inconsistent with the BD guide – the numbers are very approximate.)

Fluency

The resulting video should play fluently without any freezing. There are many causes of freezy video and I don’t even remember what I have encountered already. It is caused usually by skipping every few frames one frame or vice versa (duplicating frames).

A/V sync

Never produce file which has desynchronized audio and video tracks (and if you do subtitles, then they have to be in perfect sync too). This often happens when you transcode from one framerate (e.g. NTSC DVD, 30000/1001 fps) to another (e.g. film, 24000/1001 fps). You have to be careful when doing this.

Here is an awesome example of video out of sync.

Hardsubbing

Do not hardsub what doesn’t need to be hardsubbed. The only things you should hardsub is karaoke and maybe some small inobtrusive logo showing for few seconds that you have encoded it. Hardsubbing eats up bitrate and it usually sux since viewer can’t turn it off.


BD upscaled from DVD
Videosize

Never upscale! But downscale sometimes – many BDs are DVD upscales (see image), so you can sharpen it and downscale it (in this order!) to 720p. Try not to do this if you feel you’d loose some details. Hint: very few anime have 1080p resolution, so you can downscale to 720p quite safely.

Don’t overfilter it

Don’t overuse filters. Encoder’s task is to convert video to pretty watchable PC format, not to create something new. Keep this in mind especially when cleaning DVD image. I myself used to tend to make image superclean and I usually found afterwords that I had created image so clean it was almost detailless. Try to avoid that.

Thanks to steelista for this advice.

Other

Terminating encoding process – you can do so in the commandline window by pressing ctrl-C. There is no way how to pause it, all you can do is select text in the window (Window Menu -> Edit -> Mark), this’ll pause the process. Unpause it by pressing escape key.

ConvertToRGB32 – when you open a script in VirtualDub, you might encounter green or blue video screen instead of the picture. Try putting ConvertToRGB32() line at the end of your script to avoid this. You have to remove/comment this before encode.

Speed – be prepared that the encode might take hours. Depends on your CPU speed, of course. Also note, that Avisynth is slow 32bit bitch, so if you have i7 CPU or something as awesome as that, you’ll have to run a few encodes (I run usually 4 on i7) at the same time to load it fully. These may share encoder (x264.exe file) and filters but cannot share the AVS file. Also pay attention to cooling.

I’ll maybe add stuff here.

Useful Tools

Except tools mentioned above some of following might come in handy.

AVSPmod – text editor, supports AVS syntax and includes some tools like resize calculator.
MP4Box GUI – mux/demux MP4 files
MKV Toolnix – collection of tools for dealing with MKV (mainly muxing)
MKV Extract – MKV demuxer
MKV Cleaver – MKV demuxer with better batch support

Useful Websites

There is lot of pages about transcoding out there and believe me, you will sooner or later read tons of text on them. There are various guides scattered all over the Internet, but I picked the most helpful/important of them (in my opinion):

Avisynth wiki – the only decently complex page about Avisynth I know is this wiki; note the menu below the header – that’s the important one
mewiki – contains almost everything about x264 encoder; I go there for settings description if I need some of them
Videohelp – actually… why am I writing this? Everything is there…

Epilogue

Not much to say here, but I always conclude! Well, something about me – how I learned transcoding. One day I told a friend on IRC that I wanna know how to transcode. He sent me four lines of code. Some 3 lines of AVS script and 1 line of batch file. And that’s how I started. Thanks, steelista. Then I learned about filters, for I needed to transcode one awful DVD. The encodes were awful too, by the way… This was 3 years ago. Encoding is long run, I don’t consider myself as even good, not to say awesome guru lol (that doesn’t stop me from writing these guides, though). So, have fun.

Changelog:

v1.05
v1.05: advice from steelista added
v1.04: added absolute path for plugins to make clear that they’re not loaded from Avisynth default plugins directory

v1.03: some minor changes in introduction

MS Word 2007: Enter comment, enter note under line, etc. grayed out

Have ever happened to you that you were sent a document (.doc) from somebody using OpenOffice and you couldn’t add any comments and notes and stuff? Well, it happened to me some while ago and it took me a while to get it solved. It looks like some complex problem but it’s actually very stupid. At least, I felt stupid when I found out what is it about.

For some reason, OpenOffice when saving to .doc format they tend to lock the document. When you open it in MS Word, it results in having all the options on Revision tab grayed out. It’s all about one click to unlock the .doc, but that button is under another Lock document button, which is kinda confusing.

So, how you get there. You go to Revisions, then you click Lock the Document button and a sidebar pops up. It should say something like “Locked to protect the document from unwanted changes”. Look to the bottom of sidebar, there’s a small button Unlock Document. Hit it, and there you go.

If you have another experiences with this or something, just let me know in comments. BTW, I have only Czech version of MS Word, so the labels on buttons might be a bit imprecise. Take a look at pics to be sure.

Bluray encoding

It’s now about 8 months since I started my “encoder career” in a group ripping anime DVDs known as Animecorpx (ACX). I work for them as an encoder and what we do is basically ripping anime DVDs released in the USA to make them accessable for the rest of the world, where you can’t buy them. But I actually hate DVDs.

A word of introduction

It’s now about 8 months since I started my “encoder career” in a group ripping anime DVDs known as Animecorpx (ACX). I work for them as an encoder and what we do is basically ripping anime DVDs released in the USA to make them accessible for the rest of the world, where you can’t buy them. But I actually hate DVDs. They have low quality video, cleaning their image up is pain and they generally suck. It came out I’m not the only one in ACX who hates low res and that shitty image of DVDs. It was shitsu, my Swiss colleague, who actually made me think about an option, that I could encode BDs and that it might not be as impossible as I thought it is. As I started some research on this topic, I found out, that it’s not difficult at all. However, it requires somewhat different approach and different tools, so it took me a while to get into it and I relied on other guides and blog posts scatter on multiple pages. That’s why I decided I should write something simple to follow about encoding BDs.

Note that this text assumes some level of knowledge about encoding.

Getting BD

I’ll leave getting sources up to you, and I also won’t cover how to copy files from BD disk to HDD, since I’ve never done it. What you should get for your encoding is directory with subdirectories such as Auxdata, Clipinf, Meta, Playlist and STREAM (STREAM is the one you really need, equivalent to VIDEO on DVDs). Note that BD sources, depending on disk, have from 25 to 50 GB, so what you really need (even more than powerful CPU) is disk space.

Video

Loading to Avisynth

First, let’s look on video encoding. It’s pretty easy, you will usually find one .m2ts file per episode (or movie), so you can forget about DGIndex and creating some files as d2v. Look to STREAM dir, you’ll find loads of .m2ts files. As in DVDs, look for those big ones, size in gigabytes or tens of GB (movies have on main file e.g 40 GB, episodic anime usually 4 to 6 GB per episode, one file per episode). Good thing is, you can load them to avisynth directly, so directshowsource will do:

directshowsource("c:\path\STREAM\00032.m2ts", fps=23.9759856527702, audio=false)

Having set fps is probably not crucial, but I keep it there for safety purposes as well as audio=false, but you won’t get any from it anyway.

Resize

Now frame (image) size. If you rip a live-action movie, you’ll probably want to keep full HD, but as for anime, note that almost no anime has resolution of fHD and is either upscaled directly or, in better case, drawn with “thicker lines”. Means that you can resize it to 720p without any loss of details. You will probably want to sharpen it before resize too (never do it afterwards!), so it looks better. Add line with sharpener and resizer like this:

MSharpen(strength=90) #or whatever shapener you use; don't use Warpsharp.
LanczosResize(1280, 720)

Be careful about ratio, if you are cropping image (do that before resize), you’ll have to worry about ratio divergence after cropping and you may get to pretty ridiculous dimensions, but try to keep the divergence below 1% (or you will break Steins Gate!). For these purposes, you can use AVSPmod (links @ the end of article).

And that’s pretty much it, at least for avisynth!

Encoding

Now you will load your AVS script to encoder. Now, we’re in modern times, so do not use constant bitrate! Forget about constant sizes, that doesn’t matter anymore, and use that –crf option in x264 encoder. That’ll keep the quality constant ad you won’t have problems with insufficient bitrate in grainy or other bitrate greedy parts. 2-pass encoding might do, but still I don’t see any reason why to use it. Values of CRF might differ on what you’re encoding, anime in 720p should be in 17-19 range if you don’t want compress artifacts. Generally applies a rule that higher resolution you have, higher CRF value you may use to maintain the quality. For fHD it’s in 19-21 range, even more sometimes. So you’re encoder line (in CLI interface) will be like this (for anime):

x264.exe --profile high --crf 18.5 --preset medium --tune animation --output ..\encoded\video.mp4 skript.avs

Be prepared that encoding a movie might take like 10 or more hours; I on my dual core Intel Centrino2 laptop encode a movie for 8 to 16 hours (depends on resize, sharpen, some other cleaning, –preset option used). Be sure to have you computer well cooled. :P

Some ethics

Have you ever seen a movie BD rip from THORAnime fansub group? Well, their movies have like 12GB (sometimes less, i know). That’s about at least 3 times more than a fHD BD rip in h264 codec with two audios each 300MB should have. Always try to keep your movies (their finals size with audio and everything) under 4 GB, so it can fit DVD. (Not that many people burn them on DVD, but it’s a good rule to stick to.) Full HD episode shouldn’t exceed 500 MB or so. As for 720p, try to stick with 3 GB resp. 350 MB limit, if possible. (Not applicable to live action stuff, although 8 GB should be enough for anything up to 2.5 hours of playtime.)

Audio

Up to here, it was pretty much the same as any other encoding. Now, for extracting of streams from BD you will need a tool called HdBrStreamExtractor (download below).

Work with HdBrStreamExtractor

It’s pretty simple – load a path to you STREAM folder to it (Input) and path where you want it to spill out the extracted streams (Output) and click “Features”. It’ll show you the files in folder with their lengths, so you can easily recognize, which file is the one you need. Clicking on it will result to displaying streams in the file in the list below, where you can choose which streams and in which format you want to extract (see pic below). Note that chapters are also a stream and eventual subtitles need to be extracted this way too. Audio formats are a bit above this article’s frame, but usually you can use AC3 or THD and then encode it to AAC/AC3. Always try to take streams with highest possible channel count, or extract both (e.g. 5.1 and 2.0) if possible.

Extraction may take even half an hour, so be patient, it’s ten times bigger than DVD. As for audio transcoding, try using MeGUI or FooBar2000. Then proved useful to me.

Subtitles

Now you have your subtitle stream extracted into some strange .sup file, what to do with it? Remember SubRip? Well, there’s SupRip for BDs (download below). It’s in some ways smarter than SubRip, in some ways more stupid, but you’ll get over it.

Load the .sub file and just write in the letters for each letter in the image (yea, subs are stored in BDs same way as in DVDs). Good thing is that you can control the output for the whole line, bad thing is you can’t edit the matrix as you could in SubRip and you don’t have such good OCR auto-correct engine there either. (Actually, I still use this feature of SubRip for subtitles outputted from SupRip.)

Well, and that’s it. Edit subtitles’ style, mux toghether and you’re done. Not that hard, isn’t it?

Problems

As with anything, you might encounter some problems. Since I haven’t encountered any yet, I can’t write here anything, sorry. I will try adding here (or to a new post) issues that I or my ACX colleges will stumble on. Also, please, if you encounter any issues, try sharing you experiences here in comments! It might help other. If you won’t be able to find solution, you can also use comments to ask for help.

Tools

HdBrStreamExtractor and SupRip in one pack

BD_tools_suprip+eac.zip – you will find HdBrStreamExtractor in /eac3to317 directory, SupRip is in its own. There are also some other tools in /eac3to317 directory, you may play around with them, I don’t remember anymore what they do.

You can find encoding tools in my earlier guide here, but I haven’t translated that one to English yet, sorry. Download links are at the end as here.

Epilogue

I wanted to write it shorter, but when I write, I write long… Hope it helped you a bit and shall you encounter any issues, share them in comments. Thanks.

Version 1.0., revision 2.

Multilanguage blog posts – Polylang

I recently decided, that I want to write a cetrain guide (which will hopefully appear here soon), and I want it to be in more languages (precisely in Czech and English). So I started some research on this topic, but it was tougher than I expected, because I didn’t really know what to search for. Testing various ways of achieving my goal included updating to WP 3.3 which I feared pretty much cause of my bad experience with one WP update some time ago, but this one turned out all right, though. Then I found a plugin called Polylang.

Continue reading “Multilanguage blog posts – Polylang”