
# Copyright (C) 2006-2008 Junjiro Okajima
# Copyright (C) 2006-2008 Tomas Matejicek, slax.org
#
# LICENSE follows the described ones in lzma and squashfs.

# $Id: sqlzma.txt,v 1.37 2008-03-12 15:16:50 jro Exp $

LZMA patch for squashfs
Junjiro Okajima


o What is this?
This is a patch against lzma449.tar.bz2 and squashfs3.3.tar.gz, and
makes squashfs to support both of LZMA compression and ZLIB
compression.
The new squashfs image file (generated without '-nolzma' option) has no
backward compatibility, while the patched squashfs and its tools can handle
the old squashfs image generated by the unpatched squashfs-tools.

This patch is developed and tested on linux-2.6.24.3.


o Files
- Makefile
  a toplevel makefile which builds everything. it is recommended to use this
  makefile.
- sqlzma.h
  a global header file. this file is used from kernel and userspace utilities.
- sqmagic.h
  defines patched squashfs magic number.
- sqlzma1-XXX.patch
  a patch against lzma package. currently it is lzma449.tar.bz2 and the 'XXX
  in filename is '449'.
- sqlzma2k-YYY.patch
  a patch against squashfs kernel source. currently it is
  squashfs3.3.tar.gz and the 'YYY' in filename is '3.3'.
- sqlzma2u-YYY.patch
  a patch against squashfs utility.


o Download, Extract, Patch and Build
- choose the lzma version you use, and modify ./Makefile.
- mkdir lzmaXXX, get SourceForge/sevenzip/lzmaXXX.tar.bz2, and extract it into
  lzmaXXX dir.
- apply sqlzma1-XXX.patch to lzmaXXX.
- get SourceForge/squashfs/squashfsYYY.tar.gz, extract, and get squashfsYYY/.
- apply squashfsYYY/kernel-patches/linux-2.6.24.3/squashfsYYY-patch to your
  kernel source, and then also apply sqlzma2k-YYY.patch to it.
- apply sqlzma2u-YYY.patch to squashfsYYY.
- make them. if you have lzmaXXX and squashfsYYY in the same dir, a toplevel
  Makefile in this patchset builds everything for you.
  If you want to change their path, edit the variable definitions in
  ./Makefile.
- It builds these things.
	+ ${Squashfs}/kernel-patches/${KVer}/fs/squashfs/
		* squashfs.ko
		  a patched squashfs which supports LZMA uncompression. If you
		  didn't load unlzma.ko and sqlzma.ko before compilation of
		  squashfs.ko, the linker may produce some warnings about the
		  functions in sqlzma.ko. But you can ignore them in safe.
	+ ${Lzma}/C/7zip/Compress/LZMA_C/
		* libunlzma.a
		  LZMA uncompression user library for squashfs utilities.
		* libunlzma_r.a
		  thread-safe one.
	+ ${Lzma}/C/7zip/Compress/LZMA_C/kmod/
		* unlzma.ko
		  generic LZMA uncompression module for linux kernel.
		* sqlzma.ko
		  LZMA uncompression kernel module for squashfs, a bridge
		  library between squashfs and unlzma.ko.
	+ ${Lzma}/C/7zip/Compress/LZMA_Alone/
		* liblzma_r.a
		  thread-safe LZMA compression user library for squashfs
		  utilities.
	+ ${Squashfs}/squashfs-tools/
		* mksquashfs, unsquashfs
		  patched squashfs utilities which support LZMA compression.
- copy the modules squashfs.ko, unlzma.ko and sqlzma.ko to wherever you like.
- load these modules.
- create your filesystem image by mksquashfs.
- mount it and enjoy.


o Added options
- mksquashfs -lzmadic N
  This option sets the dictionary size for LZMA compression.
  The default value is same as the block size, even if you change the
  block size by specifying -b (block size) option for mksquashfs. The
  dictionary size always follows it by default.
  If you want to set a different value from the block size, then use
  this option ALONE or AFTER -b option.
  In append mode, you can't change the block size, but the LZMA
  dictionary size since the compression is done for every and each
  block.
  The syntax is equivalent ot -b option, ie. you can use K
  (kilobyte) or M (megabyte).
- mksquashfs -nolzma
  This option makes mksquashfs not to try LZMA compression. And it uses only
  ZLIB compression, as it was unpatched.
- no options are added to unsquashfs.
  the patched unsquashfs automatically detects whether the given squashfs
  image uses LZMA compression or not.


o LZMA and ZLIB
There are three groups of byte stream in compression.
- both of LZMA and ZLIB cannot compress.
- LZMA gets better compression.
- ZLIB gets better compression.

This patch supports both compression methods. The patched mksquashfs tries
ZLIB compression first. If the data cannot be compressed, store the data
uncompressed as unpatched mksquashfs does. If it succeeds and -nolzma option
is not specified, then keep the compressed data and try LZMA compression.
After LZMA compression, compares the sizes of the two compressed data and
discard the larger one.

Honestly speaking, I do know nothing about the compression theory. This is the
fact I got from using these libraries.
(What is this thing called in English? a rule of thumb?)


o Multi-call decompression and parallel decompression
While squashfs3.1-r2 took the approach of single-call ZLIB decompression,
squashfs3.2 and later takes the approach of multi-call which has an
advantage of memory consumption. So I tried implementing multi-call
LZMA decompression too, but I gave up. Because of memory usage. While
single-call LZMA decompression doesn't require a dictionary,
multi-call does. And its size can be mega-bytes (the dictionary size
which I checked was 8388608).
In the case of single-call LZMA decompression, it requires 64KB buffer at
maximum to store the compressed squashfs block.
Finally I decided to implement the parallel single-call LZMA decompression.
The parallel decompression is also applied to ZLIB decompression. It
also requires more memory than multi-call ZLIB decompression, but less than
multi-call LZMA decompression. And it can take an advantage of speed
performance, especially on SMP machine.


o Backporting
A user, dystryk, had backported this patch (previous version) to
linux-2.6.17. His URL is,
https://sourceforge.net/project/showfiles.php?group_id=116780&package_id=222970


Enjoy your smaller squashfs.
