README for the movix-build virtual machine

Balzs Brny <balazs@tud.at> 2003-10-23

What's this?
============
This is a compile/build system for the MoviX project
<http://movix.sourceforge.net>. It can be used by MoviX developers to create
"system" parts of the MoviX (and eMoviX and MoviX2) distribution.

The system consists of the following things:
- movix-build.img: This is a Debian GNU/Linux filesystem image of the
  virtual machine used for compiling the software.
- movix-build.sh: A shell script for calling User Mode Linux with the image.
- the "work" directory which contains scripts and information for movix-build
- (optional) target.tar.bz2: The resulting compiled "target" directory. As
  fully recompiling everything takes more than 3 hours on an Athlon XP 2000,
  and this is seldom necessary, target.tar.bz2 can be downloaded and
  unpacked in the work directory. After that, subsystems (like MPlayer) can
  be recompiled independently.

Why?
----
It is hard and boring to manually reconfigure and recompile all the parts
needed for a MoviX system. Also, different Linux distributions use different
compilers and system libraries, so sometimes programs compiled on one system
can't be used on another. This makes it hard for developers to try adding
features to the system.
With this build system, each developer has exactly the same build
environment. Scripts have been written to automate routine tasks like
recompiling the kernel and building a CD image for distribution.

What you need
=============
Currently, you need a Linux system for running User Mode Linux
<http://user-mode-linux.sourceforge.net/>. I will determine if CD-based
"live" systems like Knoppix can be used on computers that don't have Linux
installed and post that information.
Of course, you need User Mode Linux, too. It should be included in most
current distributions. On Debian GNU/Linux, you can install it and some
related tools with the following command: 

apt-get install user-mode-linux uml-utilities

User Mode Linux "emulates" (technically, it's not really emulation) a
virtual Linux machine. Because of this abstraction, programs running on the
system are noticably slower than on the host machine. 
You need a fast system or lots of time (or both). On my Athlon XP 2000 with
512 MB of RAM, recompiling everything takes over 3 hours. (But this is
seldom necessary.)
Also, source code and compiling it take a lot of additional hard disk space.
The completely unpacked system with everything (Linux kernel, MPlayer etc.)
uses about 500 MB on my machine after compilation.

You need to download the following files from http://tud.at/bb/download/movix-build/ :
 - movix-build.tar.bz2
 	This is the base system. You only need to download it from time to time
	(this will be announced).
 - system-changes.tar.bz2
 	Updates to the base system. This will be changed more often. It contains
	the build scripts and configuration files.
	The build scripts are also in the MoviX CVS, so in future they probably
	won't be in system-changes.tar.bz2 anymore.
 - target.tar.bz2
 	Compiled software and some sample release files. You can download this if
	you don't want to recompile large parts of the system.

How it works
============
The virtual machine is started using the movix-build.sh script.
It boots a fairly standard Debian GNU/Linux unstable system. (Note: unstable is
Debian's terminology for "the newest released software which wasn't tested yet
by Debian". The virtual machine has never crashed for me after (literally) days
of compiling.)
The virtual machine mounts a directory on the host computer as
"/home/movix/work". (This is the name in the virtual machine; on the host
computer, the directory is called "movix-work" in the directory where you
put movix-build.img and movix-build.sh.) This means that you can easily
exchange data with the virtual machine; you are just copying files around on
your local hard disk.
The compiled images are also saved in this directory on the host system.
The virtual machine image is not meant to be modified by "end users" (other
developers). As it is a binary file, exchanging changes is much harder than
in source code with CVS. So for now, if you need to change things in the
virtual machine, send those changes to me so I can update the original
version. There are probably better ways to do this. 

After starting the virtual machine (as a normal user!), you can login as "root"
or as "movix", both have the password "movix". (If you are afraid that this
could expose your system to crackers, you should be using a firewall on your
host machine.) Normally, you can do everything as "movix"; the build script
uses sudo to do some things that only root can do. This reduces the possibility
of dangerous errors greatly.

When logged in, you can call scripts like "download-sources",
"unpack-sources", "compile-emovix" and "build-emovix". They do all the work;
you can modify whatever you want and e.g. call compile- or build-emovix
until you are happy with the results. You can change their behaviour using
command line switches. After "build-emovix", you get files that you can use
with the rest of the eMoviX source.

Installation
============
Put the downloaded and unpacked files (movix-build.img, movix-build.sh,
movix-work/) into a directory of your choice. It is necessary that this
partition has at least about 500 MB free space (*after* unpacking
movix-build.img) if you intend to recompile everything. You can do with less
if you manually download and unpack just the software you need.
It is very probable that compiling and building images will only work with
filesystems that support Unix filesystem semantics, i.e. user IDs and access
rights. This rules out FAT and NTFS filesystems. (Probably a way for users
with some CD-based live distribution like Knoppix and only FAT partitions can
be found later.)
If you also downloaded target.tar.bz2, uncompress target.tar.bz2 there so its
contents are seen by the virtual machine as /home/movix/work/target. 
Read this file to the end, and then call movix-build.sh from the same
directory, e.g. "sh movix-build.sh" or "./movix-build.sh". Please do this as a
normal user: it is not necessary to run User Mode Linux as root.
Your movix-build virtual machine should soon open an X-terminal and give you
a login prompt where you can login as movix (or root if necessary) with the
password "movix". (By editing movix-build.sh, you can even configure more
than one xterm.)

Network configuration for the virtual machine
---------------------------------------------
This is optional; if you don't want to use "download-sources", you can
download all the programs listed in work/scripts/download-sources yourself and
put them into your work directory (/home/movix/work/source in the virtual
machine, <movix-build-dir>/movix-work/source on the host system) under the
same filenames as in "download-sources". That means, for example, that
linux-2.4.21.tar.bz2 becomes linux.tar.bz2.

If you want to use networking, you only need to change the file
movix-work/config/uml-network-config. It is a standard Debian "interfaces"
file, you can read the "interfaces" manpage to learn more about it.
(Online version of the manpage:
http://www.fifi.org/cgi-bin/man2html/usr/share/man/man5/interfaces.5.gz )

If you want to change the IP address and gateway of the UML virtual machine,
edit the file and put the right addresses on the "address" and "gateway"
lines. (Just choose an unused IP address in your subnet.)
For dhcp, you can use "iface eth0 inet dhcp" instead of the four original eth0
lines. (But I haven't tested this.)
You also need the "tun.o" kernel module on the host side (the real computer)
for the virtual network interface (or it could be compiled into your kernel).

Read the networking HOWTO on the User Mode Linux site
<http://user-mode-linux.sourceforge.net/networking.html> for additional
details.

User Mode Linux runs happily as a normal user on Linux, so you don't need to
run it as root. (It is better not to run as root for security reasons.)
However, for setting up a virtual network interface on the host, you need to
run the uml_net utility as root. On Debian, it is set up setuid root for the
uml-net group, so add your username to that group ("adduser you uml-net").
Check the documentation of your distribution on how you can do this, and read
the UML networking HOWTO.

Again, you can skip all this if you download all the source files manually.

Also, you can edit the download-sources script, replace /home/movix with your
desired directory and then run it natively on your real computer as long as
GNU wget is installed.

The download-sources script
===========================
You should edit the work/config/uml-download-config file if you want to use
download-sources. mcedit and vim are installed in the virtual machine, mcedit
should be usable even for Linux newbies. 
Change the proxy configuration, the kernel_prefix variable, and possibly the
limitrate variable. They are documented in the config file. 

After that, you can call download-sources as the movix user. 

download-sources -h gives you some command line switches that you can use to
download optional software.

The script uses "wget" to download everything. If a file with the same name
and size as offered by the server is already there, it won't be downloaded
again. So you can freely call this script again; it will complete downloads
that were aborted and won't download huge files again if they are already
downloaded. 
The downloaded files are named without a version number. This makes it
easier for the other scripts to unpack the software; however, when the
download-sources script is updated with newer software versions, you will
need to manually remove the old files so they don't end up mixed up with
parts of the new version.
I will perhaps add a "-c" option to the script so it cleans up the
downloaded files before downloading them.

The unpack-sources script
=========================
This script unpacks the downloaded sources. For example, linux.tar.bz2 will
be unpacked to a linux/ subdirectory, again without version number. (This
means that before unpacking the sources, you should remove those directories
you unpacked before. But unpacking is, just like downloading, usually done
only once for one movix-build release. Use the -d command line switch so
unpack-sources does this automatically for you.)
Of course, you can unpack updated sources yourself manually, just remember to
rename the <software>-<version.number> directory to just <software>.

The compile-emovix script
=========================
This script is used for automatically compiling the required software. When
called without parameters, it compiles all the required software packages,
that means the Linux kernel, ALSA, LIRC, em8300 (DXR3), busybox, aalib, SDL,
tv out utilities and finally MPlayer. If you want to also recompile perl or
devfsd for some reason, you can call the script with the command line
switches given by "compile-emovix -h". 

You can modify the behaviour of the script using the command line switches.
For example, if you just want to recompile MPlayer without SDL and ascii-art
support and optimized for small size (instead of speed), you can use the
command "compile-emovix -m -z -v aa,sdl". Or you can recompile everything
but the Linux kernel and ALSA (those take lots of time) with 
"compile-emovix -K -A". 

If you want to recompile only one subsystem, e.g. SDL, you usually don't
need to download and unpack all the other packages (e.g. the kernel), you
just need target.tar.bz2. However, packages that use the kernel (ALSA, LIRC,
em8300, mplayer (because of the mga_vid kernel module)) really need the
unpacked kernel tree even if you don't want to recompile it.

The software packages are sometimes not easy to configure and compile
correctly. But don't worry, this is what you don't need to do yourself
because the script does it for you. If you want to change the way something
is compiled (e.g. add or remove new options) and the switches of
compile-emovix aren't enough for you, you can look into the directory 
/home/movix/config in the virtual machine. There are some configuration
files that are used for configuring software before compilation. You can
edit those files here. Some other packages are directly configured in the
compile-emovix script, so you can also edit that one. (In this case, please
send me the changes so I can integrate them into the official distribution.)

The following software packages are used for the eMoviX distribution:
- The Linux kernel
- The ALSA sound card drivers, libraries and utilities. (Depends on the
  kernel)
- Busybox. This is a "multi-call binary" that provides most of the simple
  Unix programs that are used on eMoviX.
- aalib. This is a software library for "ASCII art" output, i.e. converting
  videos into full-screen text and displaying them so. This is considered
  ber-cool by some people but frightens others. You can exclude aalib from
  the compilation and build mplayer without aa support, saving some space in
  the initrd and therefore memory.
- sdl. The Simple DirectMedia Layer is a cross-platform multimedia library.
  It can be used by MPlayer on some computers where other ways of displaying
  video are suboptimal. It can be left out of MPlayer to save space and
  memory.
- lirc. LIRC means Linux Infrared Remote Control; you can use some remote
  controllers to use your MoviX box without a keyboard. LIRC is optional.
  (Depends on the kernel)
- em8300. This is used for DXR3 (?) cards for displaying videos on the TV,
  AFAIK. (Depends on the kernel)
- tv out utilities like nvtv, s3switch, atitvout and matroxset. They are
  used to set up the TV output of your graphic card if it is supported.
- mplayer. This is one of the most successful Free media players. It is
  extremely flexible, this is why it is used in MoviX. For distribution use,
  it should be compiled with an unpacked kernel directory because of the
  mga_vid kernel module. For your own use, you can do this without the
  kernel if you don't have a computer with a Matrox graphics card.
- perl. MoviX uses perl but you usually don't need to recompile it; a fairly
  small perl binary is included in the movix-build filesystem image. 
- devfsd. (DEVice FileSystem Daemon) The Debian version is good enough so it
  is simply copied to the MoviX image without recompiling.

The compile-movix script
========================
This is very similar to compile-emovix. Because small size is not that
important for MoviX as it is for eMoviX, MPlayer is compiled with more
features.
Also, additional volume mixer utilites will be compiled.

The build-emovix script
=======================
This script is used after everything was compiled to automatically build an
initrd ("initialization ramdisk") image that can be put onto the eMoviX CD.

The image and other files will be put into the work directory, in
release/emovix/. 

It roughly does the following:
- removes and re-creates the "initrd" directory where everything will be
  copied to
- copies the system files to the initrd
- copies the Linux kernel modules to the initrd and compresses them
- copies the compiled files to the initrd
- creates some symbolic links in the initrd directory for software that
  expects some files in other locations
- checks the MPlayer binary for libraries that aren't used. Those libraries
  are removed from the initrd.
- optionally, removes some software modules that were requested not to be
  included by the user
- optionally, it compresses some executable files with a program called UPX so
  they need less room in the filesystem, and therefore in the RAM when eMoviX
  is booted
- it calculates the hard disk space used in the initrd, creates a Linux ext2
  filesystem image with about the same size (a bit more), copies everything
  to that, and updates things like file owners and permissions.
- finally, it prepares a release with files that can be written to the CD.
  The isolinux.cfg file which sets up the ISOLINUX boot loader is adjusted
  for the created image

Command line switches:
  -h      help

  -e      don't include em8300/DXR programs/modules
  -l      don't include LIRC programs/modules
  -s      don't include SCSI kernel modules
Those options remove features from the initrd without the need to recompile
anything. This is good for saving space on the CD and in the RAM, but
removing SCSI modules means that eMoviX won't run on computers with only SCSI
CD-ROM drives (fortunately, there are not much of those today).

  -u      use the UPX packer to compress a few programs
  -U      use the UPX packer to compress more programs
The UPX packer compresses executable files to about a third of their size.
When they need to be called, they are uncompressed on-the-fly to the /tmp
directory and run from there. This means that they need *more* hard disk
space (in the eMoviX case, memory) *while* they are running but less while
they aren't running.
With -u, only programs that are called for a short time get compressed. This
decreases the initrd size by a small amount and doesn't have adverse effects
on the memory usage.
With -U, even programs that run while a movie file is played will be
compressed. This saves more space but probably needs more RAM. Probably it
won't work on systems with 32 MB of RAM.

Note that it doesn't really matter for the size of the compressed initrd.gz
image that goes onto the CD-ROM if the programs are UPX compressed or not. If
they aren't, gzip will compress them to about the same size that they had if
they were compressed with UPX. Only the size of the uncompressed initrd
filesystem changes. This is relevant for the memory usage of the booted
*MoviX* system; however, as described, it's not sure that this actually
reduces the memory consumption during playback.

So -u is a safe choice, -U might not be.

  -m      before compressing the initrd, start mc for last minute changes
This is nice if you want to take a look at the final contents of the initrd
directory before it is copied to the image. You could make some last-minute
changes, remove or add files etc.
  
  -r      release name (subdirectory where release files are put)
If you specify a release name e.g. with "-r MY-SUPER-RELEASE", the generated
files will be copied to a subdirectory with this name. This helps you
organize your releases. 

The build-movix script
======================
This is very similar to build-emovix, but more software will be copied to the
initrd.
