image_dir and image_name boot codes bug and a fix

[ian_s]

The image_dir and image_name boot codes do not now work in live mode.

May I submit a fix that will restore this funtionality.

It requires adding one line to initrd0.686 /etc/default/distro file adding :-

FLL_DM_IMAGE_FILE="siduction.686"

And changing 2 lines of code in initrd0.686 /scripts/fll file :-

Code: [Select]


--- orig_fll 2012-01-04 11:29:12.000000000 +0000
+++ correction_image_name_image_dir_fll 2012-01-31 19:45:31.000000000 +0000
@@ -981,10 +981,10 @@
  FLL_SNAP_DEV=$(losetup -f)
  losetup ${FLL_SNAP_DEV} ${rootmnt}/fll/snapshot
  fi
- FLL_ROOT_SIZE=$(ls -l "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_IMAGE_FILE}" | awk '{print $5}')
+ FLL_ROOT_SIZE=$(ls -l "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_DM_IMAGE_FILE}" | awk '{print $5}')
  FLL_ROOT_DMSIZE=$((${FLL_ROOT_SIZE}/512))
  FLL_IMAGE_DEV=$(losetup -f)
- losetup ${FLL_IMAGE_DEV} "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_IMAGE_FILE}"
+ losetup ${FLL_IMAGE_DEV} "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_DM_IMAGE_FILE}"
  FLL_DM_MAP="0 ${FLL_ROOT_DMSIZE} snapshot ${FLL_IMAGE_DEV} ${FLL_SNAP_DEV} P 8"
  dmsetup create ${FLL_DISTRO_NAME}-live --table "${FLL_DM_MAP}"
  mkdir -p "${rootmnt}/${FLL_MOUNTPOINT}"


This will restore the abillity to for example have all three versions bootable from one pen drive or hd partition by changing the names of squashfs file and/or directory where where it resides.

It would be good to have this and a force aufs solution ( previous post ) on the next release for additional live experimenter happiness ( I for example only use a multi layered aufs live box for all my computing needs )

ian_s

[devil]

Rest assured, we will look into both issues as soon as there is time. This is a bit like an operation on the open heart, it wants to be done right. Thanks for commiting this.
I just opened a ticket: http://chili.siduction.org/issues/586

greetz
devil

[ian_s]

Here is a better combined patch to initrd0.686 /scripts/fll

You can now access a dm style squashfs file with a different name and location. ( not now only siduction.686 in /siduction )
this requires also that

FLL_DM_IMAGE_FILE="siduction.686"

is added to initrd0.686 /etc/default/distro.

You can now use aufs boot code with dm style squashfs file. ( no extra force_aufs code required )

Code: [Select]

--- orig_fll 2012-01-04 11:29:12.000000000 +0000
+++ aufs_image_name_image_dir_fll 2012-02-01 10:25:46.000000000 +0000
@@ -981,10 +981,10 @@
  FLL_SNAP_DEV=$(losetup -f)
  losetup ${FLL_SNAP_DEV} ${rootmnt}/fll/snapshot
  fi
- FLL_ROOT_SIZE=$(ls -l "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_IMAGE_FILE}" | awk '{print $5}')
+ FLL_ROOT_SIZE=$(ls -l "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_DM_IMAGE_FILE}" | awk '{print $5}')
  FLL_ROOT_DMSIZE=$((${FLL_ROOT_SIZE}/512))
  FLL_IMAGE_DEV=$(losetup -f)
- losetup ${FLL_IMAGE_DEV} "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_IMAGE_FILE}"
+ losetup ${FLL_IMAGE_DEV} "${rootmnt}/${FLL_MOUNTPOINT}.mnt/${FLL_DM_IMAGE_FILE}"
  FLL_DM_MAP="0 ${FLL_ROOT_DMSIZE} snapshot ${FLL_IMAGE_DEV} ${FLL_SNAP_DEV} P 8"
  dmsetup create ${FLL_DISTRO_NAME}-live --table "${FLL_DM_MAP}"
  mkdir -p "${rootmnt}/${FLL_MOUNTPOINT}"
@@ -1011,9 +1011,16 @@
  # prepare COW union filesystem
  #
  mkdir -p "${rootmnt}${FLL_UNION_MOUNTPOINT}" "${FLL_UNION_COWMNTPNT}${FLL_UNION_COWDIR}"
-
  #
- # union module specific options
+ # if aufs required for dm style squashfs move and mount the ext4 file
+ #
+ if [ -e "${rootmnt}${FLL_MOUNTPOINT}/${FLL_DM_IMAGE_FILE}" ] ; then
+ mkdir -p "${rootmnt}${FLL_MOUNTPOINT}.force_aufs/"
+ mount -n -o move "${rootmnt}/${FLL_MOUNTPOINT}/" "${rootmnt}${FLL_MOUNTPOINT}.force_aufs/"
+ mount -t ext4 "${rootmnt}${FLL_MOUNTPOINT}.force_aufs/${FLL_DM_IMAGE_FILE}" "${rootmnt}${FLL_MOUNTPOINT}"
+ fi
+ #
+ # union module specific options  
  #
  case "${FLL_UNION_MODULE}" in
  aufs)


ian_s

[zeppo]

Hi ian_s,

We're not using aufs at all in our image. For our "virtual" layer we use dmsetup.
At the moment persistence is not working with dmsetup as we need to have an existing cow file. The initrd fll start script searches for this file mentioned as siduction-rw.You can define your cow-file at bootprompt via "persist=$PATH_ON_USB/$FILENAMEOFCOW".
We don't generate that cow file automatically at the moment.

At the moment I'm not sure if i've got your point, maybe you can explain me your goal some more

Thanks + Regards

zeppo

[ian_s]

hi zeppo,

These past years I have been using a live system using multiple layers of aufs ( formally unionfs ) since kanotix, sidux, aptosid and now siduction. for example the system I am writing to you now on has these layers:-

Code: [Select]

siducer@siduction:~$ ls-auf-fs.rb  
Mount Point:  /fll/aufs
si Ident   :  si=ebebb4dc
              /fll/cow  (rw)
              /fll/livedevp2/siduction11-1-kde/test32/h1  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/h0  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/99-usr-local-bin  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/8-git-gui-aufs-shwh  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/7-vlc-reinstall-youtube-mathomatic  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/6-adjust-prev-ffmpeg-etc  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/5-ug-programming-kdev-etc  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/4-libreoffice-with-dpkg  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/3-kidsgames-edu-extras  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/2-ug3-ruby-utils  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/1-ug2-synaptic-essent  (ro)
              /fll/siduction  (rr)
siducer@siduction:~$


In my /fll/livedevp2/siduction11-1-kde/test32/ directory I have these files directories and links

Code: [Select]

siducer@siduction:/fll/livedevp2/siduction11-1-kde/test32$ ls -l
total 127964
drwxr-xr-x  2 root root      4096 Feb  1 04:27 0
lrwxrwxrwx  1 root root        22 Feb  1 03:27 1 -> 1-ug2-synaptic-essent/
drwxr-xr-x  2 root root      4096 Jan 17 20:58 1.sqfs.dir
drwxr-xr-x  8 root root      4096 Jan 13 18:52 1-ug2-synaptic-essent
lrwxrwxrwx  1 root root        17 Feb  1 03:27 2 -> 2-ug3-ruby-utils/
drwxr-xr-x  9 root root      4096 Jan 13 20:13 2-ug3-ruby-utils
lrwxrwxrwx  1 root root        23 Feb  1 03:27 3 -> 3-kidsgames-edu-extras/
drwxr-xr-x  8 root root      4096 Jan 14 12:51 3-kidsgames-edu-extras
lrwxrwxrwx  1 root root        24 Feb  1 03:28 4 -> 4-libreoffice-with-dpkg/
drwxr-xr-x  8 root root      4096 Feb  1 03:27 4-libreoffice-with-dpkg
lrwxrwxrwx  1 root root        26 Feb  1 03:28 5 -> 5-ug-programming-kdev-etc/
drwxr-xr-x  8 root root      4096 Feb  1 03:27 5-ug-programming-kdev-etc
lrwxrwxrwx  1 root root        25 Feb  1 03:28 6 -> 6-adjust-prev-ffmpeg-etc/
drwxr-xr-x  8 root root      4096 Jan 25 04:32 6-adjust-prev-ffmpeg-etc
lrwxrwxrwx  1 root root        35 Feb  1 03:28 7 -> 7-vlc-reinstall-youtube-mathomatic/
drwxr-xr-x  8 root root      4096 Jan 25 05:29 7-vlc-reinstall-youtube-mathomatic
lrwxrwxrwx  1 root root        20 Feb  5 18:07 8 -> 8-git-gui-aufs-shwh/
drwxr-xr-x  9 root root      4096 Feb  3 13:41 8-git-gui-aufs-shwh
lrwxrwxrwx  1 root root        17 Feb  6 22:32 9 -> 9-joomla-2.5.0-1/
lrwxrwxrwx  1 root root        17 Jan 30 04:33 99 -> 99-usr-local-bin/
drwxr-xr-x  3 root root      4096 Feb  1 02:16 99-usr-local-bin
drwxr-xr-x  4 root root      4096 Feb  6 22:28 9-joomla-2.5.0-1
-rw-r--r--  1 root root 109883392 Feb  7 02:55 9-joomla-2.5.0-1.sqfs
drwxr-xr-x  3 root root      4096 Jan 10 22:48 archives
lrwxrwxrwx  1 root root        19 Jan 12 20:37 base-mount -> ../../../siduction/
drwxr-xr-x  3 root root      4096 Jan 27 22:50 chroot_scripts
etc...

As well as patching fll as described previously I have also added the functionality to add the layers at boot time, ( I have not yet decided on a final method ) at the moment here is the relevant boot menu.
It's grub4dos because I like to be able to just edit it with text editor.


title AUFSPATCH SCRIPTS-LAYERS-siduction-20120122 kde live on livedevp2

find --set-root --ignore-floppies /livedevp2.tag

kernel /siduction11-1-kde/vmlinuz0.686 boot=fll image_dir=siduction11-1-kde image_name=siduction-sqfs-kde-base-ug-20120122.686 quiet vga=791  keytable=gb tz=Europe/London fromhd=LABEL=livedevp2 firmware aufs layers_dir=/siduction11-1-kde/test32 layers=1,2,3,4,5,6,7,8,99,h0,h1

initrd /siduction11-1-kde/aufsnew-initrd0.686

From above you can see layers, so far 1 to 8 is software added ( sequentially ) over time.
layer 99 is just a few scripts added to /usr/local/bin
layer h0 and h1 are layers concerning home directory for example, firefox flash and plugins configs of kde etc. and a few links to often used file and directories on other partitions.


Now when I wan't to add new software ( It could be something i really want, or just be on a whim, an experiment, just for the hell of it ).

remount the partition rw
Create a new directory say mkdir /fll/livedevp2/siduction11-1-kde/test32/10-example-experiment
and a link to it ln -s  10-example-experiment/ 10
Create a chroot that will inlude the layers I'm running on and the new layer ( but not the home layers h0 h1 ).

a comandline to show for completeness:-
mount -v -t aufs -o br:10/=rw:8/=ro:7/=ro:6/=ro:5/=ro:4/=ro:3/=ro:2/=ro:1/=ro:0/=ro:base-mount/=ro none rt_chroot/

Now all the aufs mounts are:-

root@siduction:/fll/livedevp2/siduction11-1-kde/test32# ls-auf-fs.rb
Mount Point:  /fll/aufs
si Ident   :  si=ebebb4dc
              /fll/cow  (rw)
              /fll/livedevp2/siduction11-1-kde/test32/h1  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/h0  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/99-usr-local-bin  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/8-git-gui-aufs-shwh  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/7-vlc-reinstall-youtube-mathomatic  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/6-adjust-prev-ffmpeg-etc  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/5-ug-programming-kdev-etc  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/4-libreoffice-with-dpkg  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/3-kidsgames-edu-extras  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/2-ug3-ruby-utils  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/1-ug2-synaptic-essent  (ro)
              /fll/siduction  (rr)

Mount Point:  /fll/livedevp2/siduction11-1-kde/test32/rt_chroot
si Ident   :  si=f0224edc
              /fll/livedevp2/siduction11-1-kde/test32/10-example-experiment  (rw)
              /fll/livedevp2/siduction11-1-kde/test32/8-git-gui-aufs-shwh  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/7-vlc-reinstall-youtube-mathomatic  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/6-adjust-prev-ffmpeg-etc  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/5-ug-programming-kdev-etc  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/4-libreoffice-with-dpkg  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/3-kidsgames-edu-extras  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/2-ug3-ruby-utils  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/1-ug2-synaptic-essent  (ro)
              /fll/livedevp2/siduction11-1-kde/test32/0  (ro)
              /fll/siduction  (ro)

bind mount these directories:-
mount --bind /tmp rt_chroot/tmp
mount --bind /dev rt_chroot/dev
mount --bind /run rt_chroot/run


enter the chroot:-
chroot rt_chroot

run a script in the chroot to mount these:-
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devpts none /dev/pts

apt-get update
apt-get install any_sotware_you_want

before you leave:-

umount /proc
umount /sys
umount /dev/pts

exit the chroot:-
exit or ctrl-d

Make sure you are not accessing the chroot with say a file manager, and umount the binds and the chroot:-

umount rt_chroot/tmp
umount rt_chroot/dev
umount rt_chroot/run
umount rt_chroot

Occasionally the rt_chroot will not unmount because of open processes, you can:-
lsof rt_chroot
and kill them

Next time you boot to test out the added layer boot with
layers=1,2,3,4,5,6,7,8,10,99,h0,h1

Ok it might just have been an experimental test of some software, but remember you have an instant and better version of "microslops system restore" ( thats instant and reverseable ). ie. Just don't boot the end layer/layers in your sequence.


Ok assume now I have all the software I need, and I want it portable on a pen drive - but it's way too big in these layers of directories, I even have over 1gig of debs in these layers.

I go through the same steps as above but instead of adding a new software layer, add a new cleanup layer eg 10-cleanup/

In the chroot, apt-get clean and purge. Clean out the logs etc. if you want to ( hint you can look in /fll/siduction to see whats in the original squashfs file as a guide ).

Unmount internal directories come out of the chroot and unmount the binds but do not unmount the aufs chroot.
You are now ready to rebuild a main squashfs file ( some would call it a poor mans or frugal rem*st*r ) is that word allowed here?

mksquashfs rt_chroot my-big-bloated-new-siduction.sqfs

Replace squashfs on pen drive with new one and adjust names and or boot menu as appropriate.

---------------------------------------------------------

With the changes in the initrd0.686 outlined in the previous post:-
adding one line to initrd0.686 /etc/default/distro
changing 2 lines and adding 5 lines of code to initrd0.686 /scripts/fll

Will enable the following ( without changing the current style which is a squashfs file which now contains a sparse ext4 filesystem file )

In dmsetup live mode not changed ie. "virtual" layer using dmsetup:-
Still has the same usabillity of being an ideal way to quickly get a full HD install of up to date debian sid.
The same functionality of creating live system installed to pendrive etc.
added - the image_dir and image_name boot code now works.

Added functionallity
If you boot with the aufs boot code you have a live system using aufs:-
Ability to add as many aufs layers as you want.
Examine the files in the individual layers
Persistance works by just naming a persistant directory ie persist=siduction/my-persist.
Easy rebuilding of the base squashfs file.
Lots more options to experiment with, and have fun with live systems, if you want to.

Also still has the ability to use the previous style of squashfs file.

---------------------------------------------------------
 

As you can see on my system I have already added more, and am testing and refining:-
Automatically adding multiple layers with boot code.
Calling named scripts on HD with boot codes.
Mounting layers compressed into squashfs or other filesystems.
Easy adding self-contained modules like xampp www.apachefriends.org and the many and various BitNami stacks bitnami.org/
All potentially able to run from a live system.


ian_s