LINUX - virtually merge multiple folders with symlinks or hardlinks with cp command

How to combine/merge 2 or more folders virtually(sym/hardlinks):
#################################################################
#################################################################

URL: http://unix.stackexchange.com/questions/40634/in-ubuntu-is-there-a-way-to-virtually-merge-two-folders-without-unionfs-or-aufs

Lets say you have a Movies1 and Movies2 folder and you want their contents to be visible through a Movies folder (which has the combination of Movies1 and Movies2). They dont take up the extra space thus its virtual (because symlinks/hardlinks are involved)

Or more example type of scenario imagine folder 1 and 32 that you want to combine to folder 3:

START:
------
/testfolder
├── 1
│   ├── s
│   ├── x
│   ├── y
│   └── z
│       └── 1
└── 2
    ├── a
    │   └── 2
    ├── b
    └── c

FINISH:
-------

/testfolder
├── 3
    ├── s
    ├── x
    ├── y
    └── z
    |    └── 1
|
    ├── a
    │   └── 2
    ├── b
    └── c

Note: 1 and 2 still exist they are just not shown
SIDE NOTES:
-----------
* 3 is not in alphabetical order but it would be after the combine with a simple "tree" command.
* To get similar output download "apt-get install tree"
* shopt -s dotglob is used below for * to also "grab" hidden files
* Source file paths have to be absolute in such case (that's why I used $PWD), because cp "can make relative symbolic links only in current directory".
SOLUTION:
----------
You can "copy" whole tree using symlinks (or hard links if you want) automatically using -s option of cp (or -l for hard links).

# cd /testfolder
# mkdir 3
# shopt -s dotglob # <---- optional - turns on dotglob to grab hidden dot files with *
# cp -ans "$PWD/1/"* 3/
# cp -ans "$PWD/2/"* 3/
# shopt -u dotglob # <---- optional - turns off dotglob if you turned it on

VARIATION NOTE: the " " are optional so you could just put:
# cp -ans $PWD/1/* 3/
# cp -ans $PWD/2/* 3/

VARIATION NOTE: instead of $PWD/1/* and $PWD/2/* - I could just use the full path of the folders with an * ending.
mkdir /testfolder/3
shopt -s dotglob
cp -ans /testfolder/1/* /testfolder/3/
cp -ans /testfolder/2/* /testfolder/3/
shopt -u dotglob

NOTE ON SPEED: This is a very quick process and can be monitored -v (for verbosity) shows every transfer - realize that it doesnt really transfer it just makes a link so it makes a small (very small file) so this process is much faster then a regular cp

NOTE: $PWD is just a variable kept in the bash system at all times which tells bash what current folder your in - random fact: changing that variable changes your path - to see what it equals just type # pwd (which is a command that simple does the following # echo $PWD) - infact try it "echo $PWD" that will list the current directory

ANOTHER INTERESTING SOLUTION:
------------------------------
* You can also do the case of having folder 1 to 2 where there is no folder 3, the end result would feel the same. So folder 2 would have symlinks/hardlinks to folder 1. A user in folder 2 would see folder 1 and folder 2 contents

# cd /testfolder
# mkdir 3
# shopt -s dotglob    # <---- optional - turns on dotglob to grab hidden dot files with *
# cp -ans "$PWD/1/"* 2/
# shopt -u dotglob    # <---- optional - turns off dotglob if you turned it on

END RESULT:
/testfolder
├── 2
    ├── s
    ├── x
    ├── y
    └── z
    |    └── 1
|
    ├── a
    │   └── 2
    ├── b
    └── c
Note: 1 still exist its just not shown

Explanation of used cp options:
--------------------------------
-a or --archive
preserves attributes, links and copies directories recursively (it's in fact alias of -dR --preserve=all)
-n or --no-clobber
prevents overwriting existing files
-s or --symbolic-link
makes symbolic links instead of literal copying
-l to make hard links


PART 2 - Unvirtualizing the result folder (or simply dereferencing the links)
##############################################################################

Lets say we want to convert all of the symlinks and hardlinks to originals, first lets look at why.

So after the copy with -ans (or -vans for more verbosity with the v) - thats the -ans copy from above - we have something that looks like this:

/testfolder
├── 3
    ├── s
    ├── x
    ├── y
    └── z
    |    └── 1
|
    ├── a
    │   └── 2
    ├── b
    └── c

Which is a combination of 1 and 2
However in reality everything within 3 is just symlinks or hardlinks to the actual data so it all looks like this actually

/testfolder
├── 3
    ├── s -> /testfolder/1/s
    ├── x -> /testfolder/1/x
    ├── y -> /testfolder/1/y
    └── z 
    |    └── 1 -> /testfolder/1/z
|
    ├── a
    │   └── 2 -> /testfolder/2/a/2
    ├── b -> /testfolder/2/b
    └── c -> /testfolder/2/c

Lets say you wanted to "unvirtualizing" them so it actually looks like this:

/testfolder
├── 3  <------------------ this will be 4
    ├── s
    ├── x
    ├── y
    └── z
    |    └── 1
|
    ├── a
    │   └── 2
    ├── b
    └── c

Well in reality I dont know about dereferencing each link within 3, however we can make a new folder - lets say /testfolder/4 - where each file will be a none link file (its called copy with dereferencing). So it will be like this

/testfolder
├── 4
    ├── s
    ├── x
    ├── y
    └── z
    |    └── 1
|
    ├── a
    │   └── 2
    ├── b
    └── c

HOW TO:

# cd /testfolder
# mkdir 4
# shopt -s dotglob
# cp -RL 3/* 4
# shopt -u dotglob

NOTE: Dereferencing does a full copy instead of making a small link so this process is much longer

NOTE: again the dotglob is optional - its just to grab the hidden dot files. So if they dont exist in 3 then no need to do the dotglob commands.

SECTION 3 - REAL WORLD EXAMPLE OF SECTION 1 AND SECTION 2 WITH APT-MIRROR AND REPOSITORIES
###########################################################################################

Example of use:
apt-mirror is a program that downloads a full repository to your ubuntu/debian box (you feed it a configuration file which looks similar to sources.list from apt, and it downloads the full repositories).
apt-mirror saves everything to the save location (namely the base_path variable). My base_path is /Sally/Studies/repo1. 
The actual pacakges save into
/Sally/Studies/repo1/mirror/{domain name of repo}
The metadata of the packages which is equaly important for the repository to work is saved into
/Sally/Studies/repo1/skel/{domain name of repo}

So I was downloading the ubuntu and the debian repository so this is what I had after a successful apt-mirror:

/Sally/Studies/repo1/mirror/gb.archive.ubuntu.com/..stuff..
/Sally/Studies/repo1/mirror/ftp.us.debian.org/..stuff..
/Sally/Studies/repo1/skel/gb.archive.ubuntu.com/..stuff..
/Sally/Studies/repo1/skel/ftp.us.debian.org/..stuff..

Problem with this split is that programs like debootstrap and apt-get need everything from skel and mirror to be together (or merged) in fact that abstraction only exists with apt-mirror (as far as I am aware - and I could be completely wrong)

So to combine them, first I pick a name for the combination folder - combo1. Nice and original

cd /Sally/Studies/repo1/
mkdir combo1
shopt -s dotglob
cp -vans /Sally/Studies/repo1/mirror/* combo1
cp -vans /Sally/Studies/repo1/skel/* combo1
shopt -u dotglob

Or another alternative as you see in section 1:

cd /Sally/Studies/repo1/
mkdir combo1
shopt -s dotglob
cp -vans $PWD/mirror/* combo1
cp -vans $PWD/skel/* combo1
shopt -u dotglob

Since the cp commands will take a minute you can do it all in one shot:

cd /Sally/Studies/repo1/
mkdir combo1
shopt -s dotglob
(cp -vans $PWD/mirror/* combo1 && cp -vans $PWD/skel/* combo1) | tee cp.log
shopt -u dotglob

Or instead of $PWD we can just use the full paths:

mkdir /Sally/Studies/repo1/combo1
shopt -s dotglob
(cp -vans /Sally/Studies/repo1/mirror/* /Sally/Studies/repo1/combo1 && cp -vans /Sally/Studies/repo1/skel/* /Sally/Studies/repo1/combo1) | tee /Sally/Studies/repo1/cp.log
shopt -u dotglob

So the final result is
/Sally/Studies/repo1/combo1/gb.archive.ubuntu.com/..stuff of skel and mirror combine - but just symlink versions of it..
/Sally/Studies/repo1/combo1/ftp.us.debian.org/..stuff of skel and mirror combine - but just symlink versions of it..

Now I can point an http server (like an apache2 server) to this folder - Like this:
ln -s /Sally/Studies/repo1/combo1/ /var/www/repos/

Now I can point to them with a remote machines (which still in the same lan - or even across the internet If I have my webserver and firewall correctly setup) with /etc/apt/sources.list
deb https://10.10.10.10/repos/ftp.us.debian.org/debian wheezy main
deb https://10.10.10.10/repos/ftp.us.debian.org/gb.archive.ubuntu.com precise main

Finally lets say I want a USB copy - I cant have it just be symlinks on the USB, I need the actual data - So  I need the section 2 material to derefence the links:
Lets pretend sdX1 is our imaginary USB drive
# mount /dev/sdX1 /mnt
# mkdir /mnt/repocombo/
# shopt -s dotglob
# cp -vRL /Sally/Studies/repo1/combo1/* /mnt/repocombo
# shopt -u dotglob

NOTE: Dereferencing does a full copy instead of making a small link so this process is much longer

SelectionFile type iconFile nameDescriptionSizeRevisionTimeUser
Comments