Linux Format forums Forum Index Linux Format forums
Help, discussion, magazine feedback and more
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

bash wrap for dd | split | gzip ?

 
Post new topic   Reply to topic    Linux Format forums Forum Index -> Programming
View previous topic :: View next topic  
Author Message
silly_xp_user



Joined: Sat Oct 22, 2005 4:55 pm
Posts: 11

PostPosted: Sat Nov 04, 2006 3:52 pm    Post subject: bash wrap for dd | split | gzip ? Reply with quote

Hello,

Can anyone please help...

I have 2 ways of backing up a drive (with the output split) to a drive on a network:-

Code:
Method A)
----------
dd bs=16M if=/dev/hda | gzip -c | split -b 256m - \ \mnt\nwdrive\bak.img.gz.

gives portions of a gzipped hard drive in \mnt\nwdrive\bak.img.gz.*

Code:
Method B)
----------
dd bs=16M if=/dev/hda |  split -b 256m - \mnt\nwdrive\bak.img.

gzip  \mnt\nwdrive\bak.img.*

gives gzipped portions of a hard drive in bak.img.*.gz

However all intermediate back.img.* files are written to disk so need the space to hold them all. I don't have the space locally or on the network drive.

Method A) is the more usual way, it gzips the entire drive and then splits the output.
Method B) splits the dd of the entire drive then gzips those splits.

I prefer Method B) but I am limited on space and bandwidth.

Are there any obvious way of achieving the same thing as B) but not having to use all that space by varying parameters to dd or doing some bash scripting? Somehow I need to create and gzip one split at a time.

If at all possible I would prefer to stick to these tools plus the shell, I am not interested in G4U, ghost, etc.

Thanks for any constructive help given.

Hal.
Back to top
View user's profile Send private message
MartyBartfast
LXF regular


Joined: Mon Aug 22, 2005 8:25 am
Posts: 814
Location: Hants, UK

PostPosted: Sat Nov 04, 2006 4:59 pm    Post subject: RE: bash wrap for dd | split | gzip ? Reply with quote

Never used dd, so can't really comment, but I can offer another option:

/sbin/dump -0 -z9 -M -B 256000 -f bak.dump /usr

will give you a compressed multi volume dump of /usr in bak.dump001 bak.dump002 etc with each portion being 256 Meg,
you then need restore to recover files from the dumps
_________________
I have been touched by his noodly appendage.
Back to top
View user's profile Send private message
nelz
Site admin


Joined: Mon Apr 04, 2005 12:52 pm
Posts: 8449
Location: Warrington, UK

PostPosted: Sat Nov 04, 2006 6:23 pm    Post subject: RE: bash wrap for dd | split | gzip ? Reply with quote

I don't think dd is the right tool for this job, because you are copying every bit on the disk, even the no longer used spaces. I'd use partimage for this, which works well over a network, only backs up the parts of the disk that are in use and is capable of splitting and gzipping the files.

However, if you do want to use dd, you aren't quite correct in stating that A "gzips the entire drive and then splits the output". The use of pipes mean that dd, gzip and split are operating simultaneously. One of the advantage of this is that the data is compressed before transfer over the network, which is usually faster, unless you have a very fast network and are backing up a slow machine. In that case you could use B with a file notification program running on the destination machine, something like fileschanged which will detect each new backup file and can be told to gzip it.
_________________
"Insanity: doing the same thing over and over again and expecting different results." (Albert Einstein)
Back to top
View user's profile Send private message
wdlerner



Joined: Wed Jan 03, 2007 2:24 am
Posts: 2
Location: Virginia, USA

PostPosted: Wed Jan 03, 2007 2:42 am    Post subject: Reply with quote

Your block size seems a bit large. Can you explain why you are using a 16 Megabyte block size?

Thanks!
Back to top
View user's profile Send private message
View previous topic :: View next topic  
Display posts from previous:   
Post new topic   Reply to topic    Linux Format forums Forum Index -> Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Linux Format forums topic RSS feed 


Powered by phpBB © 2001, 2005 phpBB Group


Copyright 2011 Future Publishing, all rights reserved.


Web hosting by UKFast