I'm bad at running at the mouth in posts, so I'll try to stay on track:

I have a BBS VM. I mount some NFS share drives on the BBS VM. I also have a PBS backup server.

Last week some users mentioned they were having DL issues; I thought it was a separate issue with filenames - but I've been upgrading many systems to Debian 12 this past week and finally got around to my PBS machine. I noticed in the PBS dashboard that only 2TB of a 16TB drive (thats usually close to full) was being used. Wait, whaaa??? So I checked its NFS backup's DIRs and they were all empty!! HUH?

The rsync command I was using to backup had the --delete tag. I thought that if I modified my BBS VM and removed some /mystic files that I'd want that to be replicated on my backups....

Well; come to find out, my NAS went down sometime last week... and, my BBS VM and PBS servers NFS mount points went down. Historically, I thought this fixed itself - I'm almost sure of it, but...

The weekly rsync backup ran - and... --delete made sure ALL the backup data was - deleted. And, users weren't able to download this past week and the bug reports started flowing in.

When I found it, a simple reboot and remount fixed things... I'm currently backing up some 32TB of data; great for the drive, right? Delete all the data and then re-write it. Awesome.

I've removed the --delete tag, and am rethinking how I'm doing things... guess I need to put in REDUNDANT checks on the mountpoints, and maybe check in my backup script that if /WHATEVER/MOUNTPOINT is empty, don't backup.

Ugh; stupid, Paulie...



|07p|15AULIE|1142|07o
|08.........

--- Mystic BBS v1.12 A48 (Linux/64)
* Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (21:2/150)

On 29 Jun 2023, paulie420 said the following...

The weekly rsync backup ran - and... --delete made sure ALL the backup data was - deleted. And, users weren't able to download this past week
and the bug reports started flowing in.

ouch, saw it coming the second i saw 'rsync' ..

synced, incremental, etc backups always creeped me out because of that type of thing.. even if they save space.

--- Mystic BBS v1.12 A47 2021/12/25 (Windows/32)
* Origin: cold fusion - cfbbs.net - grand rapids, mi (21:1/616)

The rsync command I was using to backup had the --delete tag. I thought

Does '--max-delete=NUM' help your use case at all?

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: TassieBob BBS, Hobart, Tasmania (21:3/169)

Re: Backup Nightmares @ 2oFB..
By: paulie420 to All on Thu Jun 29 2023 07:49 pm

Hey...

The weekly rsync backup ran - and... --delete made sure ALL the backup data was - deleted. And, users weren't able to download this past week and the bug reports started flowing in.

Sorry for your luck dude...

Reminds me of a time that I worked for an airline (IT admin) long time ago, and we had a system that kept core dumping (it was a new app), and the core files were so big, that 2 or 3 of them filled some important mount points.

The core files were important for the developers, so I had to keep them, so I wrote a script to find them each night and compress them (they compressed really well if I recall).

I tested, tested, tested - since my script tralled through the whole system looking for core files and I was confident it was working.

I deployed the script to another important system (starting with the airlines network management system), and got a call at 1am - the system wasnt working, and I couldnt remote it to look at it. When I got to the office, my script had compressed every file on the host. Argh! - That was a reinstall and a couple of days I couldnt of spent doing something better...

Anyway, wanted to share, if you want a different approach to backup, take a look at restic - IMHO, it does a better job than a rsync approach to backup.


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

The weekly rsync backup ran - and... --delete made sure ALL the

ouch, saw it coming the second i saw 'rsync' ..

Yeah, I don't use --delete with rsync, ever... even if it's annoying cleaning up the extra files on the backup side if I ever want/need it. Worst is when some doors crash and create a 4gb data file in their wake.


--
Michael J. Ryan
+o roughneckbbs.com
tracker1@roughneckbbs.com
--- SBBSecho 3.15-Linux
* Origin: Roughneck BBS - roughneckbbs.com (21:3/149)

Does '--max-delete=NUM' help your use case at all?

COULD have; but I'm switching to a more modern backup utility... currently enamored with borg-backup, and thinking of some borg add-ons for a web-gui; dunno if I'll land here, tho.

It's interesting, b/c I found borg-import that allows me to import rsync backups directly into my borg-backups...

I might also switch to something in my NAS running openmediaserver... they have to have a backup add-on, right?? :P

I use PBS (proxmox backup server) for my PVE (proxmox) VMs, and love it - but I wish they had some simple add-on to handle basic non-PVE backups. There is a CLI command, but when I tested it a while ago I wasn't super impressed.



|07p|15AULIE|1142|07o
|08.........

--- Mystic BBS v1.12 A48 (Linux/64)
* Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (21:2/150)

The weekly rsync backup ran - and... --delete made sure ALL the backup was - deleted. And, users weren't able to download this past week and t bug reports started flowing in.

Sorry for your luck dude...

Well... I guess the title should have been 'backup issues' as I didn't lose any data; that would have been the 'nightmare'. :P I'll get there... sometimes intermediate users shouldn't get over-happy using basic tools - we think we know everything, only to have the software (working correctly) to hurt us b/c we didn't think of all the things...

Reminds me of a time that I worked for an airline (IT admin) long time ago, and we had a system that kept core dumping (it was a new app), and the core files were so big, that 2 or 3 of them filled some important mount points.

The core files were important for the developers, so I had to keep them, so I wrote a script to find them each night and compress them (they compressed really well if I recall).

I tested, tested, tested - since my script tralled through the whole system looking for core files and I was confident it was working.

I deployed the script to another important system (starting with the airlines network management system), and got a call at 1am - the system wasnt working, and I couldnt remote it to look at it. When I got to the office, my script had compressed every file on the host. Argh! - That
was a reinstall and a couple of days I couldnt of spent doing something better...

:P I suppose these issues help us learn in the end... as stated earlier, my issue wasn't as hurtful as even yours - I didn't LOSE anything; its just knowing that I could have... I didn't even know for a few days. SMFH!

Anyway, wanted to share, if you want a different approach to backup,
take a look at restic - IMHO, it does a better job than a rsync approach to backup.

I *did* want to look at restic - I think I will, before I choose to permanently land @ borg-backup; I know that both of those tools have various add-ons that add more function... something will work out, and I'll be better off for it - trust me, I still love rsync - I just realize now that you need to think of all the things - and what actual use case yer trying to get to... some 'solutions' (--delete to keep ~/mystic up to date) really aren't good choices.



|07p|15AULIE|1142|07o
|08.........

--- Mystic BBS v1.12 A48 (Linux/64)
* Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (21:2/150)