bash - Mongodb incremental backups -

-

i given task setup incremental backups mongodb replicaset, start point of course googled , not find on mongodb docs, did find question encouraged develop own solution didn't find tayra active.

i read oplog , realized easy develop replay log, turns out didn't have mongorestore me.

now have working solution bash scripts , quite easy, that's reason asking here if there flaw in logic, or maybe bite me in future.

below how implemented that:

full backup procedure:

  1. lock writes on secondary member db.fsynclock()
  2. take snapshot
  3. record last position oplog

db.oplog.rs.find().sort({$natural:-1}).limit(1).next().ts

  1. unlock writes db.fsyncunlock()

incremental backup procedure:

  1. lock writes on secondary member
  2. dump oplog recorded oplog position on full (or latest incremental ) backup: mongodump --host <secondary> -d local -c oplog.rs -o /mnt/mongo-test_backup/1 --query '{ "ts" : { $gt : timestamp(1437725201, 50) } }'
  3. record latest oplog position (same way full backups)
  4. unlock writes

full backup restore procedure:

  1. stop instances of mongod
  2. copy snapshot data dir of box primary, make sure exclude local* , mongod.lock restore technique called reconfigure breaking mirror
  3. start primary
  4. reconfigure replicaset
  5. start secondaries without data, let them perform initial sync. or copy data new primary fresh local database

restore incremental backup:

when created incremental backup stored this:

/mnt/mongo-test_backup/1/local/oplog.rs.bson /mnt/mongo-test_backup/1/local/oplog.rs.metadata.json

we're instered on oplog.rs.bson have rename it, here steps:

  1. change directory backup: cd /mnt/mongo-test_backup/1/local
  2. delete json file rm *.json
  3. rename bson file mv oplog.rs.bson oplog.bson
  4. restore : mongorestore -h <primary> --port <port> --oplogreplay /mnt/mongo-test_backup/1/local

i have scripted, may commit on github later.

question if there flaw in logic. bit suspicious procedure quite straight forward , still couldn't find documented anywhere.


Comments

  1. AmeliaFebruary 28, 2018 at 1:15 AM

    This blog share good information on MongoDB incremental backup and its procedure. Thanks for sharing

    ReplyDelete

Post a Comment

Popular posts from this blog

node.js - StackOverflow API not returning JSON -

-
i using node.js script gather data stackoverflow. know responses gzipped, put code in should take care of that. script follows: var request = require('request'); var zlib = require('zlib'); function getstackoverflowresponse(url, callback){ request(url, {encoding: null}, function(err, response, body){ if(response.headers['content-encoding'] == 'gzip'){ zlib.gunzip(body, function(err, dezipped) { callback(dezipped); }); } else { callback(body); } }); } var url = "https://api.stackexchange.com/docs/questions#pagesize=2&order=desc&min=2014-01-04&max=2014-02-02&sort=activity&tagged=apigee&filter=default&site=stackoverflow&run=true"; getstackoverflowresponse(url, function(questions) { console.log(questions); }); instead of getting json output, i'm getting following response: buffer 0d 0a 0d 0a 0d 0a 0d 0a 3c 21 44 4f 43 54 59 50 45 20 48...
Read more

python - Subclassed QStyledItemDelegate ignores Stylesheet -

-
i'm trying subclass qstyleditemdelegate remove focus rectangle in qcombobox . even though i'm calling base implementation of paint function , nothing else, result different. looks if parts of style-sheet influence bounding box of item taken consideration. class pstyleditemdelegate(qstyleditemdelegate): def __init__(self, *args, **kwds): super(pstyleditemdelegate, self).__init__(*args, **kwds) def paint(self, *args, **kwargs): qstyleditemdelegate.paint(*args, **kwargs) what have paint unmodified qstyleditemdelegate ? as suggested tried replacing pyside pyqt4 , works, appears bug. update pyside 1.1.2 1.2.1 result same. unfortunately switch breaks other parts of code, if there no other suggestions i'm going accept answer. edit bug tracked here
Read more

java - HttpClient 3.1 Connection pooling vs HttpClient 4.3.2 -

-
i have used httpclient3.1(java) connection pooling below : import org.apache.commons.httpclient.httpclient; import org.apache.commons.httpclient.multithreadedhttpconnectionmanager; import org.apache.commons.httpclient.params.httpclientparams; import org.apache.commons.httpclient.params.httpmethodparams; import org.apache.log4j.logger; public class httpclientmanager { private static logger logger = logger.getlogger(httpclientmanager.class); private static multithreadedhttpconnectionmanager connectionmanager = new multithreadedhttpconnectionmanager(); private static httpclientparams clientparams = new httpclientparams(); static { try { clientparams.makelenient(); clientparams.setauthenticationpreemptive(false); clientparams.setparameter(httpmethodparams.so_timeout, new integer(30000)); // set so_timeout clientparams.setparameter(httpmethodparams.head_body_check_timeout, new integer(3000)); // head responses ...
Read more