The context could be a virtualised cluster, where an hypervisor went suddenly down. 2 of your Mongo replicas are unavailable, only 1 is left, which then of course drops back to being secondary and read only.
You want to have this server running alone for a while while the others come back online, as you decide that it is better to have potential small inconsistency instead of not running for a few hours. The thing is that this last server will complain that the rest of the set is not available. To get it started again, you just need to make it forget about the rest of the set.
- Switch the service off
service mongodb stop
- Remove the line replSet from your /etc/mongodb.conf
- Restart the service
service mongodb start
Mongo will complain:
mongod started without --replSet yet 1 documents are present in local.system.replset [initandlisten] ** Restart with --replSet unless you are doing maintenance and no other clients are connected. [initandlisten] ** The TTL collection monitor will not start because of this. [initandlisten] ** For more info see http://dochub.mongodb.org/core/ttlcollections
- Remove the offending document in system.replset from the mongoshell
// will give you one document back db.system.replset.find() // remove all documents (there is only one) db.system.replset.remove({}) // check resultset is empty db.system.replset.find()
- Restart mongo
service mongodb stop service mongodb start
- Once the other nodes are up, add again the replSet line in /etc/mongodb.conf and restart the service.