Nov 212013
 

When configuring the attributes of members of a Mongo DB replica set, either in the mongo shell or a script, the typical procedure is to set each attribute by its array reference. The weakness in this method is that it’s necessary to know what the array index is of the host whose attribute you need to configure. This may not be possible for a generalized script.

cfg = rs.config();
cfg.members[1].priority = 2;
rs.reconfig(cfg);

However, with a bit of mongo javascript it’s possible to write a small function to cycle through the member config array, determine the index of a host and then set an attribute.

In this example, we want to configure the Mongo DB nodedb02.example.com as a hidden node. This can be done like this:

cfg = rs.config();
cfg.members.forEach( 
	function(item, i) {
		if ( item.host == "nodedb02.example.com" ) 
		{ 
			cfg.members[i].priority = 0;
			cfg.members[i].hidden = true;
			rs.reconfig(cfg);
		}  
	} 
)

If, on the other hand, you wanted to change the slave delay on a read-only reporting node, this could be accomplished by testing each node to find which has the “hidden” attribute set to “true”:

		if ( item.hidden == "true" ) 
		{ 
			cfg.members[i].slaveDelay=0
			rs.reconfig(cfg)
		}  

Use this function in Mongo JS scripts, be it with Capistrano, Puppet or some other automation technique, and everything got just a bit more hands-off.


Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

Sep 052013
 

While operating a MongoDB cluster, it can often be useful in maintenance scripts to check whether a host is the primary or secondary (master or slave). This is easy to do, and can be done without having to authenticate against the database. Scripts can be made simpler as there is no need to shield the login credentials, as there aren’t any.

When one connects a Mongo database process without credentials, using the “mongo” client command, the implicit “test” database is used. This is a default, and a kind of empty chroot database. Even though it’s an empty database, it still has read-only access to replica information. This can be very handy when automating deployment and monitoring scripts.

There are two commands in particular that are useful for determining the status of a replica set member. These are demonstrated in the following console example, which is fairly self-explanatory.

$ mongo localhost:27017
connecting to: localhost:27017/test
> db.isMaster().secondary
false
> db.isMaster().ismaster
true

This could also be executed non-interactively like this:

mongo localhost:27017 --quiet --eval 'db.isMaster().secondary'

Shell Script Snippet: wait for replica synchronization to finish

When resynchronizing a replica set slave, it will report its status as RECOVERING until it completes, at which point it becomes SECONDARY. The simple example below demonstrates a block of bash to test for this.

while [ "$RESULT" != "true" ]; do
   RESULT=$(mongo localhost --quiet --eval 'db.isMaster().secondary');
   sleep 5;
done

This is over-simplified to demonstrate the logic. In a real-world script you’d have an exit clause to prevent an endless loop in the case when the replica node never recovers.

Other useful commands can be discovered while browsing the mongo console. Find these using commands of the format:

  db.help()


Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

Mar 082013
 

AMANDA is a free and reliable backup system for Linux. Out of the box, it backs up filesystems, MySQL and PostgreSQL. It has a plugin API, and this allows one to write scripts to extend AMANDA’s functionality – to back up a Mongo database, for instance.

To do so, all that is needed is a custom script to perform the necessary database dump and clean-up commands, and some configuration in AMANDA to call this script with the correct arguments. The AMANDA script API is discussed extensively on the official Zmanda Wiki page. The plugin I’ve written, and the procedure discussed, is relevant to Linux only. This won’t work on Windows.

Mongo Backups

There are several ways of backing up a Mongo database, as discussed in the Mongo DB
Documentation. It’s worth reviewing this page before proceeding.

For a small database, the mongodump command will perform adequately, and that is the method that the ammongodb plugin script uses. For larger DBs with heavier IO, filesystem snapshotting (with LVM or ZFS) may be the more suitable solution. A later version of the script will allow this functionality.

Plugin script: ammongodb

The following script ammongodb should be copied to your AMANDA plugin directory, which may be /usr/libexec/amanda/application (depending on the OS you’re using). It can be downloaded from Github, from the link below.
ammongodb on gitub: https://github.com/matthewparsons/ammongodb

AMANDA Configuration

In order to configure AMANDA to use the plugin script, update the configuration-specific amanda.conf file with these blocks.
amanda.conf

    define script-tool script-dump-mongo-replica {
        comment "Execute mongodump script"
        plugin "ammongodb"
        execute-where client
        execute-on pre-dle-backup, post-dle-backup
        property "port" "27018"
        property "auth" "/var/mongo/auth/mongo_backup_auth.cfg"
        property "log" "/var/logs/mongo/mongo-amanda.log"
    }

    define dumptype dump-mongo-replica {
        comp-user-tar
        comment "Mongo Dump"
        script "script-dump-mongo-replica"
        auth "bsdtcp"
    } 

And then, in my disklist file, use the dumptype like this:
disklist

  proddb.example.com  /var/mongo/dump  dump-mongo-replica

Mongo Configuration

Finally, Mongo needs to be permissioned to allow the backup user to login. As from Mongo DB 2.2.2, the backup user cannot be read-only, and needs write access to the admin database.

Create a user with the credentials defined in the “auth” file defined in the amanda.conf file:
/var/mongo/auth/mongo_backup_auth.cfg

USER=backup
PASSWORD=Backup123

Create the same user on the read-only Mongo node’s admin database:

mongo localhost:27018/admin -u admin -p
   mongo> db.addUser("backup","Backup1234",false)

Invoking Backup

Using the AMANDA backup user, amandabackup

  amandabackup$>  amdump DailyDump

If all goes well, this should back up all the compressed Mongo JSON files. To debug any problems, check the AMANDA client sendbackup file, or the logfile (/var/logs/mongo/mongo-amanda.log) defined in the amanda.conf dumptype definition.

Any other queries, I’m happy to respond to in the comments below.


Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

Jun 062012
 

There are many occasions where you get locked out of a Mongo DB database. It could be that the admin password is unknown, or that the only admin user that exists is “read-only” (something that Mongo permits you to do). Fortunately, it’s fairly straightforward to remove the admin database safely and recreate the admin user.

Mongo DB has an idiomatic, not to mention pragmatic approach to user authorization. Access to the database is simplified, and not particular granular, with the idea that access control should be passed more to the connecting application itself. Thus, the default access control is to have no users at all. You need to create an “admin” user account if you want one. Nothing, however enforces this.

One creates an admin user by creating a user on the “admin” database, after logging in to Mongo:

  # mongo localhost
MongoDB shell version: 2.0.2
connecting to: localhost/test
  > use admin
switched to db admin
  >  db.addUser("admin","Password")

And that’s it.

Once this user exists, then every time you login to the database, you need to authenticate, like this:

  # mongo localhost
MongoDB shell version: 2.0.2
connecting to: localhost/test
  > use admin
switched to db admin
  mongo> db.auth("admin","Password")

Failure to authenticate will allow you your connection, but no access to any data. Once again, it’s the fact of the existence of the admin user which enforces access control, as well as the following option in the Mongo DB config file:

  auth=true

Solution

This solution will focus on removal of the admin user in a replica set, because it’s slightly more complicated, and slightly more interesting. If you’re just running a single node, the procedure is considerably easier. Just ignore the cluster steps.

Stop all Mongo processes running in the replica set. These will probably be running with the command line or config file option of “replSet”. Check by running “ps” or checking the /etc/mongod.conf file.

When no Mongo replicas are running, perform the following procedure on each node in turn. You’ll need to bring the Mongo process up at one point, but it is vital that only one replica is running at any one time, otherwise the configuration will get resynchronised from another node, and you don’t want that. You want it changed.

On each node

Change to the mongo directory where the data files reside, possibly /var/lib/mongo but specified in the configuration file by the dbPath parameter.

  # cd /var/lib/mongo

Move the admin.* files somewhere else to back them up.

  # mv admin.* /tmp

It’s that simple. Deleting these datafiles will have removed the admin credentials from the database.

Start the mongo replica daemon on this single node. Depending on your distribution and how it’s configured, it’s probably something like this:

  # /etc/init.d/mongod start
  # mongo localhost
MongoDB shell version: 2.0.2
connecting to: localhost/test
  > use admin
switched to db admin
  > db.addUser("admin","password")

At this point, the admin datafiles should have been recreated. Check that the credentials work:

  > db.auth("admin","password")
1

If this returns the numeral “1”, then the login is successful. However, if this is a cluster, the node you’re on may have been marked as stale. To check and fix:

  > db.system.users.find()
error: { "$err" : "not master and slaveok=false", "code" : 13435 }
  > rs.slaveOk()
not master and slaveok=false

The rs.slaveOk() command will allow you to query normally from the secondary:

  SECONDARY> db.system.users.find()
{ "_id" : ObjectId("4fc4972525a7b704e9a3a09e"), "user" : "admin", "readOnly" : false, "pwd" : "2f1bffb1d28a2cca21679103652b1040" }

Now stop mongo, and repeat the above procedure on the other nodes:

  # /etc/init.d/mongod stop

When you have updated the admin password on all mongod replicas, restart all nodes in the replica set.

Now, you should find that all nodes have admin user accounts, and that you can use these to authenticate against each one.


Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.