Home > grails, programming, scalability > A Grails Plugin for Sharding

A Grails Plugin for Sharding

Background

A while back Rob Woollam and I began a project to build a scalable web application for RezzMap.  We evaluated various technologies and decided on the Grails Framework because of it’s ability to quickly make progress.  Having built several scalable applications in the past I knew that building a system the scales horizontally would allow for easy growth in the future.  Given that and my use of database sharding in the past we started looking at how we would approach that in Grails.  At first we found little on the subject beyond a conversation that discussed using Hibernate Shards within Grails.  In this thread the was reference to a blog post by Lee Butts that showed how to use a “switchable datasource” (an implementation of Spring’s AbstractRoutingDataSource) to change the connection of a datasource on the fly.  So we took this blog posting and built a plugin that allows us to have shards of data within Grails based on a user account.  I have since pulled most of this logic out into a plugin that we are sharing with the Grails community.  Over time we will integrate the more advanced features of our internal plugin but I wanted to get a more basic version out there for feedback before complicating the plugin.

Shard Selection and Resolution

There are a couple of high level notes about the implementation before getting into installation and configuration of the plugin.  Typically when talking about database sharding there are a couple of decisions an implementation needs to make:

  1. How do we assign a shard to a new object?  (Shard Selection)
  2. How do we resolve the shard that a current object lives in? (Shard Resolution)

The choices we made for our plugin are to use what we call an Index database that holds a minimum of two tables.  One which the plugin creates is called which contains a record for every shard in the system along with a capacity and usage field.  This table is queried every time a new object, in our case User, is created within the system.  We assign the new object to the shard with the lowest usage to capacity ratio.  This allows for shards to be located on different types of hardware that should take a smaller or larger number of objects.  The other table is supplied by the application using the plugin and provides a mapping of the object to the shard to be used.  Then whenever a request begins for an object the application should query this table and retrieve the shard to use and then pass that to the plugin to switch to that database.

Using the Sharding plugin

Start by creating a new Grails application with the Sharding plugin (now at version 0.1), assuming Grails 1.2.1 is installed somewhere :


grails create-app ShardingExample

cd ShardingExample

grails install-plugin sharding

Next lte’s need to create a couple of domain classes:

grails create-domain-class UserIndex

grails create-domain-class Comment

Create some simple properties for the domain objects:

grails-app/domain/UserIndex.groovy

class UserIndex {

    String userName

    String shard

    static constraints = {
    }
}

grails-app/domain/Comment.groovy

class Comment {

    Integer userIndexId

    String comment

    static constraints = {
    }
}

As you can see UserIndex associates a userName with a shard, this allows the application to identify the owning shard for a given user and then switch to that shard.   The Comment object is just a sample piece of data we might store with a user.  The next thing to do is create a definition of the databases that will hold the applications data.  For this example there will be three databases an Index database and two Shard databases.  When using the plugin there must be exactly one Index database and as many Shard databases as necessary.  Here is the configuration file used to define the databases (obviously you may need to change your connection settings):

grails-app/conf/Shards.groovy:

index = {
  domainClass('UserIndex')
  shardNameFieldName('shard')

  name('shardINDEX')
  user('root')
  password('PASSWORD')
  driverClass('com.mysql.jdbc.Driver')
  jdbcUrl('jdbc:mysql://localhost:3306/shardINDEX')
  dialect(org.hibernate.dialect.MySQL5InnoDBDialect)
}
shards = {
  shard_01 {
    name('shard1001')
    user('root')
    password('PASSWORD')
    driverClass('com.mysql.jdbc.Driver')
    capacity(1000)
    jdbcUrl('jdbc:mysql://localhost:3306/shard1001')
  }
  shard_02 {
    name('shard1002')
    user('root')
    password('PASSWORD')
    driverClass('com.mysql.jdbc.Driver')
    capacity(1000)
    jdbcUrl('jdbc:mysql://localhost:3306/shard1002')
  }
}

Next create the three schema’s in the database (in this example we are using MySQL):

create schema shardINDEX;

create schema shard1001;

create schema shard1002;

Add a dependancy for the database driver you are using to grails-app/conf/BuildConfig.groovy dependencies section (in this case MySql):

    dependencies {
         runtime 'mysql:mysql-connector-java:5.1.5'
    }

Now create the default templates for both UserIndex and Comment:

grails generate-all UserIndex

grails generate-all Comment

Add the following closures to grails-app/controllers/UserIndexController.groovy (poor man’s login but you get the idea):

    def login = {
    	session.userName = params.userName
    	render "User logged in."
    }

    def logout = {
        session.userName = null
        render "User logged out"
    }

And modify the grails-app.controllers/controllers/CommentController.groovy (this will make use of the “logged in” users name and switch to the appropriate shard):

import UserIndex

class CommentController {
    def shardService

    static allowedMethods = [save: "POST", update: "POST", delete: "POST"]

    def index = {
        redirect(action: "list", params: params)
    }

    def list = {
    	def user = UserIndex.findByUserName(session.userName)
	shardService.changeByObject(user)

        params.max = Math.min(params.max ? params.int('max') : 10, 100)
        [commentInstanceList: Comment.list(params), commentInstanceTotal: Comment.count()]
    }

    def create = {
    	def user = UserIndex.findByUserName(session.userName)
    	shardService.changeByObject(user)

        def commentInstance = new Comment()
        commentInstance.properties = params
        return [commentInstance: commentInstance]
    }

    def save = {
    	def user = UserIndex.findByUserName(session.userName)
    	shardService.changeByObject(user)

        def commentInstance = new Comment(params)
        if (commentInstance.save(flush: true)) {
            flash.message = "${message(code: 'default.created.message', args: [message(code: 'comment.label', default: 'Comment'), commentInstance.id])}"
            redirect(action: "show", id: commentInstance.id)
        }
        else {
            render(view: "create", model: [commentInstance: commentInstance])
        }
    }

    def show = {
    	def user = UserIndex.findByUserName(session.userName)
    	shardService.changeByObject(user)

        def commentInstance = Comment.get(params.id)
        if (!commentInstance) {
            flash.message = "${message(code: 'default.not.found.message', args: [message(code: 'comment.label', default: 'Comment'), params.id])}"
            redirect(action: "list")
        }
        else {
            [commentInstance: commentInstance]
        }
    }

    def edit = {
    	def user = UserIndex.findByUserName(session.userName)
    	shardService.changeByObject(user)

        def commentInstance = Comment.get(params.id)
        if (!commentInstance) {
            flash.message = "${message(code: 'default.not.found.message', args: [message(code: 'comment.label', default: 'Comment'), params.id])}"
            redirect(action: "list")
        }
        else {
            return [commentInstance: commentInstance]
        }
    }

    def update = {
    	def user = UserIndex.findByUserName(session.userName)
    	shardService.changeByObject(user)

        def commentInstance = Comment.get(params.id)
        if (commentInstance) {
            if (params.version) {
                def version = params.version.toLong()
                if (commentInstance.version > version) {

                    commentInstance.errors.rejectValue("version", "default.optimistic.locking.failure", [message(code: 'comment.label', default: 'Comment')] as Object[], "Another user has updated this Comment while you were editing")
                    render(view: "edit", model: [commentInstance: commentInstance])
                    return
                }
            }
            commentInstance.properties = params
            if (!commentInstance.hasErrors() && commentInstance.save(flush: true)) {
                flash.message = "${message(code: 'default.updated.message', args: [message(code: 'comment.label', default: 'Comment'), commentInstance.id])}"
                redirect(action: "show", id: commentInstance.id)
            }
            else {
                render(view: "edit", model: [commentInstance: commentInstance])
            }
        }
        else {
            flash.message = "${message(code: 'default.not.found.message', args: [message(code: 'comment.label', default: 'Comment'), params.id])}"
            redirect(action: "list")
        }
    }

    def delete = {
    	def user = UserIndex.findByUserName(session.userName)
    	shardService.changeByObject(user)

        def commentInstance = Comment.get(params.id)
        if (commentInstance) {
            try {
                commentInstance.delete(flush: true)
                flash.message = "${message(code: 'default.deleted.message', args: [message(code: 'comment.label', default: 'Comment'), params.id])}"
                redirect(action: "list")
            }
            catch (org.springframework.dao.DataIntegrityViolationException e) {
                flash.message = "${message(code: 'default.not.deleted.message', args: [message(code: 'comment.label', default: 'Comment'), params.id])}"
                redirect(action: "show", id: params.id)
            }
        }
        else {
            flash.message = "${message(code: 'default.not.found.message', args: [message(code: 'comment.label', default: 'Comment'), params.id])}"
            redirect(action: "list")
        }
    }
}

Now run the application:

grails run-app

Navigate to http://localhost:8080/ShardingExample/userIndex in your favorite browser and create a couple of users (but leave the shard field empty) when you are done go to the list view and you will see the two new users assigned to different shards:

image

Now if navigate to the login action we created for one of the new users http://localhost:8080/ShardingExample/userIndex/login?userName=jrick.  Once you have gone to the login for the user, navigate to the comment controller http://localhost:8080/ShardingExample/comment and create some comments for the user:

image

You will then end up with a list that looks like this (http://localhost:8080/ShardingExample/comment/list):

image

Now switch to the second user created (http://localhost:8080/ShardingExample/userIndex/login?userName=lrick) and then navigate to the comment list view (http://localhost:8080/ShardingExample/comment/list) which is now empty:

image

Create a comment for this user:

image

And then the list view (http://localhost:8080/ShardingExample/comment/list) notice the ID field is not unique across shards:

image

Now switch back http://localhost:8080/ShardingExample/userIndex/login?userName=jrick and go to the list view http://localhost:8080/ShardingExample/comment/list and notice that the two shards are separated:

image

Example Application

The sample application described above can be obtained here.

About these ads
  1. Steve
    October 15, 2010 at 9:52 pm

    Hi Jeff,

    This looks interesting, but is it compatible with the spring security core and acl plugin?

    S.

    • jrick
      October 21, 2010 at 4:58 pm

      Hi,

      Yes I have used this with the security plugin. You will need to implement a UserDetailsService which is pretty straightforward. mine looks something like this:

      import org.codehaus.groovy.grails.plugins.springsecurity.GrailsDaoImpl
      import org.springframework.security.userdetails.UsernameNotFoundException
      import org.springframework.dao.DataAccessException
      import org.springframework.security.userdetails.UserDetailsService

      public class RezzUserDetailsService extends GrailsDaoImpl implements UserDetailsService {
      def userService
      protected def loadDomainUser(username, session) throws UsernameNotFoundException, DataAccessException {
      def user = userService.getUser(username)
      if (user == null) {
      log.error “User not found: $username”
      throw new UsernameNotFoundException(“User not found”, username)
      }
      return(user);
      }

      }

      In my case userService is actually a wrapper that does the shardService.change call

  2. June 21, 2011 at 10:11 am

    I tried this plugin with Grails 1.3.7 and I get the error message

    —————————————————————————————————————————————
    Domain class not found in grails-app/domain, trying hibernate mapped classes…
    No domain class found for name UserIndex. Please try again and enter a valid domain class name
    —————————————————————————————————————————————

    UserIndex is stored as grails-app/domain/shardexample/UserIndex.groovy

    I have followed the above mentioned tutorial properly.
    Can you please suggest me a solution.

    • jrick
      June 21, 2011 at 10:57 am

      Since the UserIndex has a namespace included in it you need to replace the line:

      domainClass(‘UserIndex’)

      with

      domainClass(‘shardexample.UserIndex’)

  3. June 23, 2011 at 9:41 am

    Thanks. I have the app running but when I create a user, the shards do not automatically get assigned to the new user.

    • jrick
      June 24, 2011 at 3:41 pm

      Ok, I found an issue with the plugin where the UserIndex is in a package. I have updated the plugin and released a new version. Please do a:

      install-plugin sharding

      In your project and recompile and you should be all set.

      Thanks

  1. August 3, 2010 at 7:29 pm
  2. August 10, 2010 at 10:19 pm
  3. January 5, 2011 at 4:25 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: