Amazon Linux instance running beanstalkd

amazon ec2amazon-linuxamazon-web-services

What's the best way to run beanstalkd on an instance of Amazon Linux in EC2? Beanstalkd doesn't exist in the yum repository and the upstart script included with beanstalkd doesn't seem to work out of the box. I'm wondering if someone has already solved this problem, or if I need to go and muck around with a custom EC2 user-data script + beanstalkd upstart script.

Best Answer

I ended up writing a user data script that runs when an ec2 instance is created. The script pulls down the latest source for beanstalkd, compiles, and installs the binary. The script uses a modified upstart script to control the lifecycle of the beanstalkd process.

Here's the user data script:

#!/bin/bash -ex
set -e -x

# Log output
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# Update yum first
yum groupinstall "Development Tools" -y 
yum update -y

# Download /etc/default/beanstalkd
curl -J -O https://gist.github.com/phaitour/6482469/download
tar -zxvf *.tar.gz
/bin/cp -f gist*/beanstalkd /etc/default/ 
ln -s /etc/default/beanstalkd /etc/beanstalkd.conf 
rm -rf gist*

# Download /etc/init/beanstalkd.conf
curl -J -O https://gist.github.com/phaitour/6482467/download
tar -zxvf *.tar.gz
/bin/cp -f gist*/beanstalkd.conf /etc/init/
rm -rf gist*

# Create beanstalkd user
/usr/sbin/useradd -c "beanstalk user" -s /bin/false -r -m -d /var/lib/beanstalkd/ beanstalkd

# Compile latest version of beanstalkd
wget https://github.com/kr/beanstalkd/archive/v1.9.tar.gz
tar -zxvf v1.9.tar.gz
cd beanstalkd-1.9
make
mv beanstalkd /usr/bin/beanstalkd-1.9
cd /usr/bin
rm -rf beanstalkd
ln -s beanstalkd-1.9 beanstalkd

# Start beanstalks with upstart
initctl reload-configuration
initctl start beanstalkd

The two scripts it's downloading from gist is the upstart beanstalkd script and a configuration file which sets the options used when running the program.

Here's the upstart beanstalkd.conf script:

description "simple, fast work queue"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 5 2

expect fork

script
    . /etc/default/beanstalkd
    exec su -c "exec /usr/bin/beanstalkd $BEANSTALKD_OPTIONS"
end script

Here's the configuration file copied into /etc/default/beanstalkd:

# Set some constants to be used to construct BEANSTALKD_OPTIONS

BEANSTALKD_ADDR=0.0.0.0
BEANSTALKD_PORT=11300
BEANSTALKD_USER=beanstalkd
BEANSTALKD_BINLOG_DIR=/var/lib/beanstalkd
BEANSTALKD_BINLOG_FSYNC_PERIOD=0



# Create the actual BEANSTALKD_OPTIONS string
# Copied from https://gist.github.com/shiki/515422

BEANSTALKD_OPTIONS="-l ${BEANSTALKD_ADDR} -p ${BEANSTALKD_PORT} -u ${BEANSTALKD_USER}"
if [ "${BEANSTALKD_MAX_JOB_SIZE}" != ""  ]; then
    BEANSTALKD_OPTIONS="${BEANSTALKD_OPTIONS} -z ${BEANSTALKD_MAX_JOB_SIZE}"
fi

if [ "${BEANSTALKD_BINLOG_DIR}" != "" ]; then
    if [ ! -d "${BEANSTALKD_BINLOG_DIR}" ]; then
        echo "Creating binlog directory (${BEANSTALKD_BINLOG_DIR})"
        mkdir -p ${BEANSTALKD_BINLOG_DIR} && chown ${BEANSTALKD_USER}:${BEANSTALKD_USER} ${BEANSTALKD_BINLOG_DIR}
    fi
    BEANSTALKD_OPTIONS="${BEANSTALKD_OPTIONS} -b ${BEANSTALKD_BINLOG_DIR}"
    if [ "${BEANSTALKD_BINLOG_FSYNC_PERIOD}" != "" ]; then
        BEANSTALKD_OPTIONS="${BEANSTALKD_OPTIONS} -f ${BEANSTALKD_BINLOG_FSYNC_PERIOD}"
    else
        BEANSTALKD_OPTIONS="${BEANSTALKD_OPTIONS} -F"
    fi
    if [ "${BEANSTALKD_BINLOG_SIZE}" != "" ]; then
        BEANSTALKD_OPTIONS="${BEANSTALKD_OPTIONS} -s ${BEANSTALKD_BINLOG_SIZE}"
    fi
fi
Related Topic