Planet Ceph

Aggregated news from external sources

  • January 2, 2014
    Zero To Hero Guide : : For CEPH CLUSTER PLANNING

    What it is all about :If you think or discuss about Ceph , the most common question strike to your mind is “What Hardware Should I Select For My CEPH Storage Cluster ?” and yes if you really thought of this question in your mind , congratulations you s…

  • December 5, 2013
    Ceph + OpenStack :: Part-5

    OpenStack Instance boot from Ceph VolumeFor a list of images to choose from to create a bootable volume[root@rdo /(keystone_admin)]# nova image-list+————————————–+—————————–+——–+——–+| ID …

  • December 5, 2013
    Ceph + OpenStack :: Part-4

    Testing OpenStack Glance + RBDTo allow glance to keep images on ceph RBD volume , edit /etc/glance/glance-api.confdefault_store = rbd# ============ RBD Store Options =============================# Ceph configuration file path# If using cephx …

  • December 5, 2013
    Ceph + OpenStack :: Part-3

    Testing OpenStack Cinder + RBDCreating a cinder volume provided by ceph backend[root@rdo /]#[root@rdo /]# cinder create –display-name cinder-ceph-vol1 –display-description “first cinder volume on ceph backend” 10+———————+—–…

  • December 5, 2013
    Ceph + OpenStack :: Part-2

    Configuring OpenStack

    Two parts of openstack integrates with Ceph’s block devices:

    • Images: OpenStack Glance manages images for VMs.
    • Volumes: Volumes are block devices. OpenStack uses volumes to boot VMs, or to attach volumes to running VMs. OpenStack manages volumes using Cinder services.
      • Create pools for volumes and images:
    ceph osd pool create volumes 128
    ceph osd pool create images 128
    • Configure OpenStack Ceph Client – The nodes running glance-api and cinder-volume act as Ceph clients. Each requires the ceph.conf file:
    [root@ceph-mon1 ceph]# scp ceph.conf openstack:/etc/ceph
    • Installing ceph client packages on openstack node
      • First install Python bindings for librbd
    yum install python-ceph
      • Install ceph
    [root@ceph-mon1 ceph]# ceph-deploy install openstack
    • Setup Ceph Client Authentication for both pools along with keyrings
      • Create a new user for Nova/Cinder and Glance.
    ceph auth get-or-create client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images'
    ceph auth get-or-create client.images mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' 
      • Add these keyrings to glance-api and cinder-volume nodes.
    ceph auth get-or-create client.images | ssh openstack tee /etc/ceph/ceph.client.images.keyring
    ssh openstack chown glance:glance /etc/ceph/ceph.client.images.keyring
    ceph auth get-or-create client.volumes | ssh openstack tee /etc/ceph/ceph.client.volumes.keyring
    ssh openstack chown cinder:cinder /etc/ceph/ceph.client.volumes.keyring
      • Hosts running nova-compute do not need the keyring. Instead, they store the secret key in libvirt. To create libvirt secret key you will need key from client.volumes.key
    ceph auth get-key client.volumes | ssh openstack tee client.volumes.key
      • on the compute nodes, add the secret key to libvirt create a secret.xml file
    cat > secret.xml < <EOF
    <secret ephemeral='no' private='no'>
    <usage type='ceph'>
    <name>client.volumes secret</name>
      • Generate secret from created secret.xml file , make a note of uuid of secret output
    # virsh secret-define --file secret.xml 
      • Set libvirt secret using above key
    # virsh secret-set-value --secret {uuid of secret} --base64 $(cat client.volumes.key) && rm client.volumes.key secret.xml
    • Configure OpenStack-Glance to use CEPH
      • Glance can use multiple back ends to store images. To use Ceph block devices by default, edit /etc/glance/glance-api.conf and add:
      • If want to enable copy-on-write cloning of images into volumes, also add:
    • Configure OpenStack – Cinder to use CEPH 
      • OpenStack requires a driver to interact with Ceph block devices. You must specify the pool name for the block device. On your OpenStack node, edit/etc/cinder/cinder.conf by adding:
    • If you’re using cephx authentication also configure the user and uuid of the secret you added to libvirt earlier:
    rbd_secret_uuid={uuid of secret}
    • Restart Openstack
    service glance-api restart
    service nova-compute restart
    service cinder-volume restart
    • Once OpenStack is up and running, you should be able to create a volume with OpenStack on a Ceph block device.
    • NOTE : Make sure /etc/ceph/ceph.conf file have sufficient rights to be ready by cinder and glance users.

    Please Follow Ceph + OpenStack :: Part-3 for next step in installation

  • December 5, 2013
    Ceph + OpenStack :: Part-1

    Ceph & OpenStack IntegrationWe can use Ceph Block Device with openstack through libvirt, which configures the QEMU interface tolibrbd. To use Ceph Block Devices with openstack , we must install QEMU, libvirt, and&…

  • December 5, 2013
    Ceph Installation :: Part-3

    Creating Block Device from CephFrom monitor node , use ceph-deploy to install Ceph on your ceph-client1 node.[root@ceph-mon1 ~]# ceph-deploy install ceph-client1[ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy install ceph-client1[ceph_dep…

  • December 5, 2013
    Ceph Installation :: Part-2

    CEPH Storage ClusterInstalling Ceph Deploy ( ceph-mon1 )Update your repository and install ceph-deploy on ceph-mon1 node[ceph@ceph-mon1 ~]$ sudo yum update && sudo yum install ceph-deployLoaded plugins: downloadonly, fastestmirror, securityLoad…

  • December 5, 2013
    Ceph Installation :: Part-1
    Ceph Installation Step by Step

    This quick start setup helps to deploy ceph with 3 Monitors and 2 OSD nodes with 4 OSD each node. In this we are using commodity hardware running CentOS 6.4

    Ceph-mon1 : First Monitor + Ceph-deploy machine (will be used to deploy ceph to other nodes )

    Ceph-mon2 : Second Monitor ( for monitor quorum )

    Ceph-mon3 : Third Monitor ( for monitor quorum )

    Ceph-node1 : OSD node 1 with 10G X 1 for OS , 440G X 4 for 4 OSD

    Ceph-node2 : OSD node 2 with 10G X 1 for OS , 440G X 4 for 4 OSD

    Ceph-Deploy Version is 1.3.2 , Ceph Version 0.67.4 ( Dumpling )

    Preflight Checklist 

    All the Ceph Nodes may require some basic configuration work prior to deploying a Ceph Storage Cluster.

    CEPH node setup

    • Create a user on each Ceph Node.
    sudo useradd -d /home/ceph -m ceph
    sudo passwd ceph
    • Add root privileges for the user on each Ceph Node.
    echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers
    sudo chmod 0440 /etc/sudoers
    • Configure your ceph-deploy node ( ceph-mon1) with password-less SSH access to each Ceph Node. Leave the passphrase empty , repeat this step for CEPH and ROOT users.
    ceph@ceph-admin:~ [ceph@ceph-admin ~]$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/ceph/.ssh/id_rsa): yes
    Created directory '/home/ceph/.ssh'.
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /home/ceph/.ssh/id_rsa.
    Your public key has been saved in /home/ceph/.ssh/
    The key fingerprint is:
    The key's randomart image is:
    +--[ RSA 2048]----+
    | |
    | E. o |
    | .. oo . |
    | . .+..o |
    | . .o.S. |
    | . + |
    | . o. o |
    | ++ .. . |
    | ..+*+++ |

    • Copy the key to each Ceph Node. ( Repeat this step for ceph and root users )
    [ceph@ceph-mon1 ~]$ ssh-copy-id ceph@ceph-node2
    The authenticity of host 'ceph-node2 (' can't be established.
    RSA key fingerprint is ac:31:6f:e7:bb:ed:f1:18:9e:6e:42:cc:48:74:8e:7b.
    Are you sure you want to continue connecting (yes/no)? y
    Please type 'yes' or 'no': yes
    Warning: Permanently added 'ceph-node2,' (RSA) to the list of known hosts.
    ceph@ceph-node2's password:
    Now try logging into the machine, with "ssh 'ceph@ceph-node2'", and check in: .ssh/authorized_keys
    to make sure we haven't added extra keys that you weren't expecting.
    [ceph@ceph-mon1 ~]$
    • Ensure connectivity using ping with hostnames , for convenience we have used local host file , update host file of every node with details of other nodes. PS : Use of DNS is recommended
    • Packages are cryptographically signed with the release.asc key. Add release key to your system’s list of trusted keys to avoid a security warning:
    sudo rpm --import ';a=blob_plain;f=keys/release.asc'
    • Ceph may require additional additional third party libraries. To add the EPEL repository, execute the following:
    su -c 'rpm -Uvh'
    sudo yum install snappy leveldb gdisk python-argparse gperftools-libs
    • Installing Release packages , Dumpling is the most recent stable release of Ceph. ( by the time i am creating this wiki )
    su -c 'rpm -Uvh'
    • Adding ceph to YUM , create repository file for ceph /etc/yum.repos.d/ceph.repo
    name=Ceph packages for $basearch

    name=Ceph noarch packages

    name=Ceph source packages
    • For best results, create directories on your nodes for maintaining the configuration generated by ceph . These should get auto created by ceph however in may case it gave me problems. So creating manually.
    mkdir -p /etc/ceph /var/lib/ceph/{tmp,mon,mds,bootstrap-osd} /var/log/ceph
    • By default, daemons bind to ports within the 6800:7100 range. You may configure this range at your discretion. Before configuring your IP tables, check the default iptables configuration. ::ports within the 6800:7100 range. You may configure this range at your discretion. Since we are performing test deployment we can disable iptables on ceph nodes . For moving to production this need to be attended.

    Please Follow Ceph Installation :: Part-2 for next step in installation

  • December 5, 2013
    Ceph Storage :: Introduction

    What is CEPH

    Ceph is an open-source, massively scalable, software-defined storage system which provides object, block and file system storage from a single clustered platform. Ceph’s main goals is to be completely distributed without a single point of failure, scalable to the exabyte level, and freely-available. The data is replicated, making it fault tolerant. Ceph software runs on commodity hardware. The system is designed to be both self-healing and self-managing and self awesome 🙂

    CEPH Internals

    • OSD: A Object Storage Daemon (OSD) stores data, handles data replication, recovery, backfilling, rebalancing, and provides some monitoring information to Ceph Monitors by checking other Ceph OSD Daemons for a heartbeat. A Ceph Storage Cluster requires at least two Ceph OSD Daemons to achieve an active + clean state when the cluster makes two copies of your data . 
    • Monitor: A Ceph Monitor maintains maps of the cluster state, including the monitor map, the OSD map, the Placement Group (PG) map, and the CRUSH map. Ceph maintains a history (called an “epoch”) of each state change in the Monitors, Ceph OSD Daemons, and PGs.
    • MDS: A Ceph Metadata Server (MDS) stores metadata on behalf of the Ceph Filesystem . Ceph Metadata Servers make it feasible for POSIX file system users to execute basic commands like ls, find, etc. without placing an enormous burden on the Ceph Storage Cluster.
    Note :: Please use and other official InkTank and ceph community resources as a primary source of information on ceph . This entire blog is an attempt help beginners in setting up ceph cluster and sharing my troubleshooting with you.