Cloud solution based on Nexenta with KVM and CloudStack

cloud storageIf you are looking around for alternative storage solution which can compete with big iron providers such as IBM EMC NetApp HitachiDS or how to build a Cloud solution using commercial and opensource components –Β  you should watch webinar lead by @Marek Lubinski – Senior Engineer @LeaseWEB.

Marek presented Cloud solution build on:

  • Nexenta storage HA clusters
  • Supermicro NFS heads
  • Apache Cloudstack as a cloud framework
  • KVM and vSphere hypervisors – target is to have only KVM
  • HP proliant and Dell PowerEdge servers

Marek explained in detail target set up,Β  caveats, challenges and solution to problems which were faced during implementation and migration to new platform. I highly recommend to watch presentation from the beginning to the end.

[box type=”download”] WebeX recorded presentation

Presentation in PDF format[/box]

Artur Krzywdzinski

Artur is Consulting Architect at Nutanix. He has been using, designing and deploying VMware based solutions since 2005 and Microsoft since 2012. He specialize in designing and implementing private and hybrid cloud solution based on VMware and Microsoft software stacks, datacenter migrations and transformation, disaster avoidance. Artur holds VMware Certified Design Expert certification (VCDX #077).

  • Marek Lubinski

    Thanks Artur for posting, I hope that people that follow your blog will find it useful, and won’t be scared to used NexentaStor πŸ™‚

    • artur_ka

      no problem, always happy to post interested setups and configuration. BTW, next time do forget to inform BEFORE live presentation πŸ˜›

      • Marek Lubinski

        yes yes my friend, I am very sorry, and next time will send you information before it happens πŸ˜€

  • nOon

    Good presentation, but i just don’t get why you start by using raidz2 (in your first configuration it’s like you only have the performance of 11 disks), the performance in zfs when in come to random workload is define by the number of vdev.

    With zfs and random workload you have only one way, mirroring vdev. And by the if you make the test i’m sure with the mirrored vdev 2 zeus ram is sufficient.

    How many disk did you have on your new configuration? Because 12k Iops it’s not that much

    P.S:i don’t like the ocz ssd i think you will have a lot of trouble with that, ocz have really bad reputation

    @Marek: for me today zfs is clearly the best file system but i like this post because it explain than yes zfs is great but you still have to follow some rules t use it

  • Marek Lubinski

    n00n, thanks for your reply. You are right (but partially). We started with raidz2 and we had indeed performance of 11 disks, BUT that was write performance. That means that when writing, we were able to push around 1100 raw iops in writes, but still around raw 7-8k read iops because in vdevs you get as many write iops are single vdev, but as many read iops as you have spindles in (of course combined). And this setup was just perfectly sized for what we estimated in beginning (based on our input). Really, till we moved 1200 vm’s to this setup, this was flying like hell πŸ™‚ but real life showed, that our workloads were completely different (more writes than reads).
    2 zeusram disks were basically not sufficient for us – we easily maxed them out, therefore we had to put one more to system (per zpool).
    Well in current setup we have over 200 disks, and i can’t agree that 12k iops are not much, because I ran this test via 1Gbit NFS line, so there are limits. If I would run it over 10Gbit, I would get way more from this one vmpool (and bear in mind that there is another pool for workloads). Result is veeery good.
    In terms of OCZ, I can’t say bad word as for now – only one drive failed in beginning, but after replacing it it’s fine.
    And answering your last sentence – yes, correct – but this really applies to ALL vendors. If you buy $$$$$ NetApp box and will start adding single disks to aggregates then your performance will suck big time even though you have hundreds of disks inside.
    the same goes for other brands name it EMC, IBM and other.
    You just need to do right sizing first (MOST IMPORTANT), rest is brand πŸ˜€
    People are often sold big names with shitty configuration (sata disks or low numbers) and got their dreams of high performance boxes because of the name – that’s wrong πŸ˜€
    Thank you again for your comment πŸ™‚

    • nOon

      Ok sorry i didn’t see you make the test in 1 Gb line. In that case i can understant the Iops limit.

      Yes it’s really weird you have more write than read but it shows that every workload is totally different and you have to study to be sure of your design.

      Hum you maxed the zeus ram (i mean it’s 100k writes and 800 Mo/s througput).

      But it seems you have really a good configuration by now.

      Just hope one thinks for the future PNFS

      • Marek Lubinski

        yep, correct we had in beginning mirrored zil, and those 800MB/s were maxed, so we split this zil across 2 spanned zeusram which was also almost maxed out, that’s why now we have 3 per pool.

        yeah right now configuration is really perfect. It really outperforms NetApp setup which we also have in terms of performance that can be achieved from that setup.

        Anyway thanks for constructive feedback and watching my webinar/presentation.

        • nOon

          Wow i just think about your answer and you are telling me than with 200 disks you max more than 4 zeus ram? So you have more than 1.6 Go/s in write througput??? And i think you have more than that in read througput.
          It’s just impressive. Can you have an idea of how many Iops (average) did you have on the array?
          P.S: i think i will take you in example to explain than nfs is more than valuable in a virtualized environment to my collegue (they all think FC, i’m the only nfs fan)

          • Marek Lubinski

            well iops differ, but we do lots of them πŸ™‚ and our iops are quite big in size, therefore we max out zeusrams. initial setup (from presentation) with raidz2 was designed to do 40k iops with 4-8k iosize, and we maxed it out within 1 week πŸ˜€
            Yes, your situation (about fc myth) is pretty common, and people just are afraid of IP based systems. We are only IP based (previously only NetApp customer), right now multivendor (as it states on my linkedin profile), and we just love NFS. And you can tell your colleagues, that if you size your backend correctly, then it will perform the same as FC. NetApp even made some comparisons between 1Gbit NFS vs 1Gbit iscsi vs 4Gbit FC and results were awesone πŸ™‚ later on of course the same comparison against 10Gbit NFS/iscsi – results outperformed FC 8Gbit.
            just use low-latency 10gbit switches, 10gbit hba’s in hosts and proper storage (like our final design) and it will fly like hell πŸ™‚