They said Pi clusters were CRAZY! (Mars 400)



Try out Twingate FREE for up to 5 users! Enable fine-grained access control to your resources: https://www.twingate.com/?utm_source=youtube&utm_medium=referral&utm_campaign=jeffGeerling

Special thanks to Ambedded for sending over a Mars 400 for testing. I only found out about them through one of the comments on an earlier video on Ceph clusters, so thank you SO much to all those who comment on these videos, you all are great!

Mentioned in this video (some links are affiliate links):

– Mars 400PRO: https://www.ambedded.com.tw/en/product/ceph-storage-appliance.html
– Mars 500: https://www.ambedded.com.tw/en/product/ceph-storage-appliance-500.html
– Mars 524: https://www.ambedded.com.tw/en/product/ceph-storage-appliance-524.html
– DeskPi Super6C: https://amzn.to/41hybrr
– Turing Pi 2: https://turingpi.com/product/turing-pi-2/
– Ambedded’s study on Ceph NVMe caching performance: https://www.ambedded.com.tw/en/news/Ambedded-news-018.html
– Ansible for DevOps – free copy: https://leanpub.com/ansible-for-devops/c/CTVMPCbEeXd3
– Backblaze Drive stats – heat: https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2023/

Support me on Patreon: https://www.patreon.com/geerlingguy
Sponsor me on GitHub: https://github.com/sponsors/geerlingguy
Merch: https://redshirtjeff.com
2nd Channel: https://www.youtube.com/c/GeerlingEngineering

#Sponsored

Contents:

00:00 – Are Pi clusters dumb?
00:47 – Hello Mars 400
01:36 – Twingate for remote access
02:40 – Exploring Mars
06:02 – An appliance, not a server
07:07 – Setting it up
09:39 – Rapid iteration on my own failure
10:30 – Mounting CephFS on a Pi
11:40 – From 1 Gbps to 10 Gbps
12:16 – Power consumption and noise
13:01 – Gotta go faster

source

43 thoughts on “They said Pi clusters were CRAZY! (Mars 400)”

  1. I do have to say, though Jeff. Who is this video for? It doesn’t seem to follow your usual audience. I see this channel as a “amazing stuff you can do with pi hardware around the home.” It seems, as of late, you are focusing on a lot of very high end hardware and some very extreme cutting edge implementations of arm cpus. If we are moving more into the enterprise from the home because your personal growth depends on it, then perhaps you should branch this type of content into a new channel.

    Reply
  2. 4:54 Kids these days with their "full-sized" 3.5" drives. Back in my day, we only had 5.25" drives in 3.25" high by 8 INCHES DEEP bays… and we liked it. 5MB should be more than enough for anyone!!

    Reply
  3. Just running 4 of my Dell R710 servers is killing me on power. My last utility bill was $600. And most of them are usually just at idle. I am afraid to power up my entire 20 server cluster. The monthly cost savings of just a few of these machines would pay for themselves in short order. The only concern would be running my own software and os. I don't want to be locked into a specific ecosystem. I will have to do some research, but it might be worth moving over to these.

    Reply
  4. Twingate doesn't inherently offer enhanced security by default. If you gain console access, such as through SSH for debugging purposes, you can easily move to other services and machines within the same Layer 2 network segments, bypassing TwinGate's access control. In contrast, I prefer Zerotier, which also supports ingress and egress filtering on a per-network basis, though it's not stateful, but it's free and can be self-hosted. NOW … I'm getting myself a MARS400 for the office – definitely some good stuff! 😉

    Reply
  5. Firstly I just want to say I love this. It is awesome to see more arm clusters. That being said, I think starting at $3,500 this feels a bit lacking. Especially the hard drive mounting seem super clunky. The other thing that kinda grinds my gears is why documentation is "liscense" based. Where you need to pay for support to get their documentation. Actual support I understand, but the manual and faq etc. seems a bit excessive.

    Reply
  6. The Mars 400 really makes for the textbook example of why industrial ARM has taken so long to take off, and why Ampere has done so well in contrast. It has no resale value, the company behind it has no real reputation to speak of and they gave it a heavily used product name, it has no refurb or reapplication opportunity, and it's running on non-distribute closed blobs with minimal oversight. It's an expensive e-waste hulk to dispose of the moment Ambedded decides to drop support in any form or fashion and by extension a ticking time bomb for any business that decides to rely on it for mission-critical tasks, not to mention a security risk. As a CSE, I wouldn't want that thing anywhere near any network, cluster, or farm I'm responsible for.

    Reply
  7. Palo alto needs to calm down with this pos bs. Ceph is good for only one thing: demonstrate how good is truenas.

    That chassy looks like my week-end project cobled together from old hw i've collected over the years. I get how you feel compelled to spam this but 3 nodes at 6k a pop is not really a proposition here.

    Reply
  8. Wow. I've introduced to the Ambedded in 2018, back then when the Ambedded's founder husband was teaching me Openshift & Ceph. He told that her wife is starting a ARM-based ceph cluster in a box. At that time, i believe it is using Cortex A7 or A53, but with almost the same rack chassis and 8 node config.
    in 2019, I've met her on the Taiwan Computex showcasing the Mars 400

    Reply
  9. I couldn't switch to a regular PC PSU for the Storinator so far that I've seen. My Storinator is the 60-drive model and it seems to have larger power requirements.

    Also I run all SSDs in one of them and bought 8 hybrid converters to do 128 2.5" drive bays. But logistically, there are many issues.

    1. How to connect all the SAS ports without expanders?
    2. How to fit everything in the case with all those wires?
    3. There are only 6 connectors for drives, but I need 8.
    4. The PSUs only have 50A on +5V (250W) each whereas I need about 15A with some headroom per 16 drives.

    Doing all this, I found some 12V to 5V converters from Corsair to do 20A each (100W). With 6 of these, I'll be in he safe zone.

    Logistically this is a pain to setup, but I'll be doing it this weekend.

    Reply
  10. Link aggregation is still worth it sometimes! even though the core of your network is only 10Gbe, having the storage linked to the switch at X2 (or higher) to the switch means that a single node on the network can only ever consume 50% (or less) of the total available throughput, it's good at least for multi user access. in your case though maybe not worth it, but just thought i'd mention it!

    Reply
  11. Interesting, but a little bigger than what I could use.

    Jeff can you do some content on half rack 1u equipment?

    I don’t know about the rest of the home lab community but I think that would be more interesting to us

    Reply
  12. Great video Jeff and looks like some interesting hardware. Who do you think would be the target audience for an machine like this? I'm not familiar with Ceph myself so don't know of it's benefits, but looking at your speeds I can't help feeling that a regular server would outperform in transfer speeds and come at a lower cost, although with higher power usage.

    Reply
  13. you can do " sudo !! " (doble exclamation mark) to repeat the previous failed command as SUDO this time. Avoids the hassle of copy/pasting the command as sudo.

    apt install abc123

    sudo !!

    produces:
    -> sudo apt install abc123

    God bless and have a nice day

    Reply
  14. Seems like a good idea but wish there was a version that seemed more appropriate for home servers…. might just get the blade if it still exists when I manage to get a new job 😅

    Reply
  15. II don’t get it. This will only impress amateurs who’ve never used standard server-level features like hotwswap PSUs. Even from the purely cosmetic standpoint, those front plates feel extremely cheap!! The high-end Dell 1U power vault servers used 15 years ago were way more sexy.

    Reply
  16. 142.41 W for 5 HDDs and an 8-Pi cluster — hmmm….

    I would've thought that it would be more power efficient than that.

    At that rate — that means that it would be on par with my dual Xeon E5-2697A v4 system which is currently consuming 727 W whilst running the ZFS hammer script (due to the recently discovered bug with OpenZFS 2.2.0).

    8 Pis @ 5 W = 40 W

    Scaling the 5 drives @ 20 W ea to 36 drives @ 20 W = 720 W + 40 W = 760 W.

    Reply

Leave a Comment