Try out Twingate FREE for up to 5 users! Enable fine-grained access control to your resources: https://www.twingate.com/?utm_source=youtube&utm_medium=referral&utm_campaign=jeffGeerling
Special thanks to Ambedded for sending over a Mars 400 for testing. I only found out about them through one of the comments on an earlier video on Ceph clusters, so thank you SO much to all those who comment on these videos, you all are great!
Mentioned in this video (some links are affiliate links):
– Mars 400PRO: https://www.ambedded.com.tw/en/product/ceph-storage-appliance.html
– Mars 500: https://www.ambedded.com.tw/en/product/ceph-storage-appliance-500.html
– Mars 524: https://www.ambedded.com.tw/en/product/ceph-storage-appliance-524.html
– DeskPi Super6C: https://amzn.to/41hybrr
– Turing Pi 2: https://turingpi.com/product/turing-pi-2/
– Ambedded’s study on Ceph NVMe caching performance: https://www.ambedded.com.tw/en/news/Ambedded-news-018.html
– Ansible for DevOps – free copy: https://leanpub.com/ansible-for-devops/c/CTVMPCbEeXd3
– Backblaze Drive stats – heat: https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2023/
Support me on Patreon: https://www.patreon.com/geerlingguy
Sponsor me on GitHub: https://github.com/sponsors/geerlingguy
Merch: https://redshirtjeff.com
2nd Channel: https://www.youtube.com/c/GeerlingEngineering
#Sponsored
Contents:
00:00 – Are Pi clusters dumb?
00:47 – Hello Mars 400
01:36 – Twingate for remote access
02:40 – Exploring Mars
06:02 – An appliance, not a server
07:07 – Setting it up
09:39 – Rapid iteration on my own failure
10:30 – Mounting CephFS on a Pi
11:40 – From 1 Gbps to 10 Gbps
12:16 – Power consumption and noise
13:01 – Gotta go faster
source
Jeff, ya dummy!
Who needs this? Raises his hand …slowly at first, and then yells “I DO!”
Jeff, Is it worth 10x the price of your original cluster?
I do have to say, though Jeff. Who is this video for? It doesn’t seem to follow your usual audience. I see this channel as a “amazing stuff you can do with pi hardware around the home.” It seems, as of late, you are focusing on a lot of very high end hardware and some very extreme cutting edge implementations of arm cpus. If we are moving more into the enterprise from the home because your personal growth depends on it, then perhaps you should branch this type of content into a new channel.
A single Intel i9 CPU consumes twice as much as this entire appliance! Can't complain about its efficiency.
4:54 Kids these days with their "full-sized" 3.5" drives. Back in my day, we only had 5.25" drives in 3.25" high by 8 INCHES DEEP bays… and we liked it. 5MB should be more than enough for anyone!!
Jeff ya dummy, no one is gonna run those…
Just running 4 of my Dell R710 servers is killing me on power. My last utility bill was $600. And most of them are usually just at idle. I am afraid to power up my entire 20 server cluster. The monthly cost savings of just a few of these machines would pay for themselves in short order. The only concern would be running my own software and os. I don't want to be locked into a specific ecosystem. I will have to do some research, but it might be worth moving over to these.
Twingate doesn't inherently offer enhanced security by default. If you gain console access, such as through SSH for debugging purposes, you can easily move to other services and machines within the same Layer 2 network segments, bypassing TwinGate's access control. In contrast, I prefer Zerotier, which also supports ingress and egress filtering on a per-network basis, though it's not stateful, but it's free and can be self-hosted. NOW … I'm getting myself a MARS400 for the office – definitely some good stuff! 😉
Ceph cluster or storage in general on a pi which does not have ecc is just nuts..
Stop making ads.
This is a really cool storage appliance for a homelab! I didn't know that I needed one until you reviewed it
And thanks for the free eBook!
Firstly I just want to say I love this. It is awesome to see more arm clusters. That being said, I think starting at $3,500 this feels a bit lacking. Especially the hard drive mounting seem super clunky. The other thing that kinda grinds my gears is why documentation is "liscense" based. Where you need to pay for support to get their documentation. Actual support I understand, but the manual and faq etc. seems a bit excessive.
I wonder how many you could stack… 20? 50?
You know its out of your price range when Ambedded just has a button to inquirer with no price given.
Why though? Rasberry Pis are so overpriced for how much of a bottleneck they are compared to actual X86 based computers.
Well, It's nice to have connections and get free stuff.
Can you use some of these clusters to crack the latest TI calculator signing keys for us? I can't afford to build one, myself.
Ok Jeff, enough with the raspberry pies now at this point these are relevant projects anymore
I want one
lol oh boy. The first couple times I thought you were saying CIF really weirdly
It's easier to keep 80W of CPU cool than it is 200W of CPU … well no shit Sherlock…
❤
The Mars 400 really makes for the textbook example of why industrial ARM has taken so long to take off, and why Ampere has done so well in contrast. It has no resale value, the company behind it has no real reputation to speak of and they gave it a heavily used product name, it has no refurb or reapplication opportunity, and it's running on non-distribute closed blobs with minimal oversight. It's an expensive e-waste hulk to dispose of the moment Ambedded decides to drop support in any form or fashion and by extension a ticking time bomb for any business that decides to rely on it for mission-critical tasks, not to mention a security risk. As a CSE, I wouldn't want that thing anywhere near any network, cluster, or farm I'm responsible for.
Palo alto needs to calm down with this pos bs. Ceph is good for only one thing: demonstrate how good is truenas.
That chassy looks like my week-end project cobled together from old hw i've collected over the years. I get how you feel compelled to spam this but 3 nodes at 6k a pop is not really a proposition here.
yes dad i need this for school
This is pretty good, the 4GB of memory per node worries me though. If it's just the storage node, maybe it's fine.
Love the concept, less excited in the licensing aspect of it but appreciate that I as a lowly home labber am not the audience for this product.
Wow. I've introduced to the Ambedded in 2018, back then when the Ambedded's founder husband was teaching me Openshift & Ceph. He told that her wife is starting a ARM-based ceph cluster in a box. At that time, i believe it is using Cortex A7 or A53, but with almost the same rack chassis and 8 node config.
in 2019, I've met her on the Taiwan Computex showcasing the Mars 400
I couldn't switch to a regular PC PSU for the Storinator so far that I've seen. My Storinator is the 60-drive model and it seems to have larger power requirements.
Also I run all SSDs in one of them and bought 8 hybrid converters to do 128 2.5" drive bays. But logistically, there are many issues.
1. How to connect all the SAS ports without expanders?
2. How to fit everything in the case with all those wires?
3. There are only 6 connectors for drives, but I need 8.
4. The PSUs only have 50A on +5V (250W) each whereas I need about 15A with some headroom per 16 drives.
Doing all this, I found some 12V to 5V converters from Corsair to do 20A each (100W). With 6 of these, I'll be in he safe zone.
Logistically this is a pain to setup, but I'll be doing it this weekend.
Link aggregation is still worth it sometimes! even though the core of your network is only 10Gbe, having the storage linked to the switch at X2 (or higher) to the switch means that a single node on the network can only ever consume 50% (or less) of the total available throughput, it's good at least for multi user access. in your case though maybe not worth it, but just thought i'd mention it!
Next up, you're going to link a raspberry pi cluster to a cluster of raspberry pi clusters over 10g to create a megacluster…. or a clusterf**k…. 😛
Interesting, but a little bigger than what I could use.
Jeff can you do some content on half rack 1u equipment?
I don’t know about the rest of the home lab community but I think that would be more interesting to us
Great video Jeff and looks like some interesting hardware. Who do you think would be the target audience for an machine like this? I'm not familiar with Ceph myself so don't know of it's benefits, but looking at your speeds I can't help feeling that a regular server would outperform in transfer speeds and come at a lower cost, although with higher power usage.
you can do " sudo !! " (doble exclamation mark) to repeat the previous failed command as SUDO this time. Avoids the hassle of copy/pasting the command as sudo.
apt install abc123
sudo !!
produces:
-> sudo apt install abc123
God bless and have a nice day
Love the content, but I cant play like you, cant afford all these pi's 😛
Tingate has no client for AmigaOS!!!
Seems like a good idea but wish there was a version that seemed more appropriate for home servers…. might just get the blade if it still exists when I manage to get a new job 😅
Nice appliance, Jeff! Do I understand it the right way: Ceph is a storage area networking (SAN) solution implemented completely in software?!
Sick video Geff very nice lay out wonderful design
II don’t get it. This will only impress amateurs who’ve never used standard server-level features like hotwswap PSUs. Even from the purely cosmetic standpoint, those front plates feel extremely cheap!! The high-end Dell 1U power vault servers used 15 years ago were way more sexy.
142.41 W for 5 HDDs and an 8-Pi cluster — hmmm….
I would've thought that it would be more power efficient than that.
At that rate — that means that it would be on par with my dual Xeon E5-2697A v4 system which is currently consuming 727 W whilst running the ZFS hammer script (due to the recently discovered bug with OpenZFS 2.2.0).
8 Pis @ 5 W = 40 W
Scaling the 5 drives @ 20 W ea to 36 drives @ 20 W = 720 W + 40 W = 760 W.
Hello jeff, can you explain how to check the computational capacity / calculation per second. Is this good benchmark to select better system for application or web server.