It’s a bit old, it’s a Ryzen 3500U (laptop from 2017/2018 ish), so at the older end of your range. I’m still maxing my internet speed, it just kicks the fan on.
I haven’t checked my desktop (Ryzen 5600) because I don’t hear the fan when the CPU gets pegged (never thought to check), but maybe I will the next time I download a game.
that’s definitely not in the range of like, super old cpus, but it’s also not super fast either. Modern cpus should be like 20-30% faster i think, in single core, which is what compression uses.
Realistically compression should be as aggressive as possible, because it saves bandwidth, and it’s basically a free resource,
Sure, and I have no issues with compression or encryption on the device. In fact, I used full-disk encryption for a few years and had zero issues, and I’ve done plenty of compression and decompression as well (in fact, I think my new OS uses compression at the FS layer). Most of the time, that stuff is completely a non-issue since CPUs have special instructions for common algorithms, I think they’re just using something fancy that doesn’t have hardware acceleration on my CPU or something.
I’m planning to replace it, but it still works well for what I need it for: Minecraft for the kids, Rust Dev for me, and indie games and videos every so often. I’m on integrated graphics and it’s still holding up well.
it’s my understanding that on disk compression is different from networked compression, usually networked compression uses Gzip iirc, where as on disk tends to use something like LZ, file downloads are generally less important than a file system, so you can trivially get away with costly and expensive compression.
Yeah, because you’re optimizing for different things.
server - maximize requests handled, so you need to balance CPU usage and bandwidth
disk - more bottlenecked by disk speed and capacity, so spending more time compressing is fine, provided decompression is fast (reads should exceed writes)
the server isn’t live compressing it, it’s pre compressed binaries being shipped hundreds of thousands of times over, in most cases. Compression is primarily to minimize bandwidth (and also speed up downloads, since the network speed is usually the bottleneck) you can either cache the compressed files, or do a gated download, based on decompression speed.
Usually, most disks are faster than any network connection available, so it’s pretty hard to hit that bottleneck these days. HDDs included, unless you’re using SMR drives in a specific use case, and definitely not an SSD ever.
Although on the FS side, you would optimize for minimum latency, latency really fucks up a file system, that and corrupt data, so if you can ensure a minimal latency impact, as well as a reliable compression/decompression algorithm, you can get a decent trade off of some size optimization, for a bit of latency, and CPU time.
Whether or not fs based compression is good, i’m not quite sure yet, i’m bigger on de-duplication personally.
It is for generated data, like a JSON API. Static content is often pre-compressed though, since there’s no reason to do that every request if it can be done once. Compression formats is largely limited to whatever the client supports, and gzip works pretty much everywhere, so it’s generally preferred.
At least that’s my understanding. Every project I’ve worked on has a pretty small userbase, so something like 50-100 concurrent users (mostly B2B projects), meaning we didn’t have the same problems as something like a CDN might have.
I’m not really sure how latency is related for FS operations. Are you saying if the CPU is lagging behind the read speed, it’ll mess up the stream? Or are you saying something else? I’m not an expert on filesystems.
It is for generated data, like a JSON API. Static content is often pre-compressed though, since there’s no reason to do that every request if it can be done once. Compression formats is largely limited to whatever the client supports, and gzip works pretty much everywhere, so it’s generally preferred.
yeah, if the page is dynamically generated, you would likely live compress it, but it would also be so little data that the cpu overhead would be pretty minimal, and you could still cache the compressed data if you tried hard enough. For something like steam where you’re effectively shipping and unpacking a 50-200gb zip file, there’s no reason not to statically compress it.
I’m not really sure how latency is related for FS operations. Are you saying if the CPU is lagging behind the read speed, it’ll mess up the stream? Or are you saying something else? I’m not an expert on filesystems.
it’s important because the entire system is based on a filesystem, if you’re doing regular calls to a drive, in high quantity latency is going to start being a bottleneck pretty quickly. Obviously it doesn’t matter much on certain things, but after a certain point it can start being problematic. There’s practically no chance of corruption or anything like that, unless you have a dysfunctional compression/decompression algorithm, but you would likely expect system performance to be noticeably slower in disk benchmarks specifically. Especially if you’re running really fast drives like gen 4 NVME ssds. Ideally, it shouldn’t be a huge thing, but it’s something to consider if you care about it.
There are two primary things to consider when making a functional file system, one is atomicity, because you want to be able to write data, and be certain that it was written correctly (to prevent corruption) and you want to maximize performance. File IO is always one of the slowest forms of interaction, it’s why you have ram, yes, and it’s why your CPU has cache, but optimizing drive performance in software, is still free performance gains. That’s an improvement that can make heavy read/write operations faster, more efficient, and more scalable. Which in the world of super fast modern NVMEs, is something we’re all thankful for. If you remember the switch from spinning rust, to solid state storage for operating systems, you’ll see a similar improvement. HDDs necessarily have really bad random IOPs performance, they literally physically find the data on the disk, and read it back, it’s mechanically limited, this increases latency considerably. And SSDs don’t have this problem, because they’re a matrix of registers, so you can get MASSIVELY uplifted random IOPs performance from an SSD compared to a hdd. And that’s still true today.
It’s a bit old, it’s a Ryzen 3500U (laptop from 2017/2018 ish), so at the older end of your range. I’m still maxing my internet speed, it just kicks the fan on.
I haven’t checked my desktop (Ryzen 5600) because I don’t hear the fan when the CPU gets pegged (never thought to check), but maybe I will the next time I download a game.
that’s definitely not in the range of like, super old cpus, but it’s also not super fast either. Modern cpus should be like 20-30% faster i think, in single core, which is what compression uses.
Realistically compression should be as aggressive as possible, because it saves bandwidth, and it’s basically a free resource,
Sure, and I have no issues with compression or encryption on the device. In fact, I used full-disk encryption for a few years and had zero issues, and I’ve done plenty of compression and decompression as well (in fact, I think my new OS uses compression at the FS layer). Most of the time, that stuff is completely a non-issue since CPUs have special instructions for common algorithms, I think they’re just using something fancy that doesn’t have hardware acceleration on my CPU or something.
I’m planning to replace it, but it still works well for what I need it for: Minecraft for the kids, Rust Dev for me, and indie games and videos every so often. I’m on integrated graphics and it’s still holding up well.
it’s my understanding that on disk compression is different from networked compression, usually networked compression uses Gzip iirc, where as on disk tends to use something like LZ, file downloads are generally less important than a file system, so you can trivially get away with costly and expensive compression.
Yeah, because you’re optimizing for different things.
the server isn’t live compressing it, it’s pre compressed binaries being shipped hundreds of thousands of times over, in most cases. Compression is primarily to minimize bandwidth (and also speed up downloads, since the network speed is usually the bottleneck) you can either cache the compressed files, or do a gated download, based on decompression speed.
Usually, most disks are faster than any network connection available, so it’s pretty hard to hit that bottleneck these days. HDDs included, unless you’re using SMR drives in a specific use case, and definitely not an SSD ever.
Although on the FS side, you would optimize for minimum latency, latency really fucks up a file system, that and corrupt data, so if you can ensure a minimal latency impact, as well as a reliable compression/decompression algorithm, you can get a decent trade off of some size optimization, for a bit of latency, and CPU time.
Whether or not fs based compression is good, i’m not quite sure yet, i’m bigger on de-duplication personally.
It is for generated data, like a JSON API. Static content is often pre-compressed though, since there’s no reason to do that every request if it can be done once. Compression formats is largely limited to whatever the client supports, and gzip works pretty much everywhere, so it’s generally preferred.
At least that’s my understanding. Every project I’ve worked on has a pretty small userbase, so something like 50-100 concurrent users (mostly B2B projects), meaning we didn’t have the same problems as something like a CDN might have.
I’m not really sure how latency is related for FS operations. Are you saying if the CPU is lagging behind the read speed, it’ll mess up the stream? Or are you saying something else? I’m not an expert on filesystems.
yeah, if the page is dynamically generated, you would likely live compress it, but it would also be so little data that the cpu overhead would be pretty minimal, and you could still cache the compressed data if you tried hard enough. For something like steam where you’re effectively shipping and unpacking a 50-200gb zip file, there’s no reason not to statically compress it.
it’s important because the entire system is based on a filesystem, if you’re doing regular calls to a drive, in high quantity latency is going to start being a bottleneck pretty quickly. Obviously it doesn’t matter much on certain things, but after a certain point it can start being problematic. There’s practically no chance of corruption or anything like that, unless you have a dysfunctional compression/decompression algorithm, but you would likely expect system performance to be noticeably slower in disk benchmarks specifically. Especially if you’re running really fast drives like gen 4 NVME ssds. Ideally, it shouldn’t be a huge thing, but it’s something to consider if you care about it.
There are two primary things to consider when making a functional file system, one is atomicity, because you want to be able to write data, and be certain that it was written correctly (to prevent corruption) and you want to maximize performance. File IO is always one of the slowest forms of interaction, it’s why you have ram, yes, and it’s why your CPU has cache, but optimizing drive performance in software, is still free performance gains. That’s an improvement that can make heavy read/write operations faster, more efficient, and more scalable. Which in the world of super fast modern NVMEs, is something we’re all thankful for. If you remember the switch from spinning rust, to solid state storage for operating systems, you’ll see a similar improvement. HDDs necessarily have really bad random IOPs performance, they literally physically find the data on the disk, and read it back, it’s mechanically limited, this increases latency considerably. And SSDs don’t have this problem, because they’re a matrix of registers, so you can get MASSIVELY uplifted random IOPs performance from an SSD compared to a hdd. And that’s still true today.