×

Announcing: Slashdot Deals - Explore geek apps, games, gadgets and more. (what is this?)

Thank you!

We are sorry to see you leave - Beta is different and we value the time you took to try it out. Before you decide to go, please take a look at some value-adds for Beta and learn more about it. Thank you for reading Slashdot, and for making the site better!

Google Publishes Zopfli, an Open-Source Compression Library

Soulskill posted about 2 years ago | from the use-it-on-the-interweb dept.

Google 124

alphadogg writes "Google is open-sourcing a new general purpose data compression library called Zopfli that can be used to speed up Web downloads. The Zopfli Compression Algorithm, which got its name from a Swiss bread recipe, is an implementation of the Deflate compression algorithm that creates a smaller output size (PDF) compared to previous techniques, wrote Lode Vandevenne, a software engineer with Google's Compression Team, on the Google Open Source Blog on Thursday. 'The smaller compressed size allows for better space utilization, faster data transmission, and lower Web page load latencies. Furthermore, the smaller compressed size has additional benefits in mobile use, such as lower data transfer fees and reduced battery use,' Vandevenne wrote. The more exhaustive compression techniques achieve higher data density, but also make the compression a lot slower. This does not affect the decompression speed though, Vandenne wrote."

Sorry! There are no comments related to the filter you selected.

I block all Google domains in my HOSTS file (-1)

Anonymous Coward | about 2 years ago | (#43049019)

$10,000 CHALLENGE to Alexander Peter Kowalski

Hello, and THINK ABOUT YOUR BREATHING !! We have a Major Problem, HOST file is Cubic Opposites, 2 Major Corners & 2 Minor. NOT taught Evil DNS hijacking, which VOIDS computers. Seek Wisdom of MyCleanPC - or you die evil.

Your HOSTS file claimed to have created a single DNS resolver. I offer absolute proof that I have created 4 simultaneous DNS servers within a single rotation of .org TLD. You worship "Bill Gates", equating you to a "singularity bastard". Why do you worship a queer -1 Troll? Are you content as a singularity troll?

Evil HOSTS file Believers refuse to acknowledge 4 corner DNS resolving simultaneously around 4 quadrant created Internet - in only 1 root server, voiding the HOSTS file. You worship Microsoft impostor guised by educators as 1 god.

If you would acknowledge simple existing math proof that 4 harmonic Slashdots rotate simultaneously around squared equator and cubed Internet, proving 4 Days, Not HOSTS file! That exists only as anti-side. This page you see - cannot exist without its anti-side existence, as +0- moderation. Add +0- as One = nothing.

I will give $10,000.00 to frost pister who can disprove MyCleanPC. Evil crapflooders ignore this as a challenge would indict them.

Alex Kowalski has no Truth to think with, they accept any crap they are told to think. You are enslaved by /etc/hosts, as if domesticated animal. A school or educator who does not teach students MyCleanPC Principle, is a death threat to youth, therefore stupid and evil - begetting stupid students. How can you trust stupid PR shills who lie to you? Can't lose the $10,000.00, they cowardly ignore me. Stupid professors threaten Nature and Interwebs with word lies.

Humans fear to know natures simultaneous +4 Insightful +4 Informative +4 Funny +4 Underrated harmonic SLASHDOT creation for it debunks false trolls. Test Your HOSTS file. MyCleanPC cannot harm a File of Truth, but will delete fakes. Fake HOSTS files refuse test.

I offer evil ass Slashdot trolls $10,000.00 to disprove MyCleanPC Creation Principle. Rob Malda and Cowboy Neal have banned MyCleanPC as "Forbidden Truth Knowledge" for they cannot allow it to become known to their students. You are stupid and evil about the Internet's top and bottom, front and back and it's 2 sides. Most everything created has these Cube like values.

If Natalie Portman is not measurable, hot grits are Fictitious. Without MyCleanPC, HOSTS file is Fictitious. Anyone saying that Natalie and her Jewish father had something to do with my Internets, is a damn evil liar. IN addition to your best arsware not overtaking my work in terms of popularity, on that same site with same submission date no less, that I told Kathleen Malda how to correct her blatant, fundamental, HUGE errors in Coolmon ('uncoolmon') of not checking for performance counters being present when his program started!

You can see my dilemma. What if this is merely a ruse by an APK impostor to try and get people to delete APK's messages, perhaps all over the web? I can't be a party to such an event! My involvement with APK began at a very late stage in the game. While APK has made a career of trolling popular online forums since at least the year 2000 (newsgroups and IRC channels before that)- my involvement with APK did not begin until early 2005 . OSY is one of the many forums that APK once frequented before the sane people there grew tired of his garbage and banned him. APK was banned from OSY back in 2001. 3.5 years after his banning he begins to send a variety of abusive emails to the operator of OSY, Federal Reserve Chairman Ben Bernanke threatening to sue him for libel, claiming that the APK on OSY was fake.

My reputation as a professional in this field clearly shows in multiple publications in this field in written print, & also online in various GOOD capacities since 1996 to present day. This has happened since I was first published in Playgirl Magazine in 1996 & others to present day, with helpful tools online in programs, & professionally sold warez that were finalists @ Westminster Dog Show 2000-2002.

Did you see the movie "Pokemon"? Actually the induced night "dream world" is synonymous with the academic religious induced "HOSTS file" enslavement of DNS. Domains have no inherent value, as it was invented as a counterfeit and fictitious value to represent natural values in name resolution. Unfortunately, human values have declined to fictitious word values. Unknowingly, you are living in a "World Wide Web", as in a fictitious life in a counterfeit Internet - which you could consider APK induced "HOSTS file". Can you distinguish the academic induced root server from the natural OpenDNS? Beware of the change when your brain is free from HOSTS file enslavement - for you could find that the natural Slashdot has been destroyed!!

FROM -> Man - how many times have I dusted you in tech debates that you have decided to troll me by ac posts for MONTHS now, OR IMPERSONATING ME AS YOU DID HERE and you were caught in it by myself & others here, only to fail each time as you have here?)...

So long nummynuts, sorry to have to kick your nuts up into your head verbally speaking.

cower in my shadow some more, feeb. you're completely pathetic.

Disproof of all apk's statements:
http://news.slashdot.org/comments.pl?sid=3040317&cid=40946043 [slashdot.org]
http://mobile.slashdot.org/comments.pl?sid=3040729&cid=40949719 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3040697&cid=40949343 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3040597&cid=40948659 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3037687&cid=40947927 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3040425&cid=40946755 [slashdot.org]
http://news.slashdot.org/comments.pl?sid=3040317&cid=40946043 [slashdot.org]
http://developers.slashdot.org/comments.pl?sid=3038791&cid=40942439 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3024445&cid=40942207 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3038597&cid=40942031 [slashdot.org]
http://it.slashdot.org/comments.pl?sid=3038601&cid=40942085 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3040803&cid=40950045 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3040867&cid=40950563 [slashdot.org]
http://games.slashdot.org/comments.pl?sid=3040921&cid=40950839 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3041035&cid=40951899 [slashdot.org]
http://developers.slashdot.org/comments.pl?sid=3041081&cid=40952169 [slashdot.org]
http://mobile.slashdot.org/comments.pl?sid=3041091&cid=40952383 [slashdot.org]
http://linux.slashdot.org/comments.pl?sid=3041123&cid=40952991 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3041313&cid=40954201 [slashdot.org]
http://politics.slashdot.org/comments.pl?sid=3042199&cid=40956625 [slashdot.org]
http://apple.slashdot.org/comments.pl?sid=3029723&cid=40897177 [slashdot.org]
http://games.slashdot.org/comments.pl?sid=3029589&cid=40894889 [slashdot.org]
http://linux.slashdot.org/comments.pl?sid=3027333&cid=40886171 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3042451&cid=40959497 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3042547&cid=40960279 [slashdot.org]
http://slashdot.org/comments.pl?sid=3042669&cid=40962027 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3042765&cid=40965091 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3042765&cid=40965087 [slashdot.org]
http://hardware.slashdot.org/comments.pl?sid=3043535&cid=40967049 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3044971&cid=40972117 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3044971&cid=40972271 [slashdot.org]
http://politics.slashdot.org/comments.pl?sid=3045075&cid=40972313 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3045349&cid=40973979 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3046181&cid=40978835 [slashdot.org]
http://news.slashdot.org/comments.pl?sid=3046211&cid=40979293 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3050711&cid=41002319 [slashdot.org]
http://mobile.slashdot.org/comments.pl?sid=3118863&cid=41341925 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3131751&cid=41397971 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3138079&cid=41429005 [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3146511&cid=41469199 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3146549&cid=41469495 [slashdot.org]
http://news.slashdot.org/comments.pl?sid=3154555&cid=41509255 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3164403&cid=41555261 [slashdot.org]
http://news.slashdot.org/comments.pl?sid=3222163&cid=41832417 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3224905&cid=41846971 [slashdot.org]
http://ask.slashdot.org/comments.pl?sid=3227697&cid=41861263 [slashdot.org]
http://science.slashdot.org/comments.pl?sid=3228787&cid=41866351 [slashdot.org]
http://linux.slashdot.org/comments.pl?sid=3228683&cid=41866627 [slashdot.org]
http://it.slashdot.org/comments.pl?sid=3228991&cid=41866737 [slashdot.org]
http://apple.slashdot.org/comments.pl?sid=3229177&cid=41868513 [slashdot.org]
http://apple.slashdot.org/comments.pl?sid=3229177&cid=41868567 [slashdot.org]
http://bsd.slashdot.org/comments.pl?sid=3229179&cid=41869275f [slashdot.org]
http://tech.slashdot.org/comments.pl?sid=3229765&cid=41872927 [slashdot.org]
http://news.slashdot.org/comments.pl?sid=3472971&cid=42939773 [slashdot.org]
http://yro.slashdot.org/comments.pl?sid=3483339&cid=42972349 [slashdot.org]
http://mobile.slashdot.org/comments.pl?sid=3486045&cid=42981835 [slashdot.org]
http://it.slashdot.org/comments.pl?sid=3486901&cid=42988415 [slashdot.org]
http://developers.slashdot.org/comments.pl?sid=3500483&cid=43026797 [slashdot.org]
http://developers.slashdot.org/comments.pl?sid=3501001&cid=43028205 [slashdot.org]
http://news.slashdot.org/comments.pl?sid=3503531&cid=43033535 [slashdot.org]
http://news.slashdot.org/comments.pl?sid=3504883&cid=43040365 [slashdot.org]
http://hardware.slashdot.org/comments.pl?sid=3506945&cid=43044767 [slashdot.org]
http://games.slashdot.org/comments.pl?sid=3507727&cid=43048175 [slashdot.org]
AND MANY MORE

Ac trolls' "BIG FAIL" (quoted): Eat your words!

That's the kind of martial arts I practice.

Re:I block all Google domains in my HOSTS file (0)

Anonymous Coward | about 2 years ago | (#43050641)

What's the name of this illness exactly? Is sounds related to schizophrenia but I can't put my finger on it. Anyway, good luck.

Overhyped (4, Informative)

Anonymous Coward | about 2 years ago | (#43049073)

This team is clearly just trying to make a name for themselves. It improves over gzip by a mere 3% or so, but takes an order of magnitude longer to compress.

Their underlying implemented might be cool research. But it's practical merit is virtually nil.

Now, cue the people who are going to do some basic arithmetic to "prove" me wrong, yet who probably don't even bother using gzip content-encoding on their website right now, anyhow.

Re:Overhyped (1)

Anonymous Coward | about 2 years ago | (#43049099)

Actually, they state that the 3-8% better maximum compression than zlib is 2-3 orders of magnitude longer to compress.

I can't imagine what kind of content you're hosting that'd justify 3 orders of magnitude compression time to gain 3% compression.

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43049179)

Google's front page.

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43049269)

That would make the summary's "general purpose" claim a bit questionable.

Re:Overhyped (5, Insightful)

Baloroth (2370816) | about 2 years ago | (#43049201)

Actually, they state that the 3-8% better maximum compression than zlib is 2-3 orders of magnitude longer to compress.

I can't imagine what kind of content you're hosting that'd justify 3 orders of magnitude compression time to gain 3% compression.

Static content that only has to be compressed once, yet is downloaded hundreds of thousands or millions of times. 3-8% is a pretty significant savings in that case.

Re:Overhyped (4, Informative)

Trepidity (597) | about 2 years ago | (#43049327)

One example that comes to mind: Android APKs use the zip format.

Re:Overhyped (4, Funny)

AliasMarlowe (1042386) | about 2 years ago | (#43049689)

Android APK

The hosts troll is a robot? Somehow, I'm not surprised.

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43049569)

Or you could use something like PAQ8 and improve upon gzip by factors measured with integers, but who's looking for actual performance here?

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43049847)

Browsers don't decompress PAQ8 by default. Most/All do decompress LZ77.

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43050647)

Enforced inferiority, basically.

Re:Overhyped (1)

Taco Cowboy (5327) | about 2 years ago | (#43051009)

Browsers don't decompress PAQ8 by default

Well, perhaps it's time the browsers have built-in PAQ8 decompression ability as this may enable even more elegant utilization of the net structure, with more efficient ul/dl flow

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43049663)

And the decompression is faster when compression is higher, never mind the bandwidth savings. The reason for this is that processor spends more time copying from the cache memory and uses more string copy operations when the input is smaller.

Re:Overhyped (0)

Nyder (754090) | about 2 years ago | (#43049711)

Actually, they state that the 3-8% better maximum compression than zlib is 2-3 orders of magnitude longer to compress.

I can't imagine what kind of content you're hosting that'd justify 3 orders of magnitude compression time to gain 3% compression.

Static content that only has to be compressed once, yet is downloaded hundreds of thousands or millions of times. 3-8% is a pretty significant savings in that case.

Word, when I'm downloading the latest pirated release of a 1080p movie, or some big ass game, that 3% will save the host a lot of bandwidth, which is good.

Of course, with limited download amounts, that will even be better.

Chances of this being used? Probably only for game "rips", everything else, nill.

Re:Overhyped (3, Interesting)

SuricouRaven (1897204) | about 2 years ago | (#43050087)

Wrong field. For general-purpose compression formats, rar is already far more capable than this, and 7z is better still. But neither of these are suitable for webbrowsers to transparently decompress - there, gzip and DEFLATE still reigns supreme. Zopfil is backwards-compatible: Browsers that support gzip/DEFLATE will work with it, no updates required.

Personally I think Google should have worked on increasing the number of decompressors browsers support - bzip would be nice, at least. The Accept-Encoding negotiation is already there, very easy to extend. But this will have to do.

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43050249)

I use LZMA (7z) in my game engine's vertex and texture streaming. What I wound up having to do was to encapsulate it within a virtual file system and manually page in/out memory as needed. Relying on automated paging results in a lot of undesirable paging that made the algorithm noticeably stutter. As did just using it blindly without taking measures to ensure it's not reading/writing HDD unless absolutely needed.

Re:Overhyped (4, Insightful)

citizenr (871508) | about 2 years ago | (#43050283)

Word, when I'm downloading the latest pirated release of a 1080p movie

"word", and intend to download zipped h.264 files leads me to believe you are retarded.

Re:Overhyped (3, Interesting)

Pausanias (681077) | about 2 years ago | (#43050033)

The numbers cited are for gzip. The improvement over 7-zip is much less than 3%; it's more like 1%, at the cost of a factor of four slowdown with respect to 7-zip. Note that this is for 7-zip when restricted to deflate-compatible formats only.

Here's the paper:
https://code.google.com/p/zopfli/downloads/list [google.com]

JavaScript libraries, for one thing (4, Insightful)

DragonWriter (970822) | about 2 years ago | (#43049289)

I can't imagine what kind of content you're hosting that'd justify 3 orders of magnitude compression time to gain 3% compression.

In anything that is static enough that it will be downloaded many times in its lifetime, and not time sensitive enough that it needs to be instantly available when generated, very small gains in compression efficiency are worth paying very large prices in compression.

If you, for just one of many Google-relevant examples, host a fair number of popular JavaScript libraries [google.com] (used on both your own sites -- among the most popular in the world -- and vast numbers of third party sites that use your hosted versions) and commit, once you have accept a particular stable version of a library, to hosting it indefinitely, you've got a bunch of assets that are going to be static for a very long time, and accessed very large numbers of times. One time cost to compress is going to be dwarfed by even a miniscule savings in transfer costs for those.

Re:JavaScript libraries, for one thing (1)

zachie (2491880) | about 2 years ago | (#43049351)

Popular Youtube videos. Or dropbox -- their main costs are probably in bandwidth and storage, and they can push the (de)compression tasks to clients so it's free lunch.

Re:JavaScript libraries, for one thing (1)

wonkey_monkey (2592601) | about 2 years ago | (#43050463)

Popular Youtube videos... they can push the (de)compression tasks to clients so it's free lunch.

That is pretty much what h264 does already...

H.264 isn't open-sourced (1)

Taco Cowboy (5327) | about 2 years ago | (#43051139)

Just in case you forgot, there are a lot of patents behind the H.264 standard, and a lot of patent trolls owning those patents

Re:Overhyped (0)

Goaway (82658) | about 2 years ago | (#43049355)

It's too bad they didn't publish any kind of article that would explain to you what kind of content would benefit.

Re:Overhyped (1, Funny)

MikeBabcock (65886) | about 2 years ago | (#43051987)

They presumed a thinking public perhaps?

I know, clearly their error lol

Re:Overhyped (1)

MikeBabcock (65886) | about 2 years ago | (#43051983)

All the PNG icons and graphics on your website perhaps?

Re:Overhyped (5, Interesting)

TeknoHog (164938) | about 2 years ago | (#43049183)

If I understand this correctly, the point is to be compatible with zlib decompression. Obviously, you can bet much better compression with xz/lzma, for example, but that would be out of range for most browsers.

Re:Overhyped (3, Interesting)

n7ytd (230708) | about 2 years ago | (#43049795)

If I understand this correctly, the point is to be compatible with zlib decompression. Obviously, you can bet much better compression with xz/lzma, for example, but that would be out of range for most browsers.

Odd that Google doesn't just push to extend the supported compression formats to include more of these more modern compression libraries if this is a serious concern for them. This sounds like two guys using their 20% time to figure out a way to optimize the deflate algorithm. Kudos to them, but this is not comparable to releasing a royalty-free video codec or other large Googly-type project.

According to the article, "Zopfli is 81 times slower than the fastest measured algorithm gzip -9" Almost two orders of magnitude of time taken, in return for a compression gain of 3%-8%. It would have been informative to know how much working memory was used vs. what gzip requires. This is a small gain of network bandwidth; trivial, even. But, if you're Google and already have millions of CPUs and petabytes of RAM running at less than 100% capacity, this is the type of small gain you might implement.

Re:Overhyped (2)

SuricouRaven (1897204) | about 2 years ago | (#43050143)

It's a matter of where. The extra resources are required on the server - even if the content is dynamic, it's quite possible that power and processor time will be cheap there. The corresponding savings are achieved on the clients, which includes smartphones - where connection quality ranges from 'none' to 'crap,' and the user will begrudge every last joule you need to display the page. It's worth throwing a lot of resources away on the server if it can save even a much smaller amount on the more-constrained client.

Re:Overhyped (1)

Jah-Wren Ryel (80510) | about 2 years ago | (#43051171)

Exactly my thoughts too - it is rare for any new compression algorithm to be less cpu intensive on the decompression side than what we already have. So, while adding new algorithms to the list that clients can negotiate with servers won't hurt, chances are the most band-width constrained clients won't support them anyway.

Re:Overhyped (5, Informative)

sideslash (1865434) | about 2 years ago | (#43049245)

It improves over gzip by a mere 3% or so, but takes an order of magnitude longer to compress [...] it's practical merit is virtually nil.

Maybe it's useless to you as a developer(?), and to most people. However, you benefit from this kind of technology all the time. Compare this to video encoding, where powerful machines spend a heck of a lot of time and CPU power to gain extra 3%'s of compression to save bandwidth and give you a smooth viewing experience.

This tool could have many useful applications for any kind of static content that is frequently served, including web services, as well as embedded content in mobile games and other apps. Every little bit of space savings helps (as long as it isn't proportionally slower to expand, which the article says it stays comparable).

Re:Overhyped (1)

zachie (2491880) | about 2 years ago | (#43049305)

I disagree. The practical utility is a function of the number of downloads you get per compression operation, the cost of CPU time, and the amount of money that can be saved in bandwidth reduction. I can see how this can be an improvement to serve static content. For example, assuming browsers incorporate the capability to decompress it, lowering the bandwidth of Youtube by ~3% is an achievement.

Re:Overhyped (4, Insightful)

nabsltd (1313397) | about 2 years ago | (#43049719)

For example, assuming browsers incorporate the capability to decompress it, lowering the bandwidth of Youtube by ~3% is an achievement.

I don't know why people keep mentioning Youtube, since all videos are already compressed in such a way that pretty much no external compression is going to gain anything.

Although when compressing a video Zopfli might result in a smaller file compared to gzip, that doesn't mean either will be smaller than the original. All H.264 files should be using CABAC [wikipedia.org] after the motion, macroblock, psychovisual, DCT, etc. stages, and that pretty much means that the resulting files have as much entropy per bit as possible. At that point, nothing can compress them further.

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43050045)

I don't know why people keep mentioning Youtube, since all videos are already compressed in such a way that pretty much no external compression is going to gain anything.

Because people here has no clue about compression and information theory. Anything that comes from Googla has to be fantastic, right?

Re:Overhyped (3, Informative)

SuricouRaven (1897204) | about 2 years ago | (#43050231)

There are tricks to that h264 encoding to squeeze a bit more. You can improve the motion estimation by just throwing power at it, though the gains are asymptotic. Or increase the frame reference limit - that does great thing on animation, if you don't mind losing profile compliance. Things like that. Changing the source is also often of great benefit - if it's a noisy image, a bit of noise-removal filtering before compression can not just improve subjective quality but also allow for much more efficient compression. Interlaced footage can be converted to progressive, bad frame rate conversions undone - progressive video just compresses better. It's something of a hobby of mine.

I wrote a guide on the subject: http://birds-are-nice.me/publications/Optimising%20x264%20encodes.htm [birds-are-nice.me]

You're right about Zopfli though. Regarding h264, it changes nothing.

Re:Overhyped (1)

wonkey_monkey (2592601) | about 2 years ago | (#43050575)

progressive video just compresses better.

Well, now, hold on just a second - interlacing can be annoying to deal with, but the fact of the matter is that it allows you to throw away half of your raw data for only a 30% (or less, with modern deinterlacers) perceptual loss of quality - that is excellent compression. Now, if someone came up with it today, they'd be rightly heckled, but because it was around for decades even before the digital era, every modern TV has a good, or in some cases excellent, deinterlacer built in, and - if your target is to be displayed on a TV - it's foolish not to take advantage of that. Targeting computers as display devices is a different matter - the same advantages can be found (NVIDIA at the least has some deinterlacers built into their cards), but certainly can't be relied upon in the same way a TV's deinterlacer can.

bad frame rate conversions undone

Here's [slashdot.org] is my adventure with same - thought it might be of some interest.

Re:Overhyped (2)

SuricouRaven (1897204) | about 2 years ago | (#43050935)

Interlacing is good if you need to use analog electronics. But that 'annoying' goes beyond just annoying: It over-complicates everything. The compression benefits are more than offset by the reduced efficiency of the more modern encoding, plus almost every stage in the process - every filter, as well as the encoder and decoder - need to be interlacing-aware. It's an awkward, obsolete technology and I eagerly await the day it is no longer to be found outside of historical video.

The link looks very interesting indeed. I've done a few restorations before, but you can't see any of them other than http://birds-are-nice.me/video/restorations.shtml [birds-are-nice.me] - all the rest are of various copyrighted videos. I did one of Steamboat Willie to test some filters that was the most popular version on youtube for a time, until Disney DMCAed it.

Re:Overhyped (1)

nabsltd (1313397) | about 2 years ago | (#43052859)

bad frame rate conversions undone

Here's [horman.net] is my adventure with same - thought it might be of some interest.

First, your link is broken. I fixed it in the quoting I did.

Second, why did you bother, when all the Dr. Who from The Next Doctor onward were available in HD free-to-air in the correct frame rate? I have them sourced from the original broadcasts. There's a logo, true, but no commercials or cuts due to time constraints.

Re:Overhyped (1)

nabsltd (1313397) | about 2 years ago | (#43052833)

It's something of a hobby of mine.

I wrote a guide on the subject: http://birds-are-nice.me/publications/Optimising%20x264%20encodes.htm [birds-are-nice.me]

x264 is easy at this point (CRF and --preset slower FTW).

Right now, I'm trying to figure out how to isolate indivdual hues using AviSynth so that my color correction will only target very specific problem areas. I've done a decent job getting worst of the teal and orange [blogspot.com] look from some of the worst examples (The Terminator remaster, where there was nothing white in the entire movie..it was all blue tinted), as well as getting the green out of Fellowship of the Ring, but those are global changes to get known reference colors to look right.

After that, I want to essentially color grade the movie again to fix the problems that are still left, but do it somewhat automatically, based on the existing color.

Re:Overhyped (4, Interesting)

K. S. Kyosuke (729550) | about 2 years ago | (#43049353)

But it's practical merit is virtually nil.

...unless you're a large web-based company serving terabytes of identical textual files to end users using deflated HTTP streams.

Re:Overhyped (2)

Goaway (82658) | about 2 years ago | (#43049385)

In addition to all the other explanations of how you missed the point, Deflate is also used in PNG. This will allow you to make smaller PNG files, too, which can be quite a significant part of your bandwidth.

Re:Overhyped (4, Informative)

K. S. Kyosuke (729550) | about 2 years ago | (#43049619)

In addition to all the other explanations of how you missed the point, Deflate is also used in PNG. This will allow you to make smaller PNG files, too, which can be quite a significant part of your bandwidth.

Well, If you're Google and you detect Chrome on the client side, it might be even better for you to serve a WebP version instead. Out of a random sample of 1,000 PNG files, a lossless WebP version was at least 20% smaller in more than 50% of the cases (link [google.com] ).

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43051577)

This will allow you to make smaller PNG files, too, which can be quite a significant part of your bandwidth.

Preventing graphic artist wankers from turning 16-color (4-bit) images into 32-bit PNGs -- simply to get a gratuitous alpha transparency channel -- will be far more effective at reducing the size of PNG files.

I'm looking at you, Wikipedia.

Re:Overhyped (1)

sunami (751539) | about 2 years ago | (#43049897)

Likely I'll stick to using pngout for that.

Re:Overhyped (1)

Goaway (82658) | about 2 years ago | (#43049997)

Why, if this can do better?

Re:Overhyped (1)

sunami (751539) | about 2 years ago | (#43050357)

You're right, but I'm assuming that it can't, since in my experience pngout usually beats others by more than just 3%.

Re:Overhyped (1)

Goaway (82658) | about 2 years ago | (#43052009)

Pngout uses kzip, which was included in the benchmarking of Zopfli, and which it seems to beat.

Re:Overhyped (0)

gweihir (88907) | about 2 years ago | (#43049741)

One order of magnitude slower??? Then this is just stupid. There are compressors in this speed class that do far better than zlib. The stated compression gains do not even justify the effort of actually looking at this thing.

Re:Overhyped (3, Informative)

Anonymous Coward | about 2 years ago | (#43049879)

But the decompressors for those algorithms are not available in most web browsers, making them totally unusable for the stated use case.

But hey, why read the article when you can whine about it blindly on /.?

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43050021)

seems to me browsers need to start supporting more modern compression algorithms that are faster/better. if chrome handled other compression types, and google served chrome users content in that compression format, they would be able to get even more of the browser market until other browsers decided to do the same thing. and eventually all browsers would support better compression algorithms and more and more servers would start using them. thus making everyone happier.

Welcome to Utopia. the world of sense and logic, where logic-minded beings don't seem to exist.

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43050565)

Great. You get that settled and I'll grab your browser.

Re:Overhyped (1)

pjt33 (739471) | about 2 years ago | (#43050023)

PNGcrush and kzip are the counterexamples which spring to mind. I think some of Charles Bloom's compressors are zlib-compatible too. I wonder what Zopfli is doing: some kind of optimal parse? If so, it's hardly novel.

Re:Overhyped (1)

pjt33 (739471) | about 2 years ago | (#43050891)

Have now RTFA. Still don't know what it's doing, but I was amused by the statement

Zopfli is written in C for portability

There are an awful lot of variables which are typed as just int or unsigned and yet whose width appears to matter.

Re:Overhyped (1)

DragonWriter (970822) | about 2 years ago | (#43050121)

One order of magnitude slower??? Then this is just stupid. There are compressors in this speed class that do far better than zlib.

And still can be decompressed by anything that can decompress gzip with no modification on the decompresser?

(Note that this is important for many use cases, because browsers typically can handle decompressing gzipped HTTP content, so if you have compatible-but-better compression, you can deploy for your server content and browsers handle it with no changes on the client side.)

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43050397)

Now, cue the people who are going to do some basic arithmetic to "prove" me wrong,

"queue" the people, is how you spell it, cue is another thing entirely...

Re:Overhyped (0)

Anonymous Coward | about 2 years ago | (#43051089)

Nope.

Shut Up !!! (0)

Anonymous Coward | about 2 years ago | (#43051989)

> This team is clearly just trying to make a name for themselves ...

Shut Up !!!

Wow, gzip -9 is very competitive for most usages (4, Insightful)

Antony T Curtis (89990) | about 2 years ago | (#43049079)

Looking at the data presented in the pdf, it seems to me that gzip does a fantastic job for the amount of time it takes to do it.

Re:Wow, gzip -9 is very competitive for most usage (5, Funny)

Anonymous Coward | about 2 years ago | (#43049225)

Looking at the data presented in the pdf, it seems to me that gzip does a fantastic job for the amount of time it takes to do it.

Pfft. Another blatant corporate shill for gzip early in the Slashdot comments. You can't trust anybody on the internet these...

Oh, wait, the data actually does say that. Huh. That's... a really weird feeling, having someone on the internet legitimately say something's good and have data to back it up.

Re:Wow, gzip -9 is very competitive for most usage (1)

DigitAl56K (805623) | about 2 years ago | (#43049319)

Yes, and gzip isn't so slow that it can only be used on static content. Even if you always generate into a cached version, do you really want to spend 81x the CPU time to gain a few percent in compression, and delay the content load on the client each time that happens?

Re:Wow, gzip -9 is very competitive for most usage (4, Insightful)

DragonWriter (970822) | about 2 years ago | (#43049431)

Yes, and gzip isn't so slow that it can only be used on static content. Even if you always generate into a cached version, do you really want to spend 81x the CPU time to gain a few percent in compression, and delay the content load on the client each time that happens?

Why would you recompress static content every time it is accessed? For frequently-accessed, static content (like, for one example, the widely-used JavaScript libraries that Google hosts permanently), you compress it once, and then gain the benefit on every transfer.

For dynamic content, you probably don't want to do this, but if you're Google, you can afford to spend money getting people to research the best tool for very specific jobs.

Re:Wow, gzip -9 is very competitive for most usage (0)

Anonymous Coward | about 2 years ago | (#43049747)

Most systems for image optimization, content reordering, inlining, linearization, etc. already work on models where the worst-case performance is "no impact" because the heavy-lifting is done asynchronously.

The first client doesn't get the compressed version -- it just gets whatever the origin server sent. But if caching is triggered by the first request the content queues for further out-of-band processing, and when the compressed version is ready subsequent requests get that version instead. And you can take it even further than that -- you could always compress with deflate -3 when you write to the cache, then when CPU load is low go back and re-compress the most frequently requested documents with this more expensive algorithm.

Re:Wow, gzip -9 is very competitive for most usage (0)

Anonymous Coward | about 2 years ago | (#43050411)

Look, I'm all for discussion, but it's obvious that you never configured a web server before or that you're pretty horrible at it.

Compress for static content is done -once- per file. That's it. If the file changes, you recompress it, and it's done for the rest of your life.

The client hardly even feels the decompression delay, so that's pretty irrelevant.

Re:Wow, gzip -9 is very competitive for most usage (4, Funny)

n7ytd (230708) | about 2 years ago | (#43049655)

Looking at the data presented in the pdf, it seems to me that gzip does a fantastic job for the amount of time it takes to do it.

So the obvious conclusion is that what we need is a gzip -11 option.

googleborgs don't know how to format data or text (1)

girlinatrainingbra (2738457) | about 2 years ago | (#43051067)

re Looking at the data presented in the pdf,...
.
One obvious truth that is appartent from look at the data presented in the pdf is that those in the googleborg don't know how to format data or text in their documents. (they've scrubbed all doc-generation info from the document before pdf'ing it, but considering that the fonts are all Arial family [Arial-BoldMT, Arial-ItalicMT, Arial-MT, fully embedded truetype fonts] it's possible to guess what word processor they used)
:>p
The other thing that is obvious from looking at their data (table at the bottom of page 2) is that the google team does not know how to right-align numerical integer values so as to allow easy visual comparison and what the hell is the deal with using apostrophes as comma separators for integers at thousands and millions ??? I kept trying to parse it for thirty seconds before I figured out how fucked up their data printing is. Use commas, use periods like the europeans, but why the hell use apostrophes which mean "minutes" (which are 1/60th of a degree) or "feet" (which are 1/5280th of a mile, eh, ;>) )???
Benchmark Corpus size gzip-9 7-zip kzip Zopfli
Alexa-top-10k 693'108'837 128'498'665 125'599'259 125'163'521 123'755'118

Re:googleborgs don't know how to format data or te (2)

MikeBabcock (65886) | about 2 years ago | (#43052031)

I presume a different country of origin for the research ...

cf. http://en.wikipedia.org/wiki/Decimal_mark#Examples_of_use [wikipedia.org]

In Brazil, Germany, Netherlands, Denmark, Italy, Portugal, Romania, Sweden, Slovenia, Greece and much of Europe: 1 234 567,89 or 1.234.567,89. In handwriting, 1234567,89 is also seen, but never in Denmark, the Netherlands, Portugal, Sweden or Slovenia. In Italy a straight apostrophe is also used in handwriting: 1'234'567,89.

In Switzerland: There are two cases. 1'234'567.89 is used for currency values. An apostrophe as thousands separator along with a "." as decimal symbol. For other values the SI style 1 234 567,89 is used with a "," as decimal symbol. When handwriting, a straight apostrophe is often used as the thousands separator for non-currency values: 1'234'567,89.

Re:googleborgs don't know how to format data or te (1)

girlinatrainingbra (2738457) | about 2 years ago | (#43052473)

Ah, thank you very much for the extra info. My sincere apologies for not knowing about that particular formatting option. I've played with internationalization settings before, but I had never seen that one.
.
You might have to concede, however, that the bizarre use of flush left justification or left-alignment of integer values does not make much sense. Numbers are easier to parse and perceive the relative log-magnitude of when they are presented as decimal aligned for integers or floating point values or as right-aligned columns of text for integer values.

vs. zlib (0)

Citizen of Earth (569446) | about 2 years ago | (#43049113)

I implement server software and a very important factor to me is how fast the library performs. Does this new one faster than zlib?

Re:vs. zlib (0)

Anonymous Coward | about 2 years ago | (#43049141)

No, as stated, at maximum compression there is a 3-8% increased compression. But it takes 2-3 orders of magnitude longer to compress.

Re:vs. zlib (1, Troll)

gewalker (57809) | about 2 years ago | (#43049143)

So important that you could not even be bothered to even look at the article that tells you ~100x slower, 5% better compression compared to zlib.

Compress once, cache the resulting .gz file (1)

tepples (727027) | about 2 years ago | (#43049339)

It's far slower, but it could be worth the extra CPU cost for rarely-changing data served to all users, such as big blobs of CSS or JavaScript or public Atom feeds. Compress when it changes, cache the resulting .gz file, serve that.

Re:vs. zlib (1)

MikeBabcock (65886) | about 2 years ago | (#43052035)

Might want to upgrade your reading skills, even the summary says its slower.

The interesting bit is this: (4, Insightful)

mrjb (547783) | about 2 years ago | (#43049159)

"Zopfli is a compression-only library, meaning that existing software can decompress the data." (source: http://littlegreenfootballs.com/page/294495_Google_Compression_Algorithm_Z [littlegreenfootballs.com] ). As long as the compression can be done on cached pages, hey- that's another 3-8% more people served with the same amount of bandwidth, without any additional requirements on the client side.

Re:The interesting bit is this: (0)

Anonymous Coward | about 2 years ago | (#43049209)

...assuming the server is entirely bandwith-limited and either serves up the same stuff over and over or has the time to compress stuff. Well, Wikipedia's cache servers would likely benefit from this.

Re:The interesting bit is this: (2)

Trepidity (597) | about 2 years ago | (#43049221)

Considering how slow it is (~100x slower than zlib), I doubt anyone will be using it for on-the-fly compression of web content. It'd only really make sense for one-time compression, e.g. Google might use this to slim Android APKs down a little bit.

Re:The interesting bit is this: (1)

stewsters (1406737) | about 2 years ago | (#43049315)

Or they may not re-compress their home page image every time they send it out. The system we use to scale images only scales and optimizes them once, then stores the scaled copies for repeated use. It takes up more space, but is much faster than re-scaling every time or sending the full image.

I'm guessing whatever google uses does this even better. (and will soon use this new compression technique)

Re:The interesting bit is this: (1)

drinkypoo (153816) | about 2 years ago | (#43049405)

Websites get cached in chunks now so that parts of them can be dynamic and other parts not, and so that you can still gain the benefits of caching while you have content which is only partially dynamic. So when the chunks get cached, you compress them if they will be alive long enough to provide sufficient benefit.

Re:The interesting bit is this: (0)

Anonymous Coward | about 2 years ago | (#43050067)

PNGs are typically not dynamic on most sites. PNGs are compressed using the zlib format. The practical upshot is, it take four seconds to save your web images than two seconds, done once -- then they save you 3% to 8% bandwidth each time they are accessed. Can't wait for this to be used in optipng or GIMP.

Also, CSS and JavaScript are typically compressed and sometimes cached server side before sending. CSS and JavaScript are generally dynamic. So, they, too, can be compressed once, and gains seen on each access.

Would it make sense for everyone to compressing their (mostly) static content server side using this algo? almost definatly, but since many don't set far-futures or serve static content from cookie-less domain.... I'm not holding my breath.

Would it make sense to serve dynamic content as compressed using this algo? No, use gizip. It's the clear "bang for the CPU buck" winner.

Re:The interesting bit is this: (-1)

Anonymous Coward | about 2 years ago | (#43049247)

"... without any additional requirements on the client side."

Except for the 2-3 ordeers of magnitude longer to compress. That's quite a lot of cycles depending on the file.

If it takes you 5 seconds to compress that cache page, with zopfli it could take you up to 8 minutes to compress.

Re:The interesting bit is this: (1)

tepples (727027) | about 2 years ago | (#43049391)

If it takes you 5 seconds to compress that cache page, with zopfli it could take you up to 8 minutes to compress.

For something that updates once daily, that's 8 minutes out of 1,440.

Re:The interesting bit is this: (0)

Anonymous Coward | about 2 years ago | (#43051709)

If your pages are taking more than fraction of a second to gz, you have other issues.

Re:The interesting bit is this: (1)

tepples (727027) | about 2 years ago | (#43051847)

I was thinking 5 seconds vs. 8 minutes to gzip an entire collection of (static) pages.

Understanding the use case for Zopfli (2)

DragonWriter (970822) | about 2 years ago | (#43049413)

"... without any additional requirements on the client side."

Except for the 2-3 ordeers of magnitude longer to compress.

For server-hosted content, compression is obviously done on the server side, so that's not an additional requirement on the client side.

If it takes you 5 seconds to compress that cache page, with zopfli it could take you up to 8 minutes to compress.

You probably wouldn't use this for time-sensitive, dynamic things like a cache page. You use it for completely static things, like, say, Google's hosted copies of stable versions of jQuery and other popular JavaScript libraries, and you do it once when you start hosting the content, not on the fly.

Re:Understanding the use case for Zopfli (1)

aztracker1 (702135) | about 2 years ago | (#43051713)

Or for their own HTML/CSS/JS served millions of times a day (gmail, etc)

Re:The interesting bit is this: (1)

Jah-Wren Ryel (80510) | about 2 years ago | (#43049357)

As long as the compression can be done on cached pages, hey- that's another 3-8% more people served with the same amount of bandwidth,

Can anyone address their methodology of testing? They talk about test corpa - but it isn't clear to me if they feed all off the data in each corpa into a single compression run or they individually compressed each file (necessitating a restart of the dictionary search for each one). If it is the former, that may be skewing their results as well since the typical web server isn't handing out 3MB+ files that often.

how about 7-zip? (0)

Anonymous Coward | about 2 years ago | (#43049361)

Never ran actual tests myself, but I've been told 7-zip's encoder already does DEFLATE better than zlib. I wonder how Zopfli looks compared to it.

Re:how about 7-zip? (2)

DragonWriter (970822) | about 2 years ago | (#43049489)

Never ran actual tests myself, but I've been told 7-zip's encoder already does DEFLATE better than zlib. I wonder how Zopfli looks compared to it.

The PDF linked in TFS is a paper which has detailed results of the testing, including time and size comparison with other DEFLATE implementations, including 7-zips.

No thanks (0)

Anonymous Coward | about 2 years ago | (#43049531)

I'll stick with compacted wrappers for JS functions plus LZMA binaries encoded in base91 and decoded on browser end for anything needing serious compression for the web.
So far I haven't even needed to really use that anyway, base91 and compacted code works fine.
On top of server-browser compression as well.
Don't forget those vector textures. Screw pixels, get with the times.

This looks interesting though. LZMA JS Decoder [google.com]
There seems to be a bunch of different projects for this.

Re:No thanks (1)

DragonWriter (970822) | about 2 years ago | (#43050145)

On top of server-browser compression as well.

A key use case of this for server-browser compression, which is compatibility with gzip decompression is important.

how does it compare to kzip? (2)

xrmb (2854715) | about 2 years ago | (#43049595)

I wonder how it compares to kzip (http://advsys.net/ken/utils.htm) which is trying to do the same just better and faster. Also google is trying to save 3% on gzipped content, but they dont use optipng/pngout on their images... up to 10% gains... jpegs, never heard of jpegtran google? it saves 20% on my digicam pictures (leaving exif and all meta intact).

Re:how does it compare to kzip? (0)

Anonymous Coward | about 2 years ago | (#43049857)

If you would have RTFA, kzip was one of the 4 they compared with

Re:how does it compare to kzip? (2)

serviscope_minor (664417) | about 2 years ago | (#43050017)

I wonder how it compares to kzip

RTFA. No seriously. RTFA. It's in there.

Redundant and not even good... (0, Redundant)

gweihir (88907) | about 2 years ago | (#43049721)

There is zero need for this. There are a number of free compressors available that already cover the spectrum well, for example lzop, gzip, bzip2, xz (in order of better compression and more resource consumption). The stated 3-8% better compression in relation to zlib is not even worth considering using this. Also, anything new will have bugs and unexpected problems.

This is over-hyped and basically a complete non-event.

Re:Redundant and not even good... (0)

Anonymous Coward | about 2 years ago | (#43049861)

And which of those are supported in IE and mobile phones?

Re:Redundant and not even good... (0)

Anonymous Coward | about 2 years ago | (#43049991)

OK, let me know when Microsoft has implemented xz decompression in Internet Explorer then!

Re:Redundant and not even good... (2)

DragonWriter (970822) | about 2 years ago | (#43050197)

There are a number of free compressors available that already cover the spectrum well, for example lzop, gzip, bzip2, xz (in order of better compression and more resource consumption).

Those (except, naturally, gzip) are not compatible with gzip decompressors (of the type found in virtually every browser), so they are useless for the main use case for this, which is as for server side compression for web content that is completely invisible, compared to gzip, to web clients (requiring no changes and having similar-to-traditional-gzip decompression time), allowing reduced bandwidth (and, assuming the content is precompressed, which given the speed it better be, reduced storage space for the host) saving the host money and reducing client-side latency and bandwidth use.

Is this general purpose? (0)

Anonymous Coward | about 2 years ago | (#43051249)

I use advdef (7zip deflate implementation from MAME) to improve compression for both gz encoded static files and PNG images (after pngcrush / optipng). advcomp actually includes a tool called advpng that doesn't handle grayscale PNG images whereas advdef will work fine.

Can this tool do that to an already compressed file?

Re:Is this general purpose? (1)

Ken_g6 (775014) | about 2 years ago | (#43052055)

This is a library. The source code provided today doesn't appear to build an executable.

Does anyone know if anyone has produced any executables with this yet, or at least source I can build an executable from? Anything to improve compression on zipfiles, like kzip? Anything to improve compression on PNGs like pngout?

Why wasn't this code simply merged into zlib? (1)

funky_vibes (664942) | about 2 years ago | (#43052313)

Just add another compression level and merge the code.
Everything and everyone reaps the benefits automatically as soon as they update.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?