bzip2 1.0.8

We are happy to announce the release of bzip2 1.0.8.

This is a fixup release because the CVE-2019-12900 fix in bzip2 1.0.7 was too strict and might have prevented decompression of some files that earlier bzip2 versions could decompress. And it contains a few more patches from various distros and forks.

bzip2 1.0.8 contains the following fixes:

  • Accept as many selectors as the file format allows. This relaxes the fix for CVE-2019-12900 from 1.0.7 so that bzip2 allows decompression of bz2 files that use (too) many selectors again.
  • Fix handling of large (> 4GB) files on Windows.
  • Cleanup of bzdiff and bzgrep scripts so they don’t use any bash extensions and handle multiple archives correctly.
  • There is now a bz2-files testsuite at https://sourceware.org/git/bzip2-tests.git

Patches by Joshua Watt, Mark Wielaard, Phil Ross, Vincent Lefevre, Led and Kristýna Streitová.

This release also finalizes the move of bzip2 to a community maintained project at https://sourceware.org/bzip2/

Thanks to Bhargava Shastry bzip2 is now also part of oss-fuzz to catch fuzzing issues early and (hopefully not) often.

bzip2 1.0.7

We are happy to announce the release of bzip2 1.0.7.

This is an emergency release because the old bzip2 website is gone and there were outstanding security issues. The original bzip2 home, downloads and documentation can now be found at: https://sourceware.org/bzip2/

bzip2 1.0.7 contains only the following bug/security fixes:

  • Fix undefined behavior in the macros SET_BH, CLEAR_BH, & ISSET_BH
  • bzip2: Fix return value when combining –test,-t and -q.
  • bzip2recover: Fix buffer overflow for large argv[0]
  • bzip2recover: Fix use after free issue with outFile (CVE-2016-3189)
  • Make sure nSelectors is not out of range (CVE-2019-12900)

A future 1.1.x release is being prepared by Federico Mena Quintero, which will include more fixes, an updated build system and possibly an updated SONAME default.

Please read his blog for more background on this.

NOTE/WARNING: There has been a report that the CVE-2019-12900 fix prevents decompression of some (buggy lbzip2 compressed) files that bzip2 1.0.6 could decompress. See the discussion on the bzip2-devel mailinglist. There is a proposed workaround now.

glibc 2.28 cleanup – no more memory leaks

glibc already released 2.29, but I was still on a much older version and hadn’t noticed 2.28 (which is the version that is in RHEL8) has a really nice fix for people who obsess about memory leaks.

When running valgrind to track memory leaks you might have noticed that there are sometimes some glibc data structures left.

These are often harmless, small things that are needed during the whole lifetime of the process. So it is normally fine to not explicitly clean that up. Since the memory is reclaimed anyway when the process dies.

But when tracking memory leaks they are slightly annoying. When you want to be sure you don’t have any leaks in your program it is distracting to have to ignore and filter out some harmless leaks.

glibc already had a mechanism to help memory trackers like valgrind memcheck. If you call the secret __libc_freeres function from the last exiting thread, glibc would dutifully free all memory. Which is what valgrind does for you (unless you want to see all the memory left and use --run-libc-freeres=no).

But it didn’t work for memory allocated by pthreads (libpthreads.so) or dlopen (libdl.so). So sometimes you would still see some stray “garbage” left even if you were sure to have released all memory in your own program.

Carlos O’Donell has fixed this:

Bug 23329 – The __libc_freeres infrastructure is not properly run across DSO boundaries.

So upgrade to glibc 2.28+ and really get those memory leaks to zero!

All heap blocks were freed -- no leaks are possible

Valgrind 3.15.0 with improved DHAT heap profiler

Julian Seward released valgrind 3.15.0 which updates support for existing platforms and adds a major overhaul of the DHAT heap profiler.  There are, as ever, many refinements and bug fixes.  The release notes give more details.

Nicholas Nethercote used the old experimental DHAT tool a lot while profiling the Rust compiler and then decided to write and contribute A better DHAT (which contains a screenshot of the the new graphical viewer).

CORE CHANGES

  • The XTree Massif output format now makes use of the information obtained when specifying --read-inline-info=yes.
  • amd64 (x86_64): the RDRAND and F16C insn set extensions are now supported.

TOOL CHANGES

DHAT

  • DHAT been thoroughly overhauled, improved, and given a GUI.  As a result, it has been promoted from an experimental tool to a regular tool.  Run it with --tool=dhat instead of --tool=exp-dhat.
  • DHAT now prints only minimal data when the program ends, instead writing the bulk of the profiling data to a file.  As a result, the --show-top-n and --sort-by options have been removed.
  • Profile results can be viewed with the new viewer, dh_view.html.  When a run ends, a short message is printed, explaining how to view the result.
  • See the documentation for more details.

Cachegrind

  • cg_annotate has a new option, --show-percs, which prints percentages next to all event counts.

Callgrind

  • callgrind_annotate has a new option, --show-percs, which prints percentages next to all event counts.
  • callgrind_annotate now inserts commas in call counts, and sort the caller/callee lists in the call tree.

Massif

  • The default value for --read-inline-info is now yes on Linux/Android/Solaris. It is still no on other OS.

Memcheck

  • The option --xtree-leak=yes (to output leak result in xtree format) automatically activates the option --show-leak-kinds=all, as xtree visualisation tools such as kcachegrind can in any case select what kind of leak to visualise.
  • There has been further work to avoid false positives.  In particular, integer equality on partially defined inputs (C == and !=) is now handled better.

OTHER CHANGES

  • The new option --show-error-list=no|yes displays, at the end of the run, the list of detected errors and the used suppressions.  Prior to this change, showing this information could only be done by specifying -v -v, but that also produced a lot of other possibly-non-useful messages.  The option -s is equivalent to --show-error-list=yes.

Building GDB from GIT

Since the GNU Toolchain has many shared modules it sometimes feels like you have to rebuild everything (assembler, linker, binutils tools, debugger, simulators, etc.) just to get one of the latest tools from source.

Having all this reusable shared code is fun, but it does make build times a bit long.

Luckily most of the “extras” can be disabled if all you want is a fresh new GDB. Sergio Durigan Junior added the GDB configure steps to the GDB wiki so you can build GDB in just a couple of minutes after checking it out.

git clone git://sourceware.org/git/binutils-gdb.git

GNU Tools Cauldron 2019

Simon Marchi just announced that the next GNU Tools Cauldron will be in Montreal, Canada from Thursday September 12 till Sunday September 15.

The purpose of this workshop is to gather all GNU tools developers, discuss current/future work, coordinate efforts, exchange reports on ongoing efforts, discuss development plans for the next 12 months, developer tutorials and any other related discussions. This year, the GNU Tools Cauldron crosses the Atlantic Ocean and lands in Montréal, Canada. We are inviting every developer working in the GNU toolchain: GCC, GDB, binutils, runtimes, etc.

https://gcc.gnu.org/wiki/cauldron2019

The conference is free to attend, registration in advance is required.

Usability improvements in GCC 9

David Malcolm wrote a really nice blog post on Usability improvements in GCC 9 describing nicer looking diagnostics, the JSON output format, simpler C++ errors, tracking down bad initializations, optimizer diagnostics and new fix-it hints.

Perhaps a deeper change is that we now have a set of user experience guidelines for GCC, to try to keep a focus on the programmer’s experience as we implement new diagnostics. If you’d like to get involved in GCC development, please join us on the GCC mailing list. Hacking on diagnostics is a great way to get started.

Buildbot CI for DWZ

Thanks to Tom de Vries your favorite DWARF Compression tool DWZ now has a testsuite. And there is now also a buildbot CI that tests various combinations of Fedora, CentOS and Debian on armhf, aarch64, ppc64, ppc64le, s390x, x86_64 and i386 to make sure no regressions slip in during development. Happy hacking!

New PGP key

I have a new rsa4096 PGP key (0x1AA44BE649DE760A) created using gnupg. The master ([C]ertification) key is kept offline with the [S]igning, [E]ncryption and [A]uthentication rsa2048 subkeys held on a Nitrokey. The old one (0x8370665B57816A6A) is set to expire at the end of this year (after which I will probably revoke it).

pub   rsa4096 2019-01-19 [C]
EC3C FE88 F6CA 0788 774F 5C1D 1AA4 4BE6 49DE 760A
uid Mark Wielaard mark@klomp.org
uid Mark Wielaard mjw@redhat.com
uid Mark Wielaard mjw@gnu.org
sub rsa2048 2019-01-20 [E]
31D5 60E6 EC7E 13B8 8724 A568 AE58 4E5D 7893 950F
sub rsa2048 2019-01-20 [S]
1276 8A96 7959 9010 7A0D 2FDF FC57 E3CC ACD9 9A78
sub rsa2048 2019-01-20 [A]
0EBC BDFB 00AC 689B C7C0 FEE4 85C7 C86B C843 7776

Get the new public key at https://gnu.wildebeest.org/~mark/gnupg-pub.txt or from one of the keyservers with gpg --recv-key 0x1AA44BE649DE760A.

I’ll use the new (signing) key for release tar balls (elfutils 0.176 was signed with the new key). And I am using gpg-agent --enable-ssh-support now so I can use the new (authentication) key for ssh access. To get my current ssh pubkey simply gpg --export-ssh-key 0x1AA44BE649DE760A.

Clear signed text of the above statement.

Raising Strauss awareness

     ____               
    /    \              
   |-. .-.|             
   (_@)(_@)             
   .---_  \             
  /..   \_/             
  |__.-^ /              
      }  |              
     |   [              
     [  ]               
    ]   |               
    |   [               
    [  ]                
   /   |        __      
  \|   |/     _/ /_     
 \ |   |//___/__/__/_   
\\  \ /  //    -____/_  
//   "   \\      \___.- 
 //     \\  __.----._/_ 
/ //|||\\ .-         __>
[        /         __.- 
[        [           }  
\        \          /   
 "-._____ \.____.--"    
    |  | |  |           
    |  | |  |           
    |  | |  |           
    |  | |  |           
    {  } {  }           
    |  | |  |           
    |  | |  |           
    |  | |  |           
    /  { |  |           
 .-"   / [   -._        
/___/ /   \ \___"-.     
    -"     "-           

strace patch.