librsync  2.3.2
NEWS.md
1 # librsync NEWS
2 
3 ## librsync 2.3.2
4 
5 Released 2021-04-10
6 
7  * Fix #214 heap corruption for too small kbloom. This could have crashed
8  delta operations for very small files/signatures. Strangely it didn't seem
9  to cause problems for most compilers/platforms, but did trigger errors for
10  new versions of MSVC. (ljusten,
11  https://github.com/librsync/librsync/pull/213)
12 
13  * Fix #207 and add Travis Windows checks and improve compatibility. Turn on
14  `-Wconversion -Wno-sign-conversion` warnings for clang. Add MSVC compiler
15  flags to turn off posix warnings. Make all code compile clean with no
16  warnings on all Travis platforms. Added cmake config checking for windows
17  `io.h` and improve `fileutil.c` for MSVC. Fix broken error handling in
18  `rs_file_copy_cb()`. Improved trace output, making it less spamy and more
19  consistent. Add patch checking for invalid literal lengths. Improve
20  internal variable and argument types. Add explicit type conversions.
21  (dbaarda, https://github.com/librsync/librsync/pull/208)
22 
23  * Fix a bug so patch will now fail returning RS_CORRUPT on encountering a
24  zero length copy command instead of hanging. Make copy_cb() copying more
25  data than requested an assert-fail on debug builds, and a log-warning for
26  release builds. Make trace output a little less spammy about copy_cb()
27  return values. (dbaarda, https://github.com/librsync/librsync/pull/206)
28 
29 ## librsync 2.3.1
30 
31 Released 2020-05-19
32 
33  * Fix #198 cmake popt detection using pkg-config and #199 test scripts on
34  FreeBSD. Fixes and tidies FindPOPT.cmake and Findlibb2.cmake to use
35  pkg-config correctly and behave more like official FindPackage() cmake
36  modules. Makes all test scripts use /bin/sh instead of /bin/bash. (dbaarda,
37  mandree https://github.com/librsync/librsync/pull/200)
38 
39  * Change default block_len to always be a multiple of the blake2b 128 byte
40  blocksize for efficiency. Tidy and update docs to explain using
41  rs_sig_args() and rs_build_hash_table(), add rs_file_*() utils, and
42  document new magic types. Remove really obsolete entries in TODO.md. Update
43  to Doxygen 1.8.16. (dbaarda, https://github.com/librsync/librsync/pull/195)
44 
45  * Improve hashtable performance by adding a small optional bloom filter,
46  reducing max loadfactor from 80% to 70%, Fix hashcmp_count stats to include
47  comparing against empty buckets. This speeds up deltas by 20%~50%.
48  (dbaarda, https://github.com/librsync/librsync/pull/192,
49  https://github.com/librsync/librsync/pull/193,
50  https://github.com/librsync/librsync/pull/196)
51 
52  * Optimize rabinkarp_update() by correctly using unsigned constants and
53  manually unrolling the loop for best performance. (dbaarda,
54  https://github.com/librsync/librsync/pull/191)
55 
56 ## librsync 2.3.0
57 
58 Released 2020-04-07
59 
60  * Bump minor version from 2.2.1 to 2.3.0 to reflect additional rs_sig_args()
61  and strong_len=-1 support.
62 
63  * Add public rs_sig_args() function for getting the recommend signature args
64  from the file size. Added support to rdiff for `--sum-size=-1` to indicate
65  "use minimum size safe against random block collisions". Added warning
66  output for sum-sizes that are too small to be safe. Fixed possible rdiff
67  bug affecting popt parsing on non-little-endian platforms. (dbaarda,
68  https://github.com/librsync/librsync/pull/109)
69 
70  * Fixed yet more compiler warnings for various platforms/compilers.
71  (Adsun701, texierp, https://github.com/librsync/librsync/pull/187,
72  https://github.com/librsync/librsync/pull/188)
73 
74  * Improved cmake popt handling to find popt dependencies using PkgConfig.
75  (ffontaine, https://github.com/librsync/librsync/pull/186)
76 
77  * Tidied internal code and improved tests for netint.[ch], tube.c, and
78  hashtable.h. (dbaarda, https://github.com/librsync/librsync/pull/183
79  https://github.com/librsync/librsync/pull/185).
80 
81  * Improved C99 compatibility. Add `-std=c99 -pedantic` to `CMAKE_C_FLAGS` for
82  gcc and clang. Fix all C99 warnings by making all code C99 compliant. Tidy
83  all CMake checks, `#cmakedefines`, and `#includes`. Fix 64bit support for
84  mdfour checksums (texierp, dbaarda,
85  https://github.com/librsync/librsync/pull/181,
86  https://github.com/librsync/librsync/pull/182)
87 
88  * Usage clarified in rdiff (1) man page. (AaronM04,
89  https://github.com/librsync/librsync/pull/180)
90 
91 ## librsync 2.2.1
92 
93 Released 2019-10-16
94 
95  * Fix #176 hangs calculating deltas for files larger than 4GB. (dbaarda,
96  https://github.com/librsync/librsync/pull/177)
97 
98 ## librsync 2.2.0
99 
100 Released 2019-10-12
101 
102  * Bump minor version from 2.1.0 to 2.2.0 to reflect additional RabinKarp
103  rollsum support.
104 
105  * Fix MSVC builds by adding missing LIBRSYNC_EXPORT to variables in
106  librsync.h, add -DLIBRSYNC_STATIC_DEFINE to the sumset_test target,
107  and correctly install .dll files in the bin directory.
108  (adsun701, https://github.com/librsync/librsync/pull/161)
109 
110  * Add RabinKarp rollsum support and make it the default. RabinKarp is a much
111  better rolling hash, which reduces the risk of hash collision corruption
112  and speeds up delta calculations. The rdiff cmd gets a new `-R
113  (rollsum|rabinkarp)` argument with the default being `rabinkarp`, Use `-R
114  rollsum` to generate backwards-compatible signatures. (dbaarda,
115  https://github.com/librsync/librsync/issues/3)
116 
117  * Use single-byte literal commands for small inserts in deltas. This makes
118  each small insert use 1 less byte in deltas. (dbaarda,
119  https://github.com/librsync/librsync/issues/120)
120 
121  * Fix multiple warnings (cross-)compiling for windows. (Adsun701,
122  https://github.com/librsync/librsync/pull/165,
123  https://github.com/librsync/librsync/pull/166)
124 
125  * Change rs_file_size() to report -1 instead of 0 for unknown file sizes (not
126  a regular file). (dbaarda https://github.com/librsync/librsync/pull/168)
127 
128  * Add cmake BUILD_SHARED_LIBS option for static library support.
129  BUILD_SHARED_LIBS defaults to ON, and can be set to OFF using `ccmake .` to
130  build librsync as a static library. (dbaarda
131  https://github.com/librsync/librsync/pull/169)
132 
133  * Fix compile errors and add .gitignore entries for MSVS 2019. Fixes
134  hashtable.h to be C99 compliant. (ardovm
135  https://github.com/librsync/librsync/pull/170)
136 
137 ## librsync 2.1.0
138 
139 Released 2019-08-19
140 
141  * Bump minor version from 2.0.3 to 2.1.0 to reflect additions to librsync.h.
142 
143  * Fix exporting of private symbols from librsync library. Add export of
144  useful large file functions `rs_file_open()`, `rs_file_close()`, and
145  `rs_file_size()` to librsync.h. Add export of `rs_signature_log_stats()` to
146  log signature hashtable hit/miss stats. Improve rdiff error output.
147  (dbaarda, https://github.com/librsync/librsync/issues/130)
148 
149  * Updated release process to include stable tarballs. (dbaarda,
150  https://github.com/librsync/librsync/issues/146)
151 
152  * Remove redundant and broken `--paranoia` argument from rdiff. (dbaarda,
153  https://github.com/librsync/librsync/issues/155)
154 
155  * Fix memory leak of `rs_signature_t->block_sigs` when freeing signatures.
156  (telles-simbiose, https://github.com/librsync/librsync/pull/147)
157 
158  * Document delta file format. (zmj,
159  https://github.com/librsync/librsync/issues/46)
160 
161  * Fix up doxygen comments. (dbaarda,
162  https://github.com/librsync/librsync/pull/151)
163 
164 ## librsync 2.0.2
165 
166 Released 2018-02-27
167 
168  * Improve CMake install paths configuration (wRAR,
169  https://github.com/librsync/librsync/pull/133) and platform support
170  checking when cross-compiling (fornwall,
171  https://github.com/librsync/librsync/pull/136).
172 
173  * Fix Unaligned memory access for rs_block_sig_init() (dbaarda,
174  https://github.com/librsync/librsync/issues/135).
175 
176  * Fix hashtable_test.c name collision for key_t in sys/types.h on some
177  platforms (dbaarda, https://github.com/librsync/librsync/issues/134)
178 
179  * Format code with consistent style, adding `make tidy` and `make
180  tidyc` targets for reformating code and comments. (dbaarda,
181  https://github.com/librsync/librsync/issues/125)
182 
183  * Removed perl as a build dependency. Note it is still required for some
184  tests. (dbaarda, https://github.com/librsync/librsync/issues/75)
185 
186  * Update RPM spec file for v2.0.2 and fix cmake man page install. (deajan,
187  https://github.com/librsync/librsync/issues/47)
188 
189 ## librsync 2.0.1
190 
191 Released 2017-10-17
192 
193  * Extensively reworked Doxygen documentation, now available at
194  http://librsync.sourcefrog.net/ (Martin Pool)
195 
196  * Removed some declarations from librsync.h that were unimplemented or no
197  longer ever useful: `rs_work_options`, `rs_accum_value`. Remove
198  declaration of unimplemented `rs_mdfour_file()`. (Martin Pool)
199 
200  * Remove shipped `snprintf` code: no longer acutally linked after changing to
201  CMake, and since it's part of C99 it should be widely available.
202  (Martin Pool)
203 
204  * Document that Ninja (http://ninja-build.org/) is supported under CMake.
205  It's a bit faster and nicer than Make. (Martin Pool)
206 
207  * `make check` (or `ninja check` etc) will now build and run the tests.
208  Previously due to a CMake limitation, `make test` would only run existing
209  tests and could fail if they weren't built.
210  (Martin Pool, https://github.com/librsync/librsync/issues/49)
211 
212  * Added cmake options to exclude rdiff target and compression from build.
213  See install documentation for details. Thanks to Michele Bertasi.
214 
215  * `popt` is only needed when `rdiff` is being built. (gulikoza)
216 
217  * Improved large file support for platforms using different variants
218  of `fseek` (`fseeko`, `fseeko64`, `_fseeki64`), `fstat` (`fstat64`,
219  `_fstati64`), and `fileno` (`_fileno`). (dbaarda, charlievieth,
220  gulikoza, marius-nicolae)
221 
222  * `rdiff -s` option now shows bytes read/written and speed. (gulikoza).
223  For delta operations it also shows hashtable match statistics. (dbaarda)
224 
225  * Running rdiff should not overwrite existing files (signatures, deltas and
226  new patched files) by default. If the destination file exists, rdiff will
227  now exit with an error. Add new option -f (--force) to overwrite existing
228  files. (gulikoza)
229 
230  * Improve signature memory allocation (doubling size instead of calling
231  realloc for every sig block) and added support for preallocation. See
232  streaming.md job->estimated_signature_count for usage when using the
233  library. `rdiff` uses this by default if possible. (gulikoza, dbaarda)
234 
235  * Significantly tidied signature handling code and testing, resulting in more
236  consistent error handling behaviour, and making it easier to plug in
237  alternative weak and strong sum implementations. Also fixed "slack delta"
238  support for delta calculation with no signature. (dbaarda)
239 
240  * `stdint.h` and `inttypes.h` from C99 is now required. Removed redundant
241  librsync-config.h header file. (dbaarda)
242 
243  * Lots of small fixes for windows platforms and building with MSVC.
244  (lasalvavida, mbrt, dbaarda)
245 
246  * New open addressing hashtable implementation that significantly speeds up
247  delta operations, particularly for large files. Also fixed degenerate
248  behaviour with large number of duplicate blocks like runs of zeros
249  in sparse files. (dbaarda)
250 
251  * Optional support with cmake option for using libb2 blake2 implementation.
252  Also updated included reference blake2 implementation with bug fixes
253  (dbaarda).
254 
255  * Improved default values for input and output buffer sizes. The defaults are
256  now --input-size=0 and --output-size=0, which will choose recommended
257  default sizes based on the --block-size and the operation being performed.
258  (dbaarda)
259 
260  * Fixed hanging for truncated input files. It will now correctly report an
261  error indicating an unexpected EOF was encountered. (dbaarda,
262  https://github.com/librsync/librsync/issues/32)
263 
264  * Fixed #13 so that faster slack delta's are used for signatures of
265  empty files. (dbaarda,
266  https://github.com/librsync/librsync/issues/13)
267 
268  * Fixed #33 so rs_job_iter() doesn't need calling twice with eof=1.
269  Also tidied and optimized it a bit. (dbaarda,
270  https://github.com/librsync/librsync/issues/33)
271 
272  * Fixed #55 remove excessive rs_fatal() calls, replacing checks for
273  programming errors with assert statements. Now rs_fatal() will only
274  be called for rare unrecoverable fatal errors like malloc failures or
275  impossibly large inputs. (dbaarda,
276  https://github.com/librsync/librsync/issues/55)
277 
278 ## librsync 2.0.0
279 
280 Released 2015-11-29
281 
282 Note: despite the major version bump, this release has few changes and should
283 be binary and API compatible with the previous version.
284 
285  * Bump librsync version number to 2.0, to match the library
286  soname/dylib version.
287  (Martin Pool, https://github.com/librsync/librsync/issues/48)
288 
289 ## librsync 1.0.1 (2015-11-21)
290 
291  * Better performance on large files. (VictorDenisov)
292 
293  * Add comment on usage of rs_build_hash_table(), and assert correct use.
294  Callers must call rs_build_hash_table() after loading the signature,
295  and before calling rs_delta_begin().
296  Thanks to Paul Harris <paulharris@computer.org>
297 
298  * Switch from autoconf to CMake.
299 
300  Thanks to Adam Schubert.
301 
302 ## librsync 1.0.0 (2015-01-23)
303 
304  * SECURITY: CVE-2014-8242: librsync previously used a truncated MD4
305  "strong" check sum to match blocks. However, MD4 is not cryptographically
306  strong. It's possible that an attacker who can control the contents of one
307  part of a file could use it to control other regions of the file, if it's
308  transferred using librsync/rdiff. For example this might occur in a
309  database, mailbox, or VM image containing some attacker-controlled data.
310 
311  To mitigate this issue, signatures will by default be computed with a
312  256-bit BLAKE2 hash. Old versions of librsync will complain about a
313  bad magic number when given these signature files.
314 
315  Backward compatibility can be obtained using the new
316  `rdiff sig --hash=md4`
317  option or through specifying the "signature magic" in the API, but
318  this should not be used when either the old or new file contain
319  untrusted data.
320 
321  Deltas generated from those signatures will also use BLAKE2 during
322  generation, but produce output that can be read by old versions.
323 
324  See https://github.com/librsync/librsync/issues/5
325 
326  Thanks to Michael Samuel <miknet.net> for reporting this and offering an
327  initial patch.
328 
329  * Various build fixes, thanks Timothy Gu.
330 
331  * Improved rdiff man page from Debian.
332 
333  * Improved librsync.spec file for building RPMs.
334 
335  * Fixed bug #1110812 'internal error: job made no progress'; on large
336  files.
337 
338  * Moved hosting to https://github.com/librsync/librsync/
339 
340  * Travis-CI.org integration test at https://travis-ci.org/librsync/librsync/
341 
342  * You can set `$LIBTOOLIZE` before running `autogen.sh`, for example on
343  OS X Homebrew where it is called `glibtoolize`.
344 
345 ## 0.9.7 (released 2004-10-10)
346 
347  * Yet more large file support fixes.
348 
349  * `extern "C"` guards in librsync.h to let it be used from C++.
350 
351  * Removed Debian files from dist tarball.
352 
353  * Changed rdiff to an installed program on "make install".
354 
355  * Refactored delta calculation code to be cleaner and faster.
356 
357  * \#879763: Fixed mdfour to work on little-endian machines which don't
358  like unaligned word access. This should make librsync work on
359  pa-risc, and it makes it slightly faster on ia64.
360 
361  * \#1022764: Fix corrupted encoding of some COPY commands in large
362  files.
363 
364  * \#1024881: Print long integers directly, rather than via casts to
365  double.
366 
367  * Fix printf formats for size_t: both the format and the argument
368  should be cast to long.
369 
370 ## 0.9.6
371 
372  * Large file support fixes.
373 
374  * [v]snprintf or _[v]snprintf autoconf replacement function fix.
375 
376  * Changed installed include file from rsync.h to librsync.h.
377 
378  * Migration to sourceforge for hosting.
379 
380  * Rollsum bugfix that produces much smaller deltas.
381 
382  * Memory leaks bugfix patches.
383 
384  * mdfour bigendian and >512M bugfix, plus optimisations patch.
385 
386  * autoconf/automake updates and cleanups for autoconf 2.53.
387 
388  * Windows compilation patch, heavily modified.
389 
390  * MacOSX compilation patch, modified to autoconf vararg macro fix.
391 
392  * Debian package build scripts patch.
393 
394 ## 0.9.5
395 
396  * Bugfix patch from Shirish Hemant Phatak
397 
398 ## 0.9.4: (library 1.1.0)
399 
400  * Fixes for rsync.h from Thorsten Schuett <thorsten.schuett@zib.de>
401 
402  * RLL encoding fix from Shirish Hemant Phatak <shirish@nustorage.com>
403 
404  * RPM spec file by Peter J. Braam <braam@clusterfs.com>
405 
406  * No (intentional) changes to binary API.
407 
408 ## 0.9.3
409 
410  * Big speed improvements in MD4 routines and generation of weak
411  checksums.
412 
413  * Patch to build on FreeBSD by Jos Backus <josb@cncdsl.com>
414 
415  * Suggestions to build on Solaris 2.6 from Alberto Accomazzi
416  <aaccomazzi@cfa.harvard.edu>
417 
418  * Add rs_job_drive, a generic mechanism for turning the library into
419  blocking mode. rs_whole_run now builds on top of this. The
420  filebuf interface has changed a little to accomodate it.
421 
422  * Generating and loading signatures now generates statistics.
423 
424  * More test cases.
425 
426  * I suspect there may be a bug in rolling checksums, but it probably
427  only causes inefficiency and not corruption.
428 
429  * Portability fixes for alphaev67-dec-osf5.1; at the moment builds
430  but does not work because librsync tries to do unaligned accesses.
431 
432  * Works on sparc64-unknown-linux-gnu (Debian/2.2)
433 
434 ## 0.9.2
435 
436  * Improve delta algorithm so that deltas are actually
437  delta-compressed, rather than faked.
438 
439 ## 0.9.1
440 
441  * Rename the library to `librsync'.
442 
443  * Portability fixes.
444 
445  * Include the popt library, and use it to build rdiff if the library
446  is not present on the host.
447 
448  * Add file(1) magic for rdiff.
449 
450  * Add more to the manual pages.
451 
452  * It's no longer necessary to call rs_buffers_init on a stream before
453  starting to use it: all the internal data is kept in the job, not
454  in the stream.
455 
456  * Rename rs_stream_t to rs_buffers_t, a more obvious name. Pass the
457  buffers to every rs_job_iter() call, rather than setting it at
458  startup. Similarly for all the _begin() functions.
459 
460  * rs_job_new also takes the initial state function.
461 
462  * Return RS_PARAM_ERROR when library is misused.
463 
464 ## 0.9.0
465 
466  * Redesign API to be more like zlib/bzlib.
467 
468  * Put all command-line functions into a single rdiff(1) program.
469 
470  * New magic number `rs6'
471 
472  * Change to using popt for command line parsing.
473 
474  * Use Doxygen for API documentation.
475 
476 ## 0.5.7
477 
478  * Changes stats string format.
479 
480  * Slightly improved test cases
481 
482 ## 0.5.6
483 
484  * Don't install debugging tools into /usr/local/bin; leave them in
485  the source directory.
486 
487  * Fix libhsync to build on (sgi-mips, IRIX64, gcc, GNU Make)
488 
489  * Include README.CVS in tarball
490 
491  * Back out of using libtool and shared libraries, as it is
492  unnecessary at this stage, complicates installation and slows down
493  compilation.
494 
495  * Use mapptr when reading data to decode, so that decoding should
496  have less latency and be more reliable.
497 
498  * Cope better on systems that are missing functions like snprintf.
499 
500 ## 0.5.5
501 
502  * Put genuine search encoding back into the nad algorithm, and
503  further clean up the nad code. Literals are now sent out using a
504  literal buffer integrated with the input mapptr so that data is not
505  copied. Checksums are still calculated from scratch each time
506  rather than by rolling -- this is very slow but simple.
507 
508  * Reshuffle test cases so that they use files generated by hsmapread,
509  rather than the source directory. This makes the tests quicker and
510  more reproducible, hopefully without losing coverage. Further
511  develop the test driver framework.
512 
513  * Add hsdumpsums debugging tool.
514 
515  * Hex strings (eg strong checksums) are broken up by underscores for
516  readability.
517 
518  * Stats now go to the log rather than stdout.
519 
520  * mapptr acts properly when we're skipping/rewinding to data already
521  present in the buffer -- it does a copy if required, but not
522  necessarily real IO.
523 
524 ## 0.5.4
525 
526  * Improved mapptr input code
527 
528  * Turn on more warnings if using gcc
529 
530  * More test cases
531 
532 ## 0.5.3
533 
534  * Improvements to mapptr to make it work better for network IO.
535 
536  * Debug trace code is compiled in unless turned off in ./configure
537  (although most programs will not write it out unless asked.)
538 
539  * Add libhsyncinfo program to show compiled-in settings and version.
540 
541  * Add test cases that run across localhost TCP sockets.
542 
543  * Improved build code; should now build easily from CVS through
544  autogen.sh.
545 
546  * Improved trace code.
547 
548  * Clean up to build on sparc-sun-solaris2.8, and in the process clean
549  up the handling of bytes vs chars, and of building without gcc
550 
551  * Reverse build scripts so that driver.sh calls the particular
552  script.
553 
554 ## 0.5.2
555 
556  * Use mapptr for input.
557 
558  * Implement a new structure for encoding in nad.c. It doesn't
559  encode at the moment, but it's much more maintainable.
560 
561  * More regression cases.
562 
563  * Clean up build process.
564 
565 ## 0.5.0
566 
567  * Rewrite hs_inbuf and hs_encode to make them simpler and more
568  reliable.
569 
570  * Test cases for input handling.
571 
572  * Use the map_ptr idea for input from both streams and files.
573 
574 ## 0.4.1
575 
576  * automake/autoconf now works cleanly when the build directory is
577  different to the source directory.
578 
579  * --enable-ccmalloc works again.
580 
581 ## 0.4.0
582 
583 * A much better regression suite.
584 
585 * CHECKSUM token includes the file's checksum up to the current
586  location, to aid in self-testing.
587 
588 * Various bug fixes, particularly to do with short IO returns.
LIBRSYNC_EXPORT rs_result rs_build_hash_table(rs_signature_t *sums)
Call this after loading a signature to index it.
Definition: sumset.c:275
rs_job_t * rs_delta_begin(rs_signature_t *sig)
Prepare to compute a streaming delta.
Definition: delta.c:386
The hashtable type.
Definition: hashtable.h:130