v4.0.13

This release includes minor bug fixes and improvements

Bug fixes

• When run with --zip, do not remove non-empty metadata directories.

Python adapter

• Handle conversion of pathlib.Path objects to strings when
  serializing outputs.
• Add martian.clear() as a convenience method to set all output
  parameters to null.

Documentation

• Fix some typos in python docstrings.
• Improve comments in parts of the go code.
• Add a note in the slurm template about issues with VSizeFactor.

Logging and error messages

• Improve the error message for "not queued or running" in cluster mode,
  to clarify the difference between a job that started but then terminated
  unexpectedly without reporting an error vs. a job which never started up at all.
• Auto-retry on slurm socket timeout error messages.
• Correctly detect and report memory limits imposed with cgroup v2 as well
  as cgroup v1.

v4.0.12

Bug fixes

• Fix reporting of errors during startup (c178291)
• Fix an issue with restarting failed pipelines involving a map call
  over maps (cde0239)
• Fix an issue with mrjob hitting a self-imposed address space size
  limit when running with --monitor (68756c6)

Web UI Improvements

• Change the color used to display disabled stages (f3ee627)
• Add a button to the web UI to hide the sidebar and make the graph display
  larger (75a63f6)

Other improvements

• Set a few more environment variables for limiting threads
  (2dac501, b10737d)
• Log a few environment variables to detect running on HTCondor (1ea97ef)
• Add support for self-profiling of the binaries, and enable PGO for a few
  of them (d6cade9)
• When running with --zip, enable compression for larger metadata files
  (06bfc8b)
• Assorted updates to dependencies.

v4.0.11

Changes to mrp:

• Minimum required version of go is now 1.20. This is in keeping with
  that being the oldest version of go still supported upstream.
• corrections to debug logging
• dependency updates
• do not pull _perf.data into _metadata.zip when running with --zip.
  Those files are large, and you will usually want a post-processing
  step to compute aggregate statistics from them and then delete the
  originals, which gets complicated if they're inside the zip file.
• When running with the MARTIAN_ENTERPRISE environment variable set,
  report errors during startup (e.g. errors parsing the mro input)
  to the server. Otherwise, errors only get reported if they happen
  after startup.
• Allow SerializeState calls to be aborted. For very large pipelines,
  these queries can take a long time. If they are for serving an API
  request, that request may time out or be cancelled before it returns,
  in which case we want to cancel in-flight requests, especially since
  the client may attempt to retry the request.

VSCode extension:

• corrections to handling of relative MROPATH
• dependency updates

v4.0.10 - bug fixes for cluster mode

Bug fixes:

• Fix a bug with how the executable path is computed with Go 1.19.
• Fix a bug with auto-restart in cluster mode when jobs are queued due
  to --maxjobs.

Other changes:

• Removed the bazel targets for generating tarballs. They were not
  being used, and bazel 6.0 removes native support for them.
• Update various dependencies.

v4.0.9 - Bug fixes and logging improvements

What's changed

User-visible changes

• mrjob will now log a message if stage code starts a subprocess which outlives the parent process.
• Add slurm queue check script to bazel deployment package. Previously it was inadvertently omitted.
• Thread-control environment variables (e.g. OMP_NUM_THREADS) are now set unconditionally. Previously they would only be set in cluster mode or in local mode when the thread limit was set to less than the total number of CPUs on the machine.
• The python adapter no longer supports python 2.
• Partially fix a bug which could result in the pipeline hanging after an auto-restart in cluster mode with --maxjobs set to a value other than infinity. There are some cases where this can still happen, which are still under investigation.

Developer-facing changes

• Everything in the repository should now build on windows (with potentially reduced functionality. Don't expect to be able to run pipelines).
• Fix up some docstrings.
• Add an integration test for cluster mode.
• Various dependency updates.

Full Changelog: v4.0.8...v4.0.9

v4.0.8 - improvements to build

What's Changed

• Update bazel rules to be more efficient, taking advantage of bazel 5.0.
• Add dependabot for github-actions.
• Various cleanups and minor fixes to vscode extension.
• Update mro2go to be able to generate pipeline input structs.
  It should now be more useful when building tools to generate
  .mro code for calling pipelines.
• Store timestamps in jobinfo and other places in RFC3339 format,
  with time zone. This fixes a bug where a change in time zone
  could result in incorrect computation of storage high-water-mark.
  This would mostly manifest around daylight savings time, but could
  be a problem in cluster mode if compute nodes aren't using the same
  timezone as the submit host.
• Remove gulp. Gulp wasn't doing much at this point, and has many
  dependencies which generated a continuous stream of security
  alerts. This also aligns the bazel build for the web files better
  with the non-bazel build.
• Switch from npm to yarn. Yarn's lockfile format is less prone to
  churn during updates.

Full Changelog: 10XDev/martian-public@v4.0.7...v4.0.8

Minor bugfix release v4.0.7

NOTE: We plan for this to be the final release to support python 2.7 for
the python adapter shim. Future versions will require python 3.7 or
higher.

Feature improvements:

• Log a few more python-related environment variables
  • _PYTHON_SYSCONFIGDATA_NAME and it's _CONDA-prefixed version
  • JPY_PARENT_PID
  • MPLCONFIGDIR
  • Pefixes for CUDA, OMP, MKL, HDF5, and NUMEXPR.
• Log uid, gid for user and filesystems.
• Log the job ID for "not queued or running" errors.
• Add profile mode py-spy.
• Add automatic retry condition for jobcmd error.
• Bump various dependencies.

Bug fixes:

• Switch from travis to github actions for CI.
• Minor streamlining of the bazel build.
• Fix bug preventing correct logging of memory usage for long-running
  jobs.
• Fix statfs test for overlayfs
• Fixes for python unit testing mode.

Martian 4.0.6

Feature improvements

• The way mrjob scans its child processes in order to track memory and I/O usage is now more efficient. It also polls more frequently when running with --monitor.
• On linux operating systems, mrjob is now set as a subreaper for its child processes, allowing it to collect rusage information when stage code launches child processes but fails to wait on them.
• The API backing the web UI will now set the Content-Type and other headers more accurately for most metadata files.
• Minor bumps to dependencies.

Bug fixes

• Correctly handle reattaching to a pipestance via _mrosource.
• Catch and report import errors for the stage code in the python adapter.
• Fix a error in the VSCode extension which prevented formatting from working when there was no workspace defined.

v4.0.5

Bump js-yaml from 4.0.0 to 4.1.0 in /tools/syntax/vscode (#270)

Release v4.0.4

• pug is no longer used to build graph.html for the user interface.
• The minified graph.html for the user interface is now deployed in the
  serve directory with everything else, rather than in templates.
• Memory request information is now reported in the _perf file,
  alongside actual memory usage reporting.
• The scripts used to check whether a job is still running on SGE or
  Slurm clusters are now compatible with python 3.
• The UUID set for the pipestance is now exposed to stage code via the
  MRO_UUID environment variable. The martian helper module for
  python stage code exposes this through the get_pipestance_uuid()
  convenience method.
• Added type hint comments for many of the functions in the martian helper
  module for python stage code. In particular, add NoReturn annotations
  to the martian.throw() and martian.exit() functions, to get better
  results from linting tools on code using those methods.
• Various minor dependency updates.