Ticket #27 (new enhancement)

Opened 5 years ago

Concurrency Management

Reported by: jesus Assigned to: jesus
Priority: major Milestone:
Component: component1 Version:
Keywords: Cc:

Description

Add the ability to manage concurrency in zetaback backups. Running against a lot of hosts may require multiple concurrent jobs running. Zetaback already supports correct function when run concurrently. This enhancement should add intelligence to the daemon to determine when concurrency is a good idea and when it is not.

Thoughts:

  • Launching 8 jobs and having each ssh into every machine to build a todo list is dumb.
  • Parallel jobs against the same target host can be self-defeating due to spindle contention (two at the same time but more than twice as slow)
  • Consider a ramp up philosophy... start one, measure, if it is not enough double efforts, repeat. Add a hard configurable limit to concurrency.
  • Scanning is not an insignificant effort (getting the remote FS list). Making that more parallel is self-defeating. Maybe a local cache of that would be good (1 hour TTL)?