Add the ability to manage concurrency in zetaback backups. Running against a lot of hosts may require multiple concurrent jobs running. Zetaback already supports correct function when run concurrently. This enhancement should add intelligence to the daemon to determine when concurrency is a good idea and when it is not.
- Launching 8 jobs and having each ssh into every machine to build a todo list is dumb.
- Parallel jobs against the same target host can be self-defeating due to spindle contention (two at the same time but more than twice as slow)
- Consider a ramp up philosophy... start one, measure, if it is not enough double efforts, repeat. Add a hard configurable limit to concurrency.
- Scanning is not an insignificant effort (getting the remote FS list). Making that more parallel is self-defeating. Maybe a local cache of that would be good (1 hour TTL)?