nx-cugraph: adds ancestors, descendants, and BFS algos

[eriknw]

This PR adds the following:

• ancestors
• bfs_edges
• bfs_layers
• bfs_predecessors
• bfs_successors
• bfs_tree
• descendants
• descendants_at_distance
• generic_bfs_edges

[eriknw]

@rlratzel what do you think about the three tests that fail due to running out of GPU memory? I don't think the graphs are particularly large. What can we try? Can we investigate? Could it help to call gc.collect() regularly during tests (since maybe device memory is being help by released Python objects, but Python's garbage collection doesn't trigger b/c host memory usage is low)? Or, should we just skip the tests (and maybe run them one-by-one)?

[rlratzel]

@rlratzel what do you think about the three tests that fail due to running out of GPU memory? I don't think the graphs are particularly large. What can we try? Can we investigate?

I was wondering if we have a memory leak, possibly related to PLC graph creation during the conversion from the NX Graph to the PLC Graph happening more often now that nx-cugraph has implemented more algos. Do you see this when you run tests locally, or see anything that resembles a leak when polling nvidia-smi during testing?

Could it help to call gc.collect() regularly during tests (since maybe device memory is being help by released Python objects, but Python's garbage collection doesn't trigger b/c host memory usage is low)?

Maybe? If this is happening during tests, I'd be concerned that it could happen in a large user application too, and they may not know or would prefer not to manually invoke the GC. I'll try recreating this on my workstation using your branch. I can then add a test setup hook that calls gc.collect() before each test to see if that makes a difference.

[rlratzel]

@eriknw I just ran your branch on my workstation and saw the same results - the same three tests failed with the same OOM error. I was polling nvidia-smi and never saw it exceed ~800MB, and my workstation GPUs have (I think) more memory than the V100s in the CI runners (I have 50GB cards).

I don't think this is a leak or anything gc.collect() will fix; I think there's a bug somewhere in the algos (or combination of algos) called from the tests. Are you seeing the same result when you run these tests locally?

I suspect something is causing an abrupt spike in memory which fails the test and the memory is reclaimed faster than my nvidia-smi polling interval, or there's a bug where we're computing an amount of memory to allocate that's much too big. I'm just guessing at this point though. I'll keep looking...

[rlratzel]

@eriknw - I have a minimal repro script. Can you use this to debug further? I'm going to stop for the night but I can help next week.

import networkx as nx
from networkx.algorithms import flow

G = nx.fast_gnp_random_graph(100, 0.0575, seed=42)
cut = nx.minimum_node_cut(G, flow_func=flow.boykov_kolmogorov)

G = nx.DiGraph()
G.add_nodes_from([0, 1, 2])
list(nx.algorithms.dag.antichains(G))

(base) root@98e76e671240:/Projects/eriknw/cugraph# NETWORKX_AUTOMATIC_BACKENDS=cugraph python oom_repro.py
Traceback (most recent call last):
  File "/Projects/eriknw/cugraph/oom_repro.py", line 9, in <module>
    list(nx.algorithms.dag.antichains(G))
  File "/opt/conda/lib/python3.10/site-packages/networkx/algorithms/dag.py", line 940, in antichains
    TC = nx.transitive_closure_dag(G, topo_order)
  File "/opt/conda/lib/python3.10/site-packages/networkx/utils/decorators.py", line 770, in func
    return argmap._lazy_compile(__wrapper)(*args, **kwargs)
  File "<class 'networkx.utils.decorators.argmap'> compilation 28", line 4, in argmap_transitive_closure_dag_25
    import inspect
  File "/opt/conda/lib/python3.10/site-packages/networkx/utils/backends.py", line 576, in __call__
    return self.orig_func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/networkx/algorithms/dag.py", line 811, in transitive_closure_dag
    TC.add_edges_from((v, u) for u in nx.descendants_at_distance(TC, v, 2))
  File "/opt/conda/lib/python3.10/site-packages/networkx/utils/backends.py", line 569, in __call__
    return self._convert_and_call(
  File "/opt/conda/lib/python3.10/site-packages/networkx/utils/backends.py", line 817, in _convert_and_call
    result = getattr(backend, self.name)(*converted_args, **converted_kwargs)
  File "/Projects/eriknw/cugraph/python/nx-cugraph/nx_cugraph/utils/decorators.py", line 86, in __call__
    return self.__wrapped__(*args, **kwargs)
  File "/Projects/eriknw/cugraph/python/nx-cugraph/nx_cugraph/algorithms/traversal/breadth_first_search.py", line 239, in descendants_at_distance
    distances, predecessors, node_ids = plc.bfs(
  File "bfs.pyx", line 181, in pylibcugraph.bfs.bfs
  File "utils.pyx", line 53, in pylibcugraph.utils.assert_success
RuntimeError: non-success value returned from cugraph_bfs: CUGRAPH_UNKNOWN_ERROR std::bad_alloc: out_of_memory: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp

[eriknw]

Do you see this when you run tests locally, or see anything that resembles a leak when polling nvidia-smi during testing?

Yeah, I see it (just like you) when I run the tests locally. However, if I run each failing test individually, they pass.

Thanks for investigating and making a reproducer!

[ajschmidt8]

Approving ops-codeowner file changes

[rlratzel]

We can leave this in place in order to get this PR merged, but I think this is masking a bug (likely in pylibcugraph/libcugraph, not in nx-cugraph).

I filed this issue and I'll open a PR to remove these lines and the skipped tests separately, which will be merged when the bug is fixed.

[rlratzel]

The PR to eventually fix the bug is #4068

[rlratzel]

/merge