Analysis: PBS and Cython integration

Developers
1

Hi All,

I have posted my analysis on PBS and Cython integration.

The analysis gives a brief introduction to Cython and some insight into the PBS hooks infrastructure and how Cython can be integrated with PBS.

I request the community to review the analysis and provide feedback and suggestions.

Thanks,
Prakash

1 Like
2

Hi Prakash,

Thanks for sharing this.
We like the idea of cython and are working on a response but a couple of people are on vacation.
We will provide a response when we have had a chance to get their input.

Thanks!
Saurabh

1 Like
3

Thank you Saurabh. Waiting eagerly :slight_smile:

4

Hi Prakash,

Here are our thoughts on this:

Overall, we think the use of Cython is a great idea and will have some significant benefits. You list many of the benefits. Some of the things we are particularly interested in are:

  • It will help avoid humans (us :blush: ) having to write/edit complicated and/or cookie cutter code. The python C API can be difficult to code against and requires a lot of bookkeeping that Cython handles for you. For instance, Cython will do all the reference counting helping to avoid memory leaks. This might or might now reduce the frequency of interpreter restarts.
  • Cython will also make changing Python versions much easier. We ran into an example of this and filed an issue (https://github.com/openpbs/openpbs/issues/1577). In this case, StopIteration (and generators in general) changed a lot from Python 3.6 to 3.7. It still requires human intervention to fix, but you can fix it in the python code rather than the C code. Once you do that Cython will do the appropriate C code generation.

In terms of how/where to use Cython, here are our thoughts:

  • The server (pbs_python_svr_internal.c): This is where we think the real win will be and is where we would suggest starting. The functions you list in the table under ā€œDetail on code that will cleaned upā€ are good choices. Perhaps start with just one of the simpler ones and then work through them if the first one goes well.
  • pbs_ifl: This one we had a variety of opinions on. While there are some potential benefits, and we donā€™t see it being problematic, there were some who questioned the utility. pbs_ifl is heavily used, relatively stable, mostly communication and the Python bindings are generated via SWIG.
  • Hook scripts: We are not really a fan of this idea, particularly if it becomes the only option. This impacts a much broader set of people (anybody writing a script hook) and would add a great deal of complexity. Our initial take is that this would be in exchange for relatively little performance gain, though of course that remains to be seen. One thing in particular that is a factor in our environment is that you canā€™t assume a homogeneous environment. Our compute nodes are often different architectures than the server nodes, so you canā€™t necessarily precompile and push object code out to the MOMs. We also worry this could make the hook scripts much harder to debug. We fear this would discourage people from taking advantage of a very useful and powerful feature in OpenPBS.

Thanks!
@toonen @pershey
Saurabh

2 Likes
5

Thanks for the great feedback @sdass In fact we fundamentally felt the same that you expressed, i.e. leave the python hooks themselves as python, but left it in there for completeness sake of the analysis.

With this feedback, we hope to find time to soon take the suggested changes forward and implement them.

1 Like
6

@prakashcv13 : nice start on exploring how to import PBS Python hooks implementation by using cython. I also feel that we shouldnā€™t be converting userā€™s python hook scripts to cython. But if we do, would it be like when the site hook is imported, pbs would dynamically run cython to produce C equivalent files, and then compile into shared object, before PBS executes them? That might actually add a delay when importing hooks.
On the other hand, the pbs hooks can be converted by cython since PBS internally ships them.
It would be nice too if this use of cython can de-couple python hook execution from main pbs_server. Currently, if thereā€™s a problem with the hook (like a bad mem acess) and causes a process crash, it also likely to take down pbs_server as interpreter is embedded. That doesnā€™t happen with mom hooks as pbs_mom calls ā€˜pbs_pythonā€™ to execute the hook, and if thereā€™s a problem, only the ā€˜pbs_pythonā€™ process goes down.